0G Labs, in collaboration with China Mobile, has published a research paper introducing a new framework called DiLoCoX for decentralized training of large language models. The project aims to overcome limitations related to network performance.
What is DiLoCoX?
DiLoCoX is a cutting-edge framework designed for training large language models (LLMs) exceeding 100 billion parameters in decentralized environments with limited bandwidth. It showcases new possibilities for training over slow networks, circumventing the drawbacks of existing decentralized models.
Achievements and Efficiency of DiLoCoX
The research paper shows that DiLoCoX is capable of pre-training a 107 billion parameter model over a 1 Gbps network. The project's achievements include achieving 357 times greater speeds in distributed training compared to traditional methods while maintaining a high level of model convergence. This framework is the first successful solution for training such large models in decentralized conditions.
Remarks from 0G Labs Management
“DiLoCoX is both a proof of concept and a statement of intent,” said Michael Heinrich, CEO of 0G Labs. “We are unlocking a future where AI serves as a public good, creating an open ecosystem where everyone can participate and benefit from intelligent systems.”
The release of the research paper underscores 0G Labs' commitment to democratized AI and lays the groundwork for the development of new applications based on high-performance infrastructure.
