Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs
huggingface ·
AllenAI has released Olmo-core 3, an open-source training infrastructure designed to scale Mixture-of-Experts (MoE) models into the trillion-parameter range while maintaining computational efficiency. The framework shifts from fully sharded data parallelism to distributed data parallelism, keeping experts resident on GPUs to avoid repeated weight gathering and significantly improving throughput. Key optimizations include expert, pipeline, and optimizer parallelism, alongside techniques like rowwise expert parallelism and grouped GEMM to reduce routing costs. The system also supports MXFP8 lower-precision formats, which increased training throughput by approximately 21% in benchmarks compared to BF16. AllenAI demonstrated the stack's capability with a 1.2-trillion-parameter model achieving high FLOP/s/GPU metrics and experimented with configurations reaching up to 2.38 trillion parameters. The release includes a technical report detailing experimental findings, such as issues with balanced routing scores and communication overlaps. This open infrastructure serves as the foundation for the next generation of Olmo models, aiming to provide academic researchers and smaller labs with accessible tools for advanced MoE development.
huggingface