Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline
huggingface ·
NVIDIA and AI4Bharat have released Nemotron-pretraining-Indic-v1, a massive corpus containing approximately 2.8 trillion tokens across all 22 languages listed in India's Constitution. This release is accompanied by an open, GPU-accelerated data generation pipeline built on NeMo Data Designer, allowing developers to replicate and adapt the process for their own needs. The dataset includes native-script data for all 22 languages and romanized text for twelve, covering diverse content such as mathematics, code, and general web text. To ensure quality, the workflow employs automated translation, rigorous assessment using MQM-inspired rubrics, and targeted repair of low-scoring outputs. Continued pretraining of the Nemotron 3.5 Super VL model with this synthetic data resulted in significant performance gains across multiple Indic knowledge and reasoning benchmarks. The authors emphasize that while this synthetic data expands resources, it should be combined with naturally authored Indic text to preserve local cultural context and knowledge. By sharing both the corpus and the generation recipe, the team aims to lower barriers for building stronger multilingual models for nearly 1.2 billion Indian citizens.
huggingface