Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline

Researchers at Hugging Face have released a dataset and training framework designed to improve language model performance in Indic languages by using 2.8 trillion tokens. This initiative addresses the underrepresentation of regional languages in AI by providing an open pipeline for data generation and curation. By lowering the barrier for developing multilingual models, the project aims to improve accessibility and accuracy for speakers of diverse Indian languages.