Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings

Apple researchers have demonstrated that mixing data from high-resource languages into training sets improves model performance more effectively than hyperparameter tuning for low-resource languages. This technique addresses the quality degradation that typically occurs when a model is forced to repeat limited datasets during pre-training. By prioritizing data diversity over iterative parameter adjustments, this approach provides a more scalable strategy for developing language models in scenarios where training data is scarce.