Self-Play Pretraining with Zero Data

Researchers have introduced a method called Self-Play Pretraining that allows language models to generate their own training data rather than relying on curated external datasets. This approach challenges the prevailing reliance on massive human-collected archives by testing whether models can improve through internal generation. By potentially removing the bottleneck of data scarcity, this technique could shift how foundational models are developed and scaled.