Can synthetic data, self-play, simulation, and interaction substitute for the finite supply of high-quality human-generated data?
Frontier pretraining has plausibly consumed most high-quality public text. Synthetic data clearly works where verification is cheap (math, code); whether it generalizes to open-ended domains without model collapse is contested.
View on the map → · Open in Browse →
The synthetic-data question moved from theory to measurement: the most systematic study to date finds rephrased synthetic data mixed with web text speeds pretraining 5–10x while purely generated 'textbook' data reproduces model collapse — and a run of theory papers converges on the same condition: self-play and synthetic loops keep working only while a verifier or pipeline injects genuinely new information.
Pre-training on rephrased synthetic data alone is not faster than pre-training on natural web texts; while pre-training on 1/3 rephrased synthetic data mixed with 2/3 natural web texts can speed up 5-10x.
The most systematic empirical study to date (1000+ models trained, from Meta researchers) of when synthetic pretraining data helps — and when it reproduces model-collapse patterns.
by injecting information through an external synthetic data verifier, whether a human or a better model, synthetic retraining will not cause model collapse
Verification injects enough external information to avert collapse in recursive synthetic training — but performance converges to the verifier's knowledge, formalizing why verifier quality, not data volume, is the binding constraint.
a self-synthesised data pipeline with learnable information that increases across iterations
A general condition for when self-play loops keep improving versus stall: the pipeline must add learnable information each iteration, not just more data — a principled account of the limits of data-free self-improvement.
we ground the framework on a content-rich pretraining corpus to (1) enable an exploitable generation-verification gap and reduce reward hacking, and (2) prevent mode collapse.
A concrete attempt at the contested step: extending self-play training signals beyond cheaply verifiable domains to open-ended tasks by grounding rubrics in pretraining text.