Get Started
Home
Topics
Search
Library
Research questionWhen and why can repeating a smaller dataset reduce training compute versus using more unique samples?Training on fewer unique examples can sometimes reach learning targets with less computation than training on a larger collection. The conditions and mechanism behind this small-vs-large gap are not fully explained by existing theory.
AI
Machine Learning
Reasoning
Research Paper
Statistical Machine Learning
Technology
Latest papersRecent research connected to this question, newest first.Less Data, Faster Training: repeating smaller datasets speeds up learning via sampling biasesThe evidence covers algorithmic tasks, multiple architectures and optimizers, with particular relevance to reasoning tasks. The work combines theoretical analysis with empirical interventions and attributes the effect to layer-wise growth associated with sampling biases, which are stronger for smaller datasets.research paper · Sep 4, 2026
Related questions
How can small-scale pretraining mixture experiments stay reliable when scarce high-quality data is repeated at target scale?Can text-to-image models match web-scale performance using smaller, reproducible datasets and models?How should optimizer selection and hyperparameter tuning account for longer training horizons?Does reusing Transformer layers improve language-model quality when parameter, compute, and KV-cache budgets are matched?