On-Policy Distillation keeps improving a student model when trained on a single query because the bottleneck isn’t data variety but how slowly the student absorbs teacher supervision. One query already reaches 71.5% of the token-level states that full-data training visits.
You’re distilling a big model into a small one for production. The playbook says: gather thousands of curated prompts, run the student, have the teacher grade every token. This paper looks at the extreme opposite: train on one prompt, repeated for hundreds of steps. It still recovers most of the gain from the full dataset across math, code, instruction-following, and tool-use. That reframes what “data curation” means for post-training distillation. The relevant baseline here is On-Policy Distillation as used in Qwen3, DeepSeek, GLM, and Kimi post-training stacks, where the student samples its own rollouts and the teacher supplies a full next-token distribution at every prefix.
The key reframing: OPD doesn’t really train on queries, it trains on states. A state is a prefix (x, y_<i) where the student has generated some tokens and the teacher can score the next-token distribution. A single query, rolled out 64 times per step, produces tens of thousands of distinct prefixes, each one a supervised training signal.
To measure how much of the training “space” a query covers, the authors define state coverage. They take the teacher’s final-layer hidden vector at each state, run PCA and K-means clustering over all states that full-data OPD visits, and get 200 clusters. Any run’s coverage is the fraction of those clusters its rollouts reach.
On the algorithm side, they track two things at every step: the average teacher-student log-probability disagreement (the distance), and the fraction of that distance one gradient step closes (the absorption rate).
for step in range(300):
rollouts = student.sample(query, n=64) # one query, 64 trajectories
for (x, y) in rollouts:
for i in range(len(y)):
s = (x, y[:i])
# teacher gives full distribution at every prefix
loss += KL(student(s), teacher(s))
student.update(loss)
# measure: coverage(rollouts) and distance(student, teacher)
The absorption rate falls throughout training at roughly the same pace whether you use 1 query or all 17k. So the length of a run is a property of the optimizer, not the dataset.
The prevailing assumption in post-training is that more curated problems yield more supervision, so data scale matters. This paper shows the opposite. OPD is data-overfed but algorithm-starved: a single prompt already exposes more teacher supervision than the student can absorb in hundreds of steps, and adding queries only helps when they reach genuinely new states. The load-bearing evidence is that the absorption rate decays identically for 1 query and for 17k queries, and that an off-policy run on a frozen set of 64 trajectories still takes ~200 steps to plateau.
•
The absorption rate slows at the same pace regardless of dataset size. Between step 30 and step 300, 1-query, 4-query, 16-query, and full-data runs all remove 78–84% of their starting teacher-student distance, and all slow by a similar factor. The training set doesn’t set the pace; the algorithm does.
•
One query covers 71.5% of the state clusters that full-data OPD visits, most of it within the first 100 steps.
•
16 semantically diverse queries reach 98.9% coverage and match full-data training, both in single-domain OPD and in MOPD where a student is jointly distilled from three domain teachers. 16-shot MOPD recovers 101% of full-data MOPD’s gain.
•
One-shot OPD recovers 87% of full-data OPD’s gain at step 300 on math (averaged over MATH-500, AMC 2023, AIME 2025), and holds across Qwen, Llama, and OLMo student-teacher pairs, and across code (LiveCodeBench), instruction-following (Multi-IF), and tool-use (Berkeley Function Calling Leaderboard).
•
Content-light inputs work too. An empty <think> template, or off-domain WildChat chat prompts (only 0.17% are math-related), drive OPD nearly as effectively as the real math training set. The input’s job is to start the student reasoning, not to state the task.
•
Head-to-head on the same query, OPD’s validation gain over 1000 steps is more than twice that of one-shot RL with verifiable rewards. RLVR’s outcome reward dies once the query is reliably solved; OPD’s per-token signal keeps producing gradients.
Reach for this when you’re distilling a specialist model into a smaller student and thinking about how much prompt curation to invest in. The finding says: stop optimizing for prompt count. Pick ~16 queries per domain that cover semantically distinct clusters (they used BGE-M3 embeddings plus K-means to pick representatives), and put the engineering effort into either (a) getting a better teacher or (b) making the optimizer absorb signal faster, e.g., reusing batches for multiple epochs under a per-token trust region, or up-weighting tokens where the teacher-student gap is still large.
Code is at One-Shot-OPD, built on verl. The training queries used for the difficulty ablation are listed explicitly in the paper’s appendix, so the one-shot runs are directly reproducible. No new dataset is released; everything runs on existing public sets (DAPO-Math-17k, TACO, xLAM-function-calling-60K, UltraData).
In on-policy distillation, one prompt already floods the student with more supervision than it can digest, so spend your budget on the optimizer, not the dataset. The lever that actually moves the run is the absorption rate; picking 16 semantically diverse prompts per domain is enough to saturate the data side, and everything past that is wasted collection effort.
•
State coverage is measured against a reference space built from full-data rollouts, so it tells you how much of that space a query set reaches, not what a query set covers on its own. You can’t currently estimate coverage without first running full-data OPD.
•
The MOPD result uses three domains with one teacher each. Whether 16 diverse queries per domain still suffices with 10+ teachers is untested.
•
Content-light templates work here, but a scaffold that closes the thinking block immediately collapses into short meta-replies. “Any prompt works” is too strong; the prompt has to leave the model room to reason.
•
All students are 1.5B–7B. The claim that the optimizer, not the data, is the bottleneck may not hold at frontier scale where teacher-student gaps and absorption dynamics could look different.