Relay-OPD fixes broken student rollouts during On-Policy Distillation by letting the teacher briefly grab the pen at the exact tokens where its top choice is a reflection word (But, Wait) but the student’s isn’t, cutting training trajectory length by over 50% while lifting average math accuracy by +5.73 pp over standard OPD.
You’re distilling a small model from a larger one on reasoning traces. The modern recipe is to let the student generate its own rollouts and have the teacher score each token, so training data matches what the student actually produces at inference. Problem: when the student goes off the rails at token 200, it keeps digging for another 2000 tokens, and every one of those tokens gets “supervision” that’s really just the teacher trying to salvage a doomed prefix. You pay full compute for garbage gradients.
Prior fixes are blunt. FastOPD just chops rollouts at a fixed length. TRD (Trajectory-Refined Distillation) rewrites failed trajectories offline with the teacher, which leaves telltale “as I revise the original solution” artifacts. SKD (Speculative Knowledge Distillation) swaps in teacher tokens whenever the two distributions disagree, but generic disagreement doesn’t specifically mean “the reasoning went wrong.”
The authors notice something concrete: on a failed prefix, the teacher and student disagree in a specific way. The teacher wants to emit a reflection token, English words like But, Wait, However that signal “stop, reconsider.” The student wants to plow forward with So, Now, Therefore. That asymmetry is the failure signal, and it needs no verifier, no reward model, no label.
Concretely, at each student-generated position they check two things on the current prefix: (1) is the teacher’s top-1 next token in a fixed set of reflection words, and (2) is the student’s top-K (K=5) next-token set free of any reflection word. When both hold, that’s the handoff trigger. The teacher then writes a teacher leg: the reflection token itself plus L=3 short paragraphs of continuation. Then the student resumes. A relay budget caps this to M=2 takeovers per rollout, keeping the trajectory close to the student’s own policy.
Two empirical facts justify the sparsity. Even replacing 0.35% of tokens with teacher tokens lifted accuracy by 7 pp in preliminary probes. And late takeovers help far less than early ones, because by then the teacher gets dragged along by the accumulated student context.
The student is trained on the resulting mixed trajectory with a reverse-KL-style advantage on the actually generated tokens (whether student- or teacher-emitted), inside a Group Relative Policy Optimization (GRPO)-style clipped policy-gradient loss. The whole rollout runs inside one Speculative Decoding engine where the student is the draft model and the teacher is the target, so switching legs is free.
z, takeovers, state = [], 0, "S"
while not done(z):
h = prompt + z
a_T = argmax(pi_teacher(h))
topk_S = topk(pi_student(h), K=5)
trigger = (a_T in REFLECTION) and (topk_S & REFLECTION == set())
if state == "S" and trigger and takeovers < M:
z.append(a_T) # teacher opens with reflection token
z.extend(teacher_paragraphs(h, L)) # L paragraphs via spec decoding
takeovers += 1
if takeovers == M: break
else:
z.append(sample(pi_student(h))) # student leg continues
The prevailing fix for a failed student rollout is either to truncate it (FastOPD) or to swap tokens whenever teacher and student disagree (SKD (Speculative Knowledge Distillation)). This paper shows the opposite. The right signal isn’t distributional disagreement in general, it’s a specific direction disagreement: teacher wants to reflect, student wants to press on. Intervene there, briefly, only early, then hand control back. The evidence that isolates this is the Trigger-Stop ablation, which stops at the trigger but writes no teacher tokens and gains far less than a real teacher leg.
The load-bearing result is the teacher-leg ablation. Trigger-Stop (detect the trigger, terminate the rollout, generate no teacher tokens) reaches 43.48 average accuracy. Adding a 3-paragraph teacher leg at the same trigger takes it to 46.25 (+2.77 pp). So the win isn’t from cutting bad tokens, it’s from the short teacher demonstration of how to redirect.
•
On the Qwen3-1.7B-Non-Thinking student across eight math benchmarks (AIME 2024/25/26, MATH500, AMC 2023, OlympiadBench, HMMT Nov 2025 Feb26 and Nov25), Relay-OPD averages 46.96 vs 41.23 for standard OPD (+5.73 pp) and 45.47 for the best FastOPD truncation (+1.49 pp).
•
Training trajectories shrink from 4,658 to 2,296 tokens (−50.7%), and the best checkpoint arrives at step 35 instead of 55.
•
Objective ablation: distilling on the actually generated relay tokens (46.96) beats forward-KL on the teacher’s full distribution (44.08). The mode-covering FKL forces the student to fit the teacher everywhere, including where the teacher’s guidance on a broken prefix isn’t reliable.
•
Sensitivity: teacher takeovers per rollout M=2 works best; M=4 drops to 44.01 because trajectories drift too far from the student’s own policy. Handoff K=5 beats K=1 (too trigger-happy) and K=10 (misses real divergences).
•
During training, the teacher token share falls from ~13% to 2–3% as the student improves, so the mechanism naturally tapers.
Reach for this when you’re distilling a small reasoning model from a bigger one and your OPD runs are burning most of their token budget on rollouts that went off-rails at paragraph three. Instead of truncating at a fixed length or rewriting offline, watch for the moment the teacher’s argmax lands on a reflection word while the student’s top-5 doesn’t contain one. Let the teacher write a short paragraph of redirection, then hand back. You’ll train on shorter trajectories, converge in fewer steps, and get a model that produces more compact reasoning at inference.
Code is at GitHub with a project page. Built on verl and vLLM. The reflection-token list is small and English-only, so porting to another language or a non-Qwen tokenizer will need adjustment. Training data was the English subset of DAPO-Math-17k.
Distillation failures have a specific fingerprint. Find that fingerprint before you spend gradient on the aftermath. Generic teacher-student disagreement is too noisy a trigger. The moment the teacher wants to say But and the student doesn’t even have But in its top five is a much cleaner signal that the student’s reasoning is about to fossilize a wrong turn, and it costs nothing to detect during normal decoding.
•
The trigger depends on a hand-curated list of English reflection words (But, Wait, However, etc.). Different tokenizers, non-English data, or non-reasoning tasks (code, tool use) would need a new list, and the paper doesn’t test any of those.
•
Requires teacher logit access at every student-generated position, roughly a speculative-decoding-style verification pass alongside training. Not applicable to closed API teachers.
•
Assumes the teacher meaningfully outperforms the student on redirection. As the capability gap closes (strong student, comparable teacher) the mechanism has less to correct; the authors flag this explicitly in their limitations.