On-Policy Reverse Distillation (OPRD) speeds up training a stronger student model by using a weaker teacher’s direction of improvement, not its policy itself, to amplify only the parts of the student’s own RL with verifiable rewards gradient that already point that way, letting the student surpass the teacher instead of plateauing at it.
Say you just finished expensive RL post-training on a 4B reasoning model, and now you want an 8B model with the same skills plus room to grow. Redoing all that RL from scratch on the bigger model is wasteful. The obvious shortcut, Knowledge Distillation, trains the 8B to imitate the 4B’s outputs. The modern version, On-Policy Distillation, samples responses from the student and asks the teacher for its token probabilities at each step, fixing the classic train/inference mismatch.
The problem: if you tell the 8B “match the 4B’s distribution,” you cap the 8B at the 4B’s ability. The teacher’s final policy blends three things: genuine post-training improvements, quirks inherited from its base model, and limits from being only 4B. Matching all of it drags the student down. A related setting, multi-domain consolidation, is even worse: you have four small specialist teachers (math, logic, code, games) and want one bigger generalist. Matching four different teacher distributions creates conflicts.
The question this paper attacks: how do you get the benefit of the weaker teacher’s post-training without making the teacher’s policy an optimization target?
Core intuition: the teacher’s post-training didn’t just produce a final policy, it produced a change from where it started. That change (call it the teacher’s “policy shift”) points in a useful direction in the space of next-token logits. OPRD uses that direction as a hint, not as a target.
For each token the student generates, OPRD does four things:
1.
Compute the teacher’s logit shift: post-trained teacher logits minus its pre-RL reference logits, mean-centered. Keep only the unit direction, call it d_t. Magnitude is thrown away because a shift learned inside a weak model’s capacity isn’t necessarily well-scaled for a stronger student.
2.
Compute the student’s normal Group Relative Policy Optimization (GRPO) gradient g_t from verifier reward.
3.
Split g_t into a piece along d_t and a piece perpendicular to it.
4.
Multiply the along-d_t piece by (1 + λ). Leave the perpendicular piece alone.
That’s it. The teacher never defines a loss. It only rescales a component that was already in the student’s own verifier-driven gradient.
Two properties fall out. First, if the verifier gradient is zero, the rescaled gradient is zero, so stationary points of RL training don’t move. Second, the rescaled update is guaranteed to make at least as much progress as the plain verifier update, plus a nonnegative bonus proportional to how aligned student and teacher already were.
One subtlety: the alignment coefficient (dot product of g_t and d_t) can be positive (student and teacher agree) or negative (verifier wants the student to depart from the teacher). Amplifying only the positive side works fine early. Amplifying the negative side too aggressively at the start is risky because early student rollouts are noisy, so the paper warms up the negative branch over training.
for token_t in student_rollout:
g_t = advantage_t * grad_logprob(student, y_t, prefix)
delta = center(teacher_logits(prefix) - ref_logits(prefix))
d_t = delta / norm(delta) # unit direction only
u_t = dot(d_t, g_t) # alignment scalar
lam = lam_pos if u_t >= 0 else lam_neg(step) # warmup on neg branch
g_tilde = g_t + lam * u_t * d_t # amplify projection
backprop(g_tilde)
The headline setting is Qwen3-4B teaching Qwen3-8B on math (AIME’24/'25, HMMT’25, OlympiadBench) and on four Reasoning Gym tasks. All methods start from the same student checkpoint, get the same prompts, batch size, and rollout budget.
•
Successive transfer, averaged over 5 checkpoints: OPRD hits 51.9 on math vs 44.0 for the best baseline (KDRL) and 39.4 for plain OPD. On Reasoning Gym, 55.2 vs 44.4. Plain OPD flatlines near the 4B teacher’s score; OPRD blows past it. The paper reports 33-67% fewer updates to hit teacher-level performance vs GRPO.
•
Multi-teacher (four 4B specialists → one 8B): OPRD reaches 58.8 average Pass@1 vs 47.7 for Mix-RL, and beats every individual specialist on its own task. The authors argue the projection-onto-teacher-direction structure reduces cross-task interference, similar in spirit to PCGrad but between student gradient and teacher shift rather than between task gradients.
•
Strong-to-weak also works: an 8B teaching a 1.7B on AIME’24 or a 0.6B on Knights & Knaves. OPRD beats OPD by 3.79 and 29.20 points respectively, showing the method doesn’t require the student to be bigger.
•
Ablation on where the guidance direction comes from: using the teacher-minus-reference shift is faster and more sustained than using an OPD-style teacher-minus-student gradient or a privileged-context self-distillation gradient.
•
Style analysis: on 101 stylistic features (connectives, punctuation, sentence structure), OPRD’s outputs stay closer to a plain GRPO-trained student than to the teacher, while OPD drifts toward teacher style. The authors read this as evidence the teacher is accelerating the student’s own optimization rather than redirecting it.
One clean negative result: on Color Cube, if you use the raw base model as the teacher’s reference, the shift d_t encodes a strong “shorten responses” component (the teacher underwent length collapse during its own RL). OPRD then over-amplifies shortening and plateaus at 52.5% Pass@1, below GRPO. Moving the reference to step 30, after the length collapse, jumps performance to 89.5%. Their diagnosis: the projection step magnifies reward-irrelevant biases shared by student and teacher gradients, and reference choice controls how much of that bias enters d_t.
•
If you’re doing successive-generation post-training and already have a smaller RL-trained checkpoint plus its pre-RL reference, OPRD is a cheap add-on. The paper measures +11.9% wall-clock and +10.2% peak GPU memory vs plain GRPO in their 4B→8B setup, because the frozen 4B teacher and reference are both smaller than the 8B student and only need forward passes.
•
Multi-teacher consolidation is where the story is strongest. If you have domain specialists and want one generalist without cross-task tradeoffs, this is worth testing directly against a mixed-batch RL baseline. The paper shows the student beats every specialist on its own domain.
•
Reference-policy choice is not a detail. If the teacher’s RL run included a sharp length change or other stylistic collapse, the pre-RL checkpoint is a bad reference. Pick a checkpoint from after the collapse. The paper shows this fix on two Reasoning Gym tasks, but doesn’t give a general procedure for detecting when your reference is contaminated.
•
Skip OPRD if the student’s rollouts are mostly invalid. In their 8B→1.7B Knights & Knaves experiment, the small student produces so many failed responses that group-relative advantages vanish, the student gradient goes to zero, and there’s nothing for OPRD to amplify. OPD still works there because it doesn’t depend on verifier reward. The authors suggest a short SFT warm-up before switching to OPRD.
•
Default hyperparameter: λ = 0.5 with the negative branch warmed up over the first 30-75 updates. Performance is flat above λ = 0.5, so precise tuning isn’t important.
•
All experiments are within the Qwen3 family, 0.6B to 8B, on math and logic tasks with clean verifiers. The authors flag code generation, agentic environments, and larger scales as untested.
•
The method needs the teacher’s pre-RL checkpoint, not just its final weights. If you only have the final teacher (common for third-party or API-only models) you cannot compute the shift.
•
The projection-and-amplify step magnifies whatever is shared between student gradient and teacher shift, including reward-irrelevant biases like length preference. The paper’s mitigation (later reference checkpoint) is task-specific and not automated.
•
Gains are reported as checkpoint averages over 5 evenly-spaced snapshots to capture learning speed, not just endpoint accuracy. The absolute endpoint gap over KDRL is smaller than the averaged gap on some tasks, so “OPRD wins” is partly a statement about sample efficiency, not just final score.
•
Comparisons to concurrent methods that also use the teacher’s policy shift (Direct-OPD, W2S-OPD) are done with hyperparameters from the original papers, not retuned for this setting.