W2S-OPD improves a strong student model using only weaker models as teachers, by subtracting a weak pair’s logits to isolate a capability direction and adding it onto the student’s own base, letting the student surpass its 4B teacher on math reasoning.
You’re fine-tuning your best in-house 8B reasoning model. The usual playbook for On-Policy Distillation is to score the student’s own rollouts with a bigger, smarter teacher. But once your student is the strongest thing you have, there’s no bigger teacher to call. The alternative, training several domain experts at 8B and merging them (as in MOPD), costs real GPU-months.
This paper asks a different question: can a bunch of weaker models, ones that already exist or can be RL-trained cheaply at 4B, still supervise the 8B student? Prior Weak-to-strong generalization work argued yes in principle. W2S-OPD gives a concrete on-policy recipe.
The trick is to never distill toward the weak model directly. Instead, pick two weak models where one is better than the other at the target skill: call them positive and negative. At each token, look at the difference between their raw output scores (logits). What they agree on cancels; what remains points in the direction of the extra capability the positive model has.
Now add that direction, scaled by a knob α, onto the student’s own base model logits. Softmax the result. That synthetic distribution is the proxy teacher: it carries the isolated skill but stays close to distributions the student already produces. The student generates rollouts, and at every position minimizes reverse KL divergence to this proxy teacher.
The paper proposes three ways to build the contrast pair, all using Qwen3 models: (i) a 4B model after Group Relative Policy Optimization (GRPO) RL training vs. the same 4B before RL, isolating the RL-instilled skill; (ii) an off-the-shelf 4B vs. 0.6B, isolating what scale alone buys; (iii) one 4B model conditioned on a correct hint vs. the same model conditioned on a wrong hint, isolating a per-instance nudge toward the right answer. All three are cheaper than training an 8B expert.
for prompt in dataset:
rollout = student.sample(prompt)
for t, prefix in enumerate(rollout):
z_base = student_base.logits(prefix)
z_pos = positive_model.logits(prefix)
z_neg = negative_model.logits(prefix)
teacher = softmax(z_base + alpha * (z_pos - z_neg))
loss += kl(student.dist(prefix), teacher) # reverse KL
loss.backward()
Multiple contrast pairs compose additively: sum the α-weighted differences before the softmax and one distillation run merges several skills.
The default assumption in distillation is that supervision has to come from somewhere at least as good as you. This paper shows the opposite. What transfers is not the teacher’s competence but the direction between two weaker models. Absolute skill level cancels out; only the gap matters. The load-bearing evidence is that the 8B student beats its own 4B post-RL teacher on math, and still improves even when both contrast models are strictly weaker than it.
The finding that makes the thesis credible: in the smaller/larger setting, where both the 4B and 0.6B base models are weaker than the 8B student and no RL training happens at all, W2S-OPD still lifts the student +6.0% absolute on math and +1.2% on code averages. Every supervision source is worse than the student, yet the student improves. That’s the mechanism talking, not teacher quality bleeding through.
•
Pre-RL / Post-RL contrast: W2S-OPD beats standard OPD by 11.4% relative on math and 3.7% on code (single-teacher). The 8B student surpasses the 4B math expert it was distilling from, while plain OPD stays below that expert.
•
Contrastive hints (single 4B model, correct vs. wrong hint): +1.4% math, +1.1% code over the student base.
•
Out-of-domain: trained only on math, W2S-OPD lifts GPQA-Diamond from 38.9 to 56.5 and still improves IFBench, whereas OPD degrades IFBench below the untrained student, meaning direct distillation absorbs the small expert’s limitations while the directional version does not.
•
α is a soft knob: too small and there’s no signal, too large and the proxy teacher drifts from the student’s distribution and hurts. A wide moderate range beats OPD.
•
Token-level analysis using Schoenfeld episodes shows the three contrast types reinforce different reasoning phases: post-RL and hint contrasts strengthen planning and monitoring tokens; the scale contrast strengthens the analyze/implement steps.
Runtime overhead is roughly 20% per step over OPD, since three frozen models are scored instead of one.
Reach for this when you’re post-training the strongest reasoning model you own and the leaderboard-topping open teacher no longer clears your bar. Instead of paying to RL-train a same-size domain expert, RL-train a much smaller one (say 4B), keep its pre-RL checkpoint, and use the pair as a contrast. Your student rolls out as usual; the proxy teacher scores those rollouts with the smaller pair’s logit difference added onto your student’s own base. The dispatch pattern extends naturally to multiple domains: route each prompt to its matching pair, sum the directions, one training run.
The authors promise code at GitHub. All experiments use Qwen3-8B (non-thinking mode) as student with 4B/0.6B variants as contrasts, trained on top of the verl framework. No new datasets are released; training uses public math (DeepMath-103k) and code (Eurus-RL-Code) corpora. If you already have any pair of same-family models where one is stronger, you have a proxy teacher for free.
When you’ve outgrown your teachers, distill the gap between two weak models, not the models themselves. The absolute skill of the supervisor is a red herring; what carries information is the direction from worse to better, and that direction transfers across scale as long as the proxy teacher stays anchored to something the student already sounds like.
•
All results are within one model family (Qwen3). Whether the logit-arithmetic trick works when the contrast pair and the student come from different tokenizers or pretraining recipes is untested; the method literally requires adding logits over a shared vocabulary.
•
The gains scale with the gap inside the contrast pair. A 4B vs. 1.7B pair helps less than 4B vs. 0.6B. If you only have models that are close in capability, the signal shrinks.
•
The α coefficient needs tuning per setting and has a non-monotone sweet spot. Too aggressive and the proxy teacher drifts off the student’s distribution, at which point performance dips below plain OPD.