Dual-Anchored Policy Distillation fixes a training-time cheat in On-Policy Self-Distillation where the teacher sees the answer key but the student won’t at inference. By adding a self-conditioned bridge distribution so teacher and student see the same information, it cuts wrong-answer hallucinations by 73% late in training and keeps gains at 32B where the baseline flatlines.
Suppose you’re distilling a smaller reasoning model from a stronger one, and to make the teacher’s supervision denser you let it peek at the ground-truth solution while generating token-level targets. Cheap trick, dense signal. The catch: your student learns to output tokens that only make sense if it also had the answer. At inference it doesn’t, so it confabulates. It writes “I recall the answer is 43” mid-derivation and moves on.
This is the Privilege Illusion problem in on-policy self-distillation. The dominant baseline, On-Policy Self-Distillation, is exactly this setup: sample a rollout from the student, then supervise each token using the same model conditioned on the reference solution. Prior fixes try to filter or reweight the privileged teacher signal. This paper argues those fixes miss the point because the teacher and student still see different information.
The core move is to introduce a third distribution that closes the information gap. At each token position, the paper considers three versions of the same policy: one sees only the prompt (None, matches inference), one sees the prompt plus the other completion (Cross, the privileged teacher), and one sees the prompt plus the completion it’s currently predicting (Self). Self is the new piece. It’s trainable like None but information-matched to Cross.
Self becomes a bridge. Instead of asking None to imitate Cross directly (the asymmetric setup that causes the illusion), Dual-Path Anchoring trains along two paths: an unconditioned path that aligns the two None distributions using Self as a proxy, and a privileged path that aligns Self to Cross when both see privileged info. The mechanism relies on shared parameters: nudging Self on reference-side tokens shifts Cross on rollout-side tokens too, because they’re the same network.
Then Dual-Source Anchoring observes that a reference completion isn’t the only useful guidance source. On-policy rollouts, when the student gets stronger, contain their own useful reasoning. So DSA runs the whole DPA construction in both directions: reference guides rollout, and rollout guides reference. A single scalar λ balances them.
for (x, y_star) in batch:
y = student.sample(x) # on-policy rollout
for s, s_bar in [(y, y_star), (y_star, y)]:
p_none = student(x, s_prefix)
p_cross = student(x, s_prefix, s_bar).detach() # privileged teacher
p_self = student(x, s_prefix, s) # bridge, trainable
loss += KL(p_cross, p_none) # original OPSD term
loss += KL(p_none.detach(), p_self) # Inference Anchor
loss += KL(p_cross, p_self) # Privileged Anchor
loss.backward()
All teacher distributions are stop-gradient; the divergence is a component-clipped forward KL over the full vocabulary.
The prevailing fix for privilege illusion is to keep the asymmetric teacher-student setup and clean up the signal afterward: filter tokens, reweight logits, route around bad supervision. This paper shows the opposite. The problem isn’t the teacher’s signal quality, it’s that teacher and student see different information at all. Fix the asymmetry at the source by inserting a trainable distribution that shares the teacher’s view but the student’s parameters. The load-bearing evidence isn’t the headline benchmark lift, it’s the isolated intervention showing that just swapping in the Self distribution (the Privileged Anchor alone) cuts wrong claims by 45% and adds +6.22 Avg@12 points while touching nothing else.
The single most load-bearing finding: swapping the student’s None distribution for the information-matched Self distribution, holding everything else fixed, drops Wrong Claims from 37 to a much lower rate per 10,000 generations and reverses the On-Policy Self-Distillation performance decay. That’s the ablation that proves information asymmetry is the mechanism, not teacher quality.
•
On Qwen3-4B across six benchmarks (math, code, instruction following), DAPD beats OPSD by +2.00 points on average.
•
Scale behavior is the most striking result. OPSD’s lift over the base model collapses from +5.19 at 1.7B to at most +0.28 from 8B through 32B. Larger rollouts are more useful, but the privilege illusion tax grows with them. DAPD holds +2.41, +2.13, +3.06 at 8B, 14B, 32B.
•
Late-stage (steps 250-300) wrong claims drop 73% relative to OPSD.
•
Ablations confirm both paths (unconditioned + privileged) and both directions (reference-guided + rollout-guided) contribute. Optimal λ shifts from 0.5 at 1.7B to 0.2 at 4B+, meaning larger models should trust their own rollouts more.
•
Reference-free variant: replace the ground-truth reference with a second independent rollout. Still beats OPSD by +2.41 at 4B and 8B. Adding a correctness verifier on top helps more.
Reach for this when you’re doing on-policy distillation of a reasoning model and giving your teacher any kind of privileged context the student won’t have at test time. Reference solutions are the obvious case, but the framing extends to retrieved documents, tool traces, verifier feedback, or intermediate plans. The fix is architectural: add a Self distribution that sees what the teacher sees but shares the student’s weights, and train alignment through it rather than across the information gap. The reference-free result matters here: if you have a verifier but no gold references, sampling two independent rollouts and using verification to pick a reliable one recovers most of the gains.
Code is at GitHub. Training uses LoRA adapters, so the compute overhead is bounded even though DAPD constructs multiple distributions per token. Inference cost is identical to the base model. Experiments are on Qwen3 1.7B through 32B trained on OpenThoughts data, evaluated on AIME24/25, HMMT25, LiveCodeBench v5, BFCL v3, and IFBench.
•
Training-time compute goes up meaningfully. Each token now requires forward passes through None, Cross, and Self in both completion directions. Inference cost is unchanged, but if you’re already GPU-bound during distillation, budget for it.
•
The coefficient balance (λ and the per-anchor weights) shifts with scale, and the paper’s sweeps show the optimum moves non-trivially between 1.7B and 32B. Expect to retune when you change model size or architecture.
•
The verified-rollout variant that removes the reference dependency assumes you have an automatic correctness signal. That works for math and code with unit tests; it doesn’t obviously transfer to open-ended generation, dialogue, or tasks where “correct” is fuzzy.