u-OPSD trains a reasoning model on its own outputs by using majority-vote agreement across sampled answers as a stand-in for a gold solution, then distilling the vote-conditioned teacher into the disagreeing rollouts. It matches or beats supervised On-Policy Self-Distillation with no labels, lifting Qwen3-4B/8B non-thinking by 8.5–10.7% on math.
You’ve fine-tuned a small model for a reasoning task, say extracting structured decisions from documents. You have a pile of unlabeled prompts and no budget to hand-label solutions. The dominant recipes need something external: RL with verifiable rewards needs a gold answer to verify against, on-policy distillation needs a stronger teacher model, and On-Policy Self-Distillation needs a written reference solution to condition its “teacher-self” on. All three cap your improvement at what you can pay to label or what a bigger model already knows.
This paper asks whether the privileged signal a self-teacher needs can be manufactured from the model’s own samples. If yes, unlabeled prompts become training data with no annotator and no larger model in the loop.
Start with the idea the paper is copying from and mutating. In On-Policy Self-Distillation, one model plays two roles: a “student” that sees only the problem, and a “teacher” (same weights, gradients off) that sees the problem plus the gold written solution. The teacher’s next-token distribution is richer because it’s peeking at the answer. You distill that richer distribution into the student along the student’s own rollouts, giving dense token-level supervision instead of a sparse pass/fail reward. The catch: you need the gold solution.
u-OPSD replaces the gold solution with a pseudo-solution built from the model itself. For each unlabeled prompt:
1.
Sample G=8 independent rollouts at training temperature.
2.
Extract the boxed final answer from each. Take the majority vote as the pseudo-answer. If fewer than half the rollouts agree (threshold τ=0.5), skip this prompt.
3.
Partition rollouts into agreeing (majority answer) and disagreeing sets.
4.
Pick the longest agreeing rollout as the pseudo-solution y+. This becomes the teacher’s privileged context.
5.
For each disagreeing rollout y-, walk its prefixes and minimize forward KL between the teacher’s next-token distribution (conditioned on prompt + y+ + prefix of y-) and the student’s (conditioned on prompt + prefix of y- only).
rollouts = [sample(pi, x) for _ in range(G)]
answers = [parse_boxed(r) for r in rollouts]
pseudo_ans = majority_vote(answers)
if vote_fraction(pseudo_ans, answers) < tau: skip
agree = [r for r,a in zip(rollouts,answers) if a==pseudo_ans]
disagree = [r for r,a in zip(rollouts,answers) if a!=pseudo_ans and a is not None]
if not disagree: skip
y_plus = longest(agree)
for y_minus in disagree:
for t in range(len(y_minus)):
loss += KL(teacher(x, y_plus, y_minus[:t]), student(x, y_minus[:t]))
Two skip conditions matter: no confident majority (vote isn’t trustworthy), or unanimous agreement (nothing to correct). This carves out a natural curriculum on the model’s competence frontier, problems it can sometimes get right.
The prevailing label-free approach is to turn model self-agreement into a scalar reward and run RL, treating consensus like a noisy verifier (TTRL (Test-Time Reinforcement Learning) and self-certainty methods do this). This paper shows the opposite. Consensus is far more useful as conditioning context for dense token-level distillation than as a scalar reward for policy gradient. The majority answer isn’t a grade; it’s a stand-in for the reference solution the teacher-self peeks at. The load-bearing evidence is the gap versus label-free RL baselines run under the same rollout budget, not the headline lift over the base model.
Against label-free RL baselines matched on rollout budget (TTRL (Test-Time Reinforcement Learning), RENT, Intuitor), u-OPSD wins by 7.0–11.3% on average in non-thinking mode and 0.8–1.4% in thinking mode. That gap is the one that isolates the mechanism: same signal (self-agreement), same compute, but used as distillation context instead of a scalar reward. Secondary findings:
•
On five math benchmarks (AIME 2024, AIME 2025, HMMT25, MATH500, AMC 2023), u-OPSD beats the base Qwen3-4B/8B by 8.5% and 10.7% in non-thinking mode, and beats supervised On-Policy Self-Distillation (which uses gold solutions) by 3.2% and 2.3%.
•
In thinking mode the base models are already at ~75%, so headroom is thinner: u-OPSD gains 1.9–2.2% over base and roughly ties supervised OPSD.
•
Ablation on what the teacher conditions on: stripping the pseudo-solution down to just the final boxed answer costs 10.3–15.8% and drops below the untrained base. The full reasoning trace, not the answer, is what makes the teacher’s distribution informative.
•
Pseudo-label quality probe: 86.7% of pseudo-labels match the true gold answer, so ~13% of the training signal is wrong yet it still beats gold-supervised OPSD.
Reach for this when you have a domain with a parseable, canonicalizable answer (math, code that either passes tests or doesn’t, extraction with exact-match) and a pile of unlabeled prompts. Concretely: you have a Qwen-class model that gets a task right maybe 40–60% of the time. Sample 8 rollouts per prompt, majority-vote the answer, and distill the vote-conditioned teacher into the losers. No annotators, no bigger model, no verifier beyond “can I parse and canonicalize this answer.”
Code is on GitHub with the training recipe. It builds on the OPSD codebase and uses LoRA rank 64, so a single-node setup is realistic. Training was 150 steps on a 30k-prompt subset of OpenThoughts, evaluated on Qwen3-4B, 8B, 30B-A3B-Instruct, and 4B-Instruct. The recipe transferred to the 30B mixture-of-experts model with no hyperparameter tuning.
When you have unlabeled prompts and a checkable answer format, feed the model’s own majority-vote solution back in as teacher context, not as a reward. This only pays when the base model is strong enough to vote correctly most of the time and weak enough to still be wrong on individual rollouts. On models already near ceiling, or on tasks where you can’t canonicalize the answer for voting, the mechanism has nothing to grip.
•
Requires a checkable, canonicalizable answer. The whole method depends on exact-match voting over parsed final answers. Open-ended generation, summarization, or dialogue would need a softer notion of consensus that the paper doesn’t provide.
•
The advantage collapses in the thinking-mode regime. With Qwen3 thinking mode already at ~75% base accuracy, u-OPSD ties supervised OPSD rather than beating it. Read the headline gains as “non-thinking mode” gains; the paper is explicit about this.
•
Bounded by whichever wrong answer the base model prefers. ~13% of pseudo-labels were wrong in-domain, and the authors didn’t stress-test what happens when the model is confidently wrong more often. No mechanism yet for detecting or down-weighting low-quality consensus beyond a global vote-fraction threshold.