TGOPD checks whether a frozen teacher can actually solve each training prompt before letting its per-token gradients touch the student. When the teacher fails a 3-sample verifier audit, the update falls back to Group Relative Policy Optimization (GRPO), lifting otherwise-idle teacher GPUs from 9.8% to 78.9% utilization.
Suppose you’re distilling a small coding model from a stronger one. The usual recipe is On-Policy Distillation: sample rollouts from the student, then have the teacher score every token so the student is pulled toward the teacher’s distribution. It’s much more sample-efficient than learning from a single pass/fail reward per rollout.
But this only works if the teacher is right. Reverse KL divergence is mode-seeking, so it concentrates the student on the teacher’s most confident behavior. On prompts where the teacher is confidently wrong (very common in code), you dutifully train the student to imitate wrong answers. Existing fixes look at token-level entropy or teacher-student disagreement, which measure uncertainty but not correctness.
The intuition: before you trust the teacher’s dense per-token signal on a prompt, make the teacher actually try the prompt a few times and grade its attempts with the same verifier you use for the student.
Concretely, for each prompt the frozen teacher generates K_T=3 full rollouts. A verifier (unit tests for code, rule-based judge for math and instruction-following) scores them. Their pass rate q_T(x) is the teacher’s estimated reliability on this specific prompt. If at least 2 of 3 pass (threshold τ=2/3), the gate opens: the update uses standard OPD with the teacher’s log-prob gap as the per-token advantage. If the gate closes, the teacher signal is dropped entirely and the update uses Group Relative Policy Optimization (GRPO) on the student’s own rollouts, which gives a group-centered verifier reward. The two signals are never blended on the same prompt.
The systems trick that makes this cheap: in asynchronous OPD, the teacher node normally sits idle while the student decodes, because teacher scoring is just a forward pass over already-generated tokens. TGOPD launches the K_T probe generations during that idle window.
for x in batch:
# runs concurrently with student rollout
probes = [teacher.generate(x) for _ in range(K_T)]
q_T = mean(verifier(x, p) for p in probes)
rollouts = student.generate(x, G=4)
if q_T >= tau: # audit passed
A = opd_advantage(rollouts, teacher)
else: # audit failed
A = grpo_advantage(rollouts, verifier)
update(student, rollouts, A)
The prevailing reliability-aware distillation approach is to read the teacher’s distribution: entropy, agreement with the student, likelihood ratios. This paper shows the opposite. Confidence is not correctness. Make the teacher take the test, grade it with the same verifier the student sees, and only then decide whether its dense gradient is trustworthy. The load-bearing evidence is a diagnostic showing that on low-reliability code prompts, the teacher’s most confident sampled answer is wrong 84% of the time (AUROC of confidence-vs-correctness is a near-random 0.51 on code).
The diagnostic drives everything else: on code, teacher self-confidence carries almost no information about whether the teacher is actually right (AUROC 0.51, vs 0.73 on math). This is exactly where mode-seeking Reverse KL divergence does the most damage, and exactly where TGOPD helps most.
•
Code, 35B student on LiveCodeBench: every other distillation method causes negative transfer, i.e. the distilled student scores below the untrained base (Vanilla OPD −0.8, TrOPD −2.5, RG-OPD −3.5, an RLSD (Self-Distilled RLVR)-style baseline −4.1). TGOPD is the only method with positive transfer (+3.0 over base) and it beats the teacher itself by +1.3.
•
Across all six domain×scale settings (4B and 35B students, on math, code, instruction-following), TGOPD beats Vanilla OPD. Largest average gains are on code (+3.0 / +2.9), then IF (+1.6 / +1.9), then math (+1.5 / +1.2), matching the confidence-vs-correctness diagnostic.
•
Compute: teacher-node GPU utilization rises from 9.8% to 78.9% (single-domain 4B). A matched 35B code run shows the audit is not strictly free: mean step time increases by 5.9%.
•
Threshold ablation (K_T=5) shows an inverted-U with a broad peak around a simple majority; a strict all-must-pass threshold discards useful signal and underperforms Vanilla OPD.
•
Closed-gate policy ablation: simply masking failed-audit prompts recovers roughly 90% of TGOPD’s gain. The Group Relative Policy Optimization (GRPO) fallback is the better default but only narrowly. Most of the win comes from blocking bad teacher signal, not from what replaces it.
Reach for this when you’re distilling a stronger model into a smaller one for a domain where you already have a programmatic verifier (unit tests, a math grader, a format checker). Instead of trusting the teacher on every prompt, spend a few extra teacher samples per prompt to audit it; drop the dense KL signal on prompts it fails and fall back to a verifier-only update. The finding that ~90% of the benefit comes from masking alone is the practical takeaway: even without a full Group Relative Policy Optimization (GRPO) path, just refusing to distill on prompts the teacher can’t solve is most of the win.
The paper doesn’t mention a code release; experiments run on the slime asynchronous RL framework. Training data is DAPO-Math-17k for math, CodeI/O input-output prediction for code, and filtered Nemotron-Cascade 2 for instruction-following. Students are Qwen3.5-4B and Qwen3.6-35B-A3B; teachers are the same base models trained with Group Relative Policy Optimization (GRPO) per domain.
Confidence is not correctness. If you have a verifier, make the teacher take the test before you distill from it.
•
Requires an automatic verifier per domain. The whole method degenerates on open-ended tasks where you can’t cheaply grade a teacher rollout, and the authors flag this as future work.
•
The gate is binary and the fallback margin over plain masking is small (0.15–0.60 points). If you can’t afford K_T=3 extra teacher generations per prompt, most of the benefit is available from a simpler “skip prompts where the teacher fails” rule.
•
The 78.9% utilization figure depends on asynchronous OPD leaving the teacher idle. In a synchronous or already-saturated pipeline, probes stop being free and the measured 5.9% step-time overhead on the 35B code run is the more honest number to plan around.