For On-Policy Distillation across models with different tokenizers, restricting the student-teacher loss to a tiny top-16 slice of the shared vocabulary at cleanly-aligned positions matches or beats methods that try to supervise every position, because filling the coverage gap injects weakly-aligned gradients.
You want to distill knowledge from a strong teacher LLM (say Qwen) into a smaller student from a different family (say Llama). The modern recipe is On-Policy Distillation: let the student generate a response, then nudge its next-token distribution toward the teacher’s at each position. This works cleanly when both models share a tokenizer, because “position i” and “vocabulary entry w” mean the same thing on both sides.
With different tokenizers, neither assumption holds. The same text gets chopped into different tokens, and the vocabularies only partially overlap. A line of recent work (Universal Logit Distillation (ULD), SimCT, Byte-Prefix Marginalization, and others) treats this as a coverage problem: recover supervision on as many response positions and vocabulary entries as possible, through sorted-logit matching, byte-level mapping, or grouping multiple tokens into aligned spans. The assumption is that more supervision coverage means more learning signal. This paper challenges that assumption.
Start with a simple baseline: tokenize the student’s response with both tokenizers, then find strict positions, places where exactly one student token and one teacher token cover the same text span. At those positions, restrict both predicted distributions to the shared vocabulary and apply reverse KL. Everything else (positions where tokenizations disagree, call these mismatch groups) gets dropped.
The authors measure what this throws away and find, surprisingly, very little:
•
Even when static vocabulary overlap is as low as ~40%, strict positions still cover 85-97% of student-generated tokens.
•
At those strict positions, the shared vocabulary carries >99% of each model’s predicted probability mass on average.
•
A student-selected top-16 subset of the shared vocabulary (the 16 entries the student assigns highest probability to, then evaluated on both sides) still carries >93% of teacher mass.
So they test two directions. First, shrink supervision further: use only top-16 shared-vocab entries at strict positions. Second, expand supervision: add a mean-squared-error loss on the log-probability of each full mismatch span, giving 100% structural coverage. Pseudocode for one training step:
resp = student.sample(prompt)
groups = align_token_groups(resp, student_tok, teacher_tok)
loss = 0
for g in groups:
if g.is_strict_1to1:
p_s, p_t = restrict_to_shared_vocab(student(g), teacher(g))
topk = top_k_indices(p_s, k=16) # student picks support
loss += reverse_kl(p_s[topk], p_t[topk])
elif use_span_mse: # ablation only
loss += (log_prod(student, g) - log_prod(teacher, g))**2
loss.backward()
To diagnose why adding span MSE hurts, they measure the cosine between the strict-loss gradient and the span-loss gradient at saved checkpoints, and the ratio of their norms.
Across three teacher-student pairs (Qwen2.5-7B\u2192Llama-3.2-3B, Granite-4.1-8B\u2192Phi-4-mini, Granite-4.1-8B\u2192Qwen2.5-7B) trained on math (DAPO-Math-17k) and code (CodeForces (open-r1)) and evaluated on MATH500, GSM8K, AIME, HumanEval, MBPP, LiveCodeBench:
•
Top-16 strict matches full shared-vocab strict. The compact version keeps at least 96% of the full-average accuracy gain over the undistilled student, and both beat the four cross-tokenizer baselines (Universal Logit Distillation (ULD), Extended ULD, GOLD, SimCT) on every pair’s full average.
•
Adding span MSE always hurts. Across 18 positive-weight settings (three pairs \u00d7 six weights from 0.25 to 1.5), every one scored 0.27 to 1.20 percentage points below the strict-only baseline on the full average. Coverage went up, accuracy went down.
•
The gradient diagnostic lines up with the accuracy drop. The cosine between span-loss and strict-loss gradients hovers near zero (sometimes negative), while a control that splits strict positions into two halves and compares their gradients stays substantially higher. Meanwhile, the span-to-strict gradient norm ratio grows over training, meaning a fixed-weight span loss would exert more and more pull in a direction weakly correlated with the strict objective. This is consistent with (though not proof of) the span signal actively interfering.
•
A larger-teacher setup (Qwen3-235B\u2192Granite-8B) and an agent benchmark (ALFWorld) show the same pattern: top-16 strict wins.
•
If you’re already doing cross-tokenizer OPD and reaching for span-level or byte-level recovery methods to “cover more positions”, worth testing a strict-position + top-16 shared-vocab baseline first. The paper’s evidence is that this simple setup is competitive with or beats the more elaborate options on math and code reasoning benchmarks.
•
The top-k support is student-selected, meaning you pick the 16 vocabulary entries the student currently ranks highest, then score both models on that same set. Picking by teacher scores or a static set would be a different method and isn’t what was tested here.
•
The gradient-cosine diagnostic (does my auxiliary loss point in roughly the same direction as my main loss, relative to a same-objective split-half control?) is a reusable tool for any multi-term training objective. If a new auxiliary term’s gradient is less aligned with the main loss than two halves of the main loss are with each other, that’s a warning sign worth checking before scaling the weight.
•
The reasoning (strict positions already carry most of the mass, so recovered supervision is marginal and possibly noisy) is specific to settings where tokenizations align often. If you’re distilling across languages or modalities where strict alignment is rare, the trade-off could flip. Worth re-measuring strict coverage on your own data before transferring the recommendation.
•
Code is at the anonymous repo linked in the paper.
•
The evaluation is on math and code reasoning with instruction-tuned or base students in the 3B-8B range (plus one 235B\u21928B pair). Chat, long-form generation, multilingual, or multimodal distillation could behave differently.
•
The span supervision tested is a specific choice: MSE on the log of the product of token probabilities in each mismatch span. Other mismatch-group objectives (e.g. byte-prefix marginalization applied carefully, or the span representation alignment from the related work) are not re-run under the authors’ setup, so the claim is “this particular span loss hurts”, not “all mismatch supervision is harmful”.
•
The gradient diagnostic shows weak directional agreement and growing relative magnitude, which is consistent with the accuracy drop but is not a causal demonstration; the authors are careful to say it “may help explain” the result.
•
Hyperparameters, teacher/student temperatures, and KD weight are fixed at defaults from the paper’s training recipe. Baseline methods are run through KDFlow with those same defaults, which may not match each baseline’s originally-tuned configuration.