Road MOPD trains one student from several domain-specialist teachers by having each teacher also score the other domains’ trajectories, then uses that agreement to reweight tokens and project away conflicting gradient components, recovering 5.6–8.8 points of medical accuracy that vanilla multi-teacher distillation loses.
You want one small open model that is good at math, code, and medicine. The standard recipe is multi-teacher On-Policy Distillation: for each training prompt, route it to the specialist teacher for that domain and have that teacher grade the student’s tokens densely. Math prompts go to the math teacher, medical prompts to the medical teacher, and so on. The student updates on all of them.
This works when the domains are friendly. The paper shows that jointly training math and code mostly preserves single-domain performance. But adding medicine to either math or code consistently drags medical accuracy below what you’d get training on medicine alone. The authors measure this directly: a large fraction of layers show conflicting gradients between the medical update and the math or code update, and the fraction grows over training. Routing decides who supervises what, but every teacher’s update still lands on the same shared weights. Domain routing alone doesn’t coordinate those updates.
The named baseline the paper leans on is Vanilla MOPD MOPD, the straightforward route-and-distill setup.
The core idea: when you’re training on a medical prompt, don’t only ask the medical teacher what the student should have said. Also ask the math and code teachers to score the same student trajectory. Their reactions are a cheap signal of whether this update will clash with those other capabilities.
Road MOPD uses that extra signal in two places.
First, token-level credit assignment. For each generated token, compare the routed teacher’s log-probability to a frozen reference model, and do the same for each auxiliary teacher. If both the routed and auxiliary teachers push in the same direction relative to the reference, that token is probably a transferable reasoning move (say, a logical connective useful across domains), so amplify its training signal. If they disagree, this token likely encodes a domain-specific correction that would hurt the other capability, so attenuate it. Concretely, the per-token reward multiplier sits in a clipped range around 1, moving up on agreement and down on disagreement.
Second, preventive gradient projection. Token-level reweighting doesn’t guarantee the final parameter update is compatible, because gradients from many tokens get summed. So they also compute an auxiliary gradient: what gradient would the auxiliary teacher’s supervision produce on this same trajectory? For each parameter tensor, if the target-domain gradient and this auxiliary gradient point in opposing directions (negative dot product), project the target gradient to remove the conflicting component. A norm-retention floor (set to 0.6 in experiments) caps how much of the original target gradient’s magnitude you can erase, so you never fully sacrifice the lesson you were trying to teach. The auxiliary gradient is used only as a constraint. It never becomes its own optimizer step.
This is different from PCGrad, which compares gradients computed on each domain’s own data. Road MOPD’s gradients are computed on the same trajectory with different teachers, so the comparison probes compatibility at the current behavior rather than between separately-sourced tasks.
for each training step:
rollouts = student.generate(prompts) # on-policy
for d in domains:
# score the same tokens with every teacher + frozen reference
logp_routed, logp_aux[...], logp_ref = score(rollouts[d])
agree = tanh((logp_routed - logp_ref)/tau) \
* tanh((logp_aux - logp_ref)/tau)
lam_t = clip(1 + rho * agree, 0.5, 1.5) # per-token weight
g_target = grad(ppo_loss(routed, advantage=lam_t * ...))
g_aux = grad(ppo_loss(aux_teacher, same_rollouts))
g_tilde = project_away_conflict(g_target, g_aux, floor=0.6)
update = mean_over_domains(g_tilde); adamw_step(update)
Students are Qwen3-4B Qwen3 and Qwen3-1.7B with thinking disabled. Teachers are specialist Qwen3-4B checkpoints for math and code and II-Medical-8B II-Medical-8B for medicine. Evaluation spans 11 benchmarks across the three domains, including HumanEval+ HumanEval+, LiveCodeBench LiveCodeBench, AIME AIME 2024/2025, HMMT 2025, MedQA MedQA, MedXpertQA MedXpertQA, MedMCQA MedMCQA, and SuperGPQA-Med.
The headline result is medical recovery under high conflict. On Qwen3-4B Math × Medical and Code × Medical, Vanilla MOPD actually drops medical accuracy below the untrained student’s 41.01. Road MOPD reaches medical averages of 48.23 and 47.75 respectively, roughly matching the single-domain OPD reference of 47.90, while also posting the highest math or code average in each pairing. In other words, fixing medicine didn’t cost the other domain. On the smaller Qwen3-1.7B student, where single-domain medical training itself tops out lower, Road MOPD beats the strongest multi-domain medical baseline by 3.88 and 5.17 points.
In the already-friendly Math × Code setting, Road MOPD still leads the two-domain average, mostly through a code gain. In three-domain training (math + code + medicine) it leads every domain and beats the next-best multi-domain method by 1.18 points on average.
Two supporting observations. First, teacher disagreement correlates with raw gradient conflict: binning batches by disagreement, the fraction with conflicting gradients climbs from 24% to 40% in math and 46% to 76% in medicine. This is correlational, not causal, but it’s what the token-reweighting scheme was banking on. Second, an ablation removing either the token weighting or the gradient projection degrades results in both domains, and replacing the adaptive norm floor with full projection also hurts. Each piece pulls weight.
If you’re building a general-purpose small model by distilling from multiple specialist teachers, the practical takeaway is that domain routing isn’t enough when domain styles differ a lot. Medicine distilled from an RL-tuned medical model collides with math and code distillation in ways that pure math-plus-code training doesn’t. The paper’s diagnostic (track the fraction of layers with conflicting gradients between your domain pairs over training) is cheap and worth running before you commit to a joint-training recipe.
If you see persistent conflicts, Road MOPD’s recipe is plausible to adapt. The ingredients you need: compatible tokenizers across teachers, access to teacher log-probabilities (not just generations), a frozen reference model (the student’s initialization works), and roughly K× the scoring cost per step for K teachers. The gradient projection adds a second backward per auxiliary teacher per step, which the authors flag as the main overhead. For a student that already has stable PPO-style distillation infrastructure, bolting this on is more an engineering tax than a research project.
If you can’t afford both pieces, the ablation suggests token-level credit assignment is doing more of the work in the compatible-domain direction, while gradient projection is what protects the disadvantaged domain under high conflict. Pick based on which failure you have.
What’s not established: that this scales past three domains, that it helps for non-reasoning domains (chat, retrieval, multilingual), or that cheaper proxies for the auxiliary gradient would work. The paper tests three text-reasoning domains with Qwen3-family teachers sharing a tokenizer. The paper doesn’t mention a code release.
The evaluation covers math, code, and medicine only, with at most three joint domains. The authors are explicit that broader domain mixes and larger teacher pools remain untested, and that efficient teacher selection becomes the open problem at scale.
The method requires teachers whose tokenizers match the student’s, so you can score identical token IDs at identical prefixes. Mixing teachers from different model families (say, Llama and Qwen) isn’t addressed and would need a tokenizer-agnostic variant.
The gradient-projection guarantees are local: they bound the auxiliary loss change for one tensor-level step before clipping and AdamW preconditioning. They don’t promise anything about the final optimizer step or auxiliary-domain performance after training. The medical gains are the empirical evidence that the local constraint translates to useful behavior, not a theorem.
Finally, the correlation between teacher disagreement and gradient conflict is suggestive but exploratory. It supports the design but doesn’t prove disagreement causes the gradient clashes that hurt performance.