When distilling a diffusion teacher that uses classifier-free guidance, matching only the guided output lets positive- and negative-branch errors cancel. This paper supervises the positive branch and the conditional direction separately, restoring stable student behavior across inference guidance scales.
You’ve shipped a video generation product where a big teacher model sees dense per-frame control (a pose skeleton on every frame) and you’re distilling it into a cheaper student that only gets a few keyframes. Users then twist the Classifier-free guidance slider at inference to trade prompt adherence for fidelity. If your distilled student was only trained at one guidance value, moving that slider can wreck the output, even though the student looked fine during training.
The dominant recipe here is On-Policy Distillation for diffusion: the student rolls out its own denoising trajectory, and the teacher is queried at those same states to provide per-step targets. Prior branch-retaining variants like DiffusionOPD just match the final CFG-composed velocity. The paper argues that objective is quietly broken whenever the teacher’s negative branch sees information the student doesn’t.
CFG evaluates the model twice per step: once with the target prompt (positive branch) and once with a null or negative prompt (negative branch), then mixes them as v_guided = γ·v+ + (1−γ)·v−. The naive distillation loss matches teacher and student on v_guided. The problem: only the weighted sum of the two branch errors is constrained. The positive-branch error can shrink while the negative-branch error grows, and the composed prediction still looks fine at the training γ.
The authors call this failure Negative Branch Asymmetry. It shows up specifically when the teacher’s negative branch carries privileged inputs the student lacks. Example: the teacher sees a reference image on both branches; the student is text-only on both. The teacher’s v− encodes something the student’s v− structurally cannot, so pushing the composed prediction to match forces compensation across branches.
The fix, Positive–Direction Matching, splits the objective into two terms: match the positive prediction directly, and separately match the CFG conditional direction d = v+ − v−. Both errors being zero is now the only zero-loss solution, so branches cannot trade off against each other. A weight λ controls the direction term; they use λ=1. They also include Independent Branch Matching as a foil, which just matches v+ and v− independently with the naive weights.
# per student-visited state x_t, timestep t
vT_pos = teacher(x_t, cond_pos) # teacher w/ prompt (+ ref if privileged)
vT_neg = teacher(x_t, cond_neg) # teacher null branch (may keep ref)
vS_pos = student(x_t, cond_pos)
vS_neg = student(x_t, cond_neg)
e_pos = vT_pos - vS_pos
d_T, d_S = vT_pos - vT_neg, vS_pos - vS_neg # CFG conditional directions
loss = ||e_pos||^2 + lam * ||d_T - d_S||^2 # PDM
For the video setting they also supervise only the first K=8 of 50 denoising steps to keep training tractable (a full-trajectory step takes ~790s vs ~150s at K=8).
The prevailing move when distilling a CFG-using teacher is to treat the guided velocity as the training target, because that’s what the model actually denoises with at inference. This paper shows the opposite: matching the composed prediction is under-determined, and you have to supervise the positive branch and the conditional direction before CFG mixes them. The load-bearing evidence is the branch-error tracking under privileged conditioning: naive matching drives the positive-branch error down while the negative-branch error climbs, which is invisible at training γ but explodes when the inference slider moves.
•
The diagnostic experiment is the load-bearing result. In text-rendering distillation, where teacher and student share null-text on the negative branch, positive-branch updates also reduce the negative-branch error and naive matching is fine. In reference-conditioned distillation, where the teacher’s negative branch keeps the reference image, naive matching drives positive-branch error down and negative-branch error up over training. PDM keeps both errors falling.
•
Guidance-scale robustness. For pose-controlled video at train γ=5, evaluated across other inference γ values, naive OPD degrades sharply (worst at γ=1) while PDM and IBM stay stable across the sweep.
•
Matched-scale quality. At γ=5, jointly trained across pose, depth, and scribble, PDM beats naive OPD, SFT, and off-policy distillation on control fidelity across all three modalities. IBM is second-best on most metrics, which the authors read as evidence that removing cross-branch compensation is what matters.
•
Text-rendering sanity check. OCR reward at γ=1 is 75.24 (teacher), 73.87 (naive), 74.48 (PDM). All three track each other across the guidance sweep, confirming that naive matching only breaks under privileged negative conditioning, not universally.
•
Ablations. λ=1 gives best control accuracy; K=8 hits the sweet spot on the cost/fidelity curve.
Reach for this when you’re distilling a control-rich diffusion teacher into a lighter student that will be shipped with a user-facing guidance scale, and the teacher’s negative branch sees something the student doesn’t. Concretely: dense-to-sparse control (dense pose to keyframes), reference-image teachers distilled into text-only students, or any setup where the negative-branch conditioning of teacher and student differs. Replace your composed-velocity loss with the positive prediction MSE plus a λ=1 term on the difference of v+ − v− between teacher and student. Keep everything else the same.
The project page is rethinking-cfg-opd.github.io. The paper doesn’t specify a released code repo or dataset artifact; the video benchmark is described as filtered clips from OpenHumanVid with a 600-clip stratified test set, but the split itself isn’t stated to be released. The main model touched is Wan-VACE for the video experiments, with Stable Diffusion 3.5-Medium and FLUX.2-klein-4B used in the image-domain diagnostics.
When distilling a guided teacher, supervise the branches, not the sum. The composed velocity is what the model runs at inference, but it’s the wrong training target the moment teacher and student don’t see the same thing on the negative branch. Match v+ and the direction v+ − v− separately, or user-side guidance-scale tweaks will surface distortions you never saw during training.
•
The video results are on one base model (Wan-VACE 1.3B) with LoRA adapters at one training guidance value (γ=5). Generalization to full fine-tuning, larger backbones, or teachers with very different CFG schemas isn’t tested.
•
PDM and IBM share the same zero-loss solution, so PDM’s edge over IBM is empirical, not theoretical. The authors say so directly. If your setup weights branches differently, IBM might match or beat PDM.
•
The mechanism only bites under privileged negative conditioning. If teacher and student share null-text on the negative branch (the standard text-to-image case), naive composed matching is fine and PDM buys you little. Diagnose your setup before switching objectives.