VCSD teaches a vision-language model to lean harder on the actual image by having the model’s own Exponential moving average copy score each next token twice, once with the real image and once with a blank black image, then distilling the difference. On a 7-benchmark average, this lifts Qwen3-VL-8B by ~3.75 pp over the same on-policy self-distillation setup without any answer key, crop, or external teacher.
Say you’re fine-tuning a multimodal model that reads screenshots for a UI agent. The usual post-training story is On-Policy Distillation: sample the student’s own rollouts, but grade each token against a stronger teacher model. That teacher is expensive to run and often doesn’t exist for your domain. The self-distillation variant On-Policy Self-Distillation drops the teacher by using an Exponential moving average copy of the student itself, but then the target signal is nearly identical to what the student already predicts, so nothing new gets learned.
Recent fixes give the self-teacher some privileged crutch the student doesn’t get: the ground-truth answer text, a chain-of-thought trace, or a pre-cropped region highlighting the relevant part of the image. Those crutches require labels, annotators, or a separate localization model. This paper asks whether you can generate the teacher-student asymmetry from nothing but the same image blanked out.
The student rolls out a response to (prompt, image) as normal. At each token position in that response, the Exponential moving average teacher is run twice on the exact same prefix: once conditioned on the real image, once conditioned on a same-size all-black image. Subtract the two log-probability vectors and you get, per vocabulary token, a score for “how much does the real image content push this token up versus the blank.” Call that score \u0394.
The naive move would be to distill \u0394 directly, but a token can have huge positive \u0394 while still being wildly implausible under the real image. So VCSD uses the two teacher outputs for different jobs: the real-image distribution defines the plausible candidate set (keep only tokens with probability at least \u03b2=0.1 of the top token), and \u0394 reweights within that set. In plain terms: multiply each surviving token’s probability by exp(\u03b1 \u00b7 \u0394), renormalize, and distill that sharpened target into the student with standard forward-KL. Termination tokens are exempted from the reweighting to keep generation stable.
for (prompt, image) in batch:
response = student.sample(prompt, image) # on-policy rollout
for t, prefix in enumerate(response):
p_real = ema_teacher.next_token(prompt, image, prefix)
p_blank = ema_teacher.next_token(prompt, BLACK, prefix)
delta = log(p_real) - log(p_blank)
support = {v: p_real[v] >= 0.1 * max(p_real)}
target = normalize(p_real * exp(alpha * delta), over=support)
loss += kl(target.detach(), student.next_token(prompt, image, prefix))
ema_teacher.params = 0.95 * ema_teacher.params + 0.05 * student.params
The paper also shows the target has a clean interpretation: it’s the exact solution of a one-step KL-regularized policy update where \u0394 plays the role of an implicit “visual evidence reward” and the real-image distribution is the reference policy. Separately, \u0394 approximates the Pointwise Mutual Information between the token and the image content, given the text context.
The prevailing recipe for making self-distillation informative is to inject a privileged signal into the teacher: the answer, a reasoning trace, or a cropped region of interest. This paper shows you don’t need any of that. Blank out the image and diff the two teacher predictions. The gap itself is the supervision signal, because it isolates exactly the token preferences that come from the specific image rather than from language priors. The clearest evidence is the anchor ablation: keeping only the raw log-ratio (\u03bb=0, no real-image anchor) achieves comparable accuracy, confirming the contrast, not the anchor, drives the gains. The anchor’s job is separately to prevent the model drifting into wrong-language outputs.
•
The load-bearing ablation is the plausibility support. Without it (\u03b2=0, reweight the whole vocabulary), training starts fine but degrades steadily over the run because tokens with huge \u0394 but tiny real-image probability poison the recursive target. With \u03b2=0.1, accuracy holds stable. This says the mechanism only works because the real-image distribution is used as a gate, not just a starting point.
•
On the headline seven-benchmark average, VCSD beats the answer-hint On-Policy Self-Distillation baseline at every scale tested: Qwen3-VL 2B/4B/8B goes from 62.27/71.30/72.51% (base) to 67.04/73.16/76.26% (VCSD), and gains persist across Qwen3.5 2B/4B/9B (+2.9 to +4.3 pp over base). The answer-hint baseline itself is inconsistent on Qwen3.5, sometimes not beating the base model at all.
•
The contrast strength \u03b1 is robust in a plateau: \u03b1 \u2208 [1, 1.5] all work within ~1%. Turning it off (\u03b1=0) loses 2.33 pp; cranking it to \u03b1=2 collapses back to the no-contrast level.
•
The control image barely matters: black, Gaussian noise, Gaussian blur, and even no image at all give similar results. What matters is that instance-specific content is gone.
•
Forward KL beats reverse KL (+2.27 pp) and Jensen\u2013Shannon Divergence (+0.79 pp) as the distillation divergence, consistent with wanting to cover the whole shaped target rather than mode-seek.
•
Qualitative: on a MathVista place-value question, base and OPSD miscount the thousand-cubes; VCSD counts them correctly. Token-level heatmaps show VCSD concentrates positive \u0394 on visually grounded words (“roof”, “shingles”, “beige”) rather than distributing it evenly.
Reach for this when you’re post-training a VLM for a task where you have (image, prompt) pairs but don’t have gold answers, reasoning traces, or bounding boxes, and you don’t have a stronger model to distill from. Document VQA on internal documents, screenshot understanding for an agent, or industrial inspection are natural fits. The training loop is the same as any on-policy distillation setup: one extra teacher forward pass per prefix on a black image, no extra inference cost after training.
The paper does not mention a code release. Reproduction should be straightforward from the equations: it’s on top of standard On-Policy Self-Distillation with a second teacher forward on a same-size black RGB image, \u03b1=1, \u03b2=0.1, forward KL at temperature 2, EMA rate 0.05. Training data was ViRL39K, evaluated on BLINK, MMStar, MathVista, V*Bench, HRBench, and HallusionBench.
A blanked-out control input is a free source of self-supervision: the model’s own reaction to what got removed tells you which predictions were actually grounded. This trick is specifically for cases where you want the model to depend more on a particular input channel (here, the image) than on its language prior, and it only helps because the real-input distribution is used to gate which tokens the contrast is allowed to move.
•
All results are on one training set (ViRL39K) and one model family lineage (Qwen). There’s no evidence yet the trick transfers to other VLMs, other modalities (audio, video), or tasks where the “blank” analog is less obvious.
•
Every step now requires two teacher forward passes instead of one. The paper reports a fixed 90-step budget on 8 B200 GPUs but doesn’t quantify the wall-clock or memory overhead versus vanilla On-Policy Self-Distillation.
•
The unrestricted variant (no plausibility gate) degrades over training. That’s a warning that the method’s stability depends on tuning \u03b2, and longer training runs than the 90 steps evaluated here might expose failure modes the current experiments don’t reach.