VDiff-Bench turns “spot the difference” into a 4-way multiple-choice test with human-verified plausible wrong answers, showing that even top Multimodal Large Language Models frequently pick “no difference” on subtle low-level changes like noise and texture, with within-model gaps of 37–54 percentage points between semantic and low-level changes.
You’re building an image-editing evaluator or a visual QA agent. You ask an Multimodal Large Language Model whether two images differ, and it confidently says “no difference” even though one is noticeably smoother or noisier. That failure poisons any downstream reward signal for image editing, because the judge cannot tell whether the requested edit actually happened or whether an unwanted side-effect was introduced.
Prior benchmarks in this space, like Spot-the-Diff and CLEVR-Change, mostly score free-form captions describing what changed. Scoring those captions requires either fragile n-gram metrics like BLEU-4 or an LLM-as-a-Judge, both of which have known reliability problems. They also cover a narrow slice of change types (mostly semantic object edits) and often use simple synthetic scenes that modern models solve easily. The authors argue there is no clean, deterministic way to ask “can this model actually see this specific difference?” across a broad set of change categories.
The contribution is a benchmark design, not a new model. Each item shows two images plus four text options: the real difference, two plausible-but-false differences, and a fixed “no difference” option that is always wrong (every pair does contain a real change). The model just returns a letter, so scoring is exact-match. No judge, no caption metric.
The trick is in the wrong answers. Random distractors would be too easy, so the authors generate ground-truth-conditioned hard negatives: they feed the image pair and the real difference to Gemini 2.5 Pro and GPT-5.5, and ask for false statements that stay close to the true change (swap the affected object, reverse a direction, tweak an attribute). A human expert then filters, rewrites, or discards weak candidates. This forces the model to actually look, rather than pattern-match on which option “sounds like a difference.”
Image pairs come from a mix: existing datasets (Spot-the-Diff, OmniEdit, OmniDiff, MotionEdit), scanned children’s puzzle books, and programmatic augmentations on FFHQ faces (added Gaussian noise, RGB shifts, skin-aware smoothing, gamma illumination changes). Position and OCR edits are synthesized by prompting GPT-image-2 and Gemini-3-Pro-Image with localized edit instructions. Every pair is mapped into one of 10 categories spanning semantic changes (position, motion, appearance, substitution, OCR) and low-level changes (whole-image color, noise, texture, illumination).
The question construction loop, in pseudo-code:
for pair in image_pairs:
gt_diff = human_annotated_difference(pair)
candidates = mllm_generate_false_diffs(pair, gt_diff) # Gemini/GPT
hard_negatives = human_filter_and_rewrite(candidates)[:2]
options = shuffle([gt_diff, *hard_negatives, "no difference"])
yield {"images": pair, "options": options, "answer": gt_diff}
Final size: 1,756 questions over 1,543 distinct image pairs. Random guessing scores 25%; “informed” guessing (never pick no-difference) scores 33.3%.
Eleven MLLMs were evaluated. Overall accuracy spans 35.8% to 89.6%. Gemini 3.1 Pro leads at 89.6%, with Kimi-K2.5 (89.2%), Gemini 3.5 Flash (88.8%), and Doubao Seed 1.6 Vision (87.7%) close behind. No single model wins every category.
The headline result is a dissociation between semantic and low-level perception. Four models show large within-model gaps: Grok 4.3 and three 7–8B open models (Qwen3-VL-8B Thinking, InternVL3.5-8B, LLaVA-OneVision-Qwen2-7B) drop 37.2 to 53.8 percentage points when moving from semantic to low-level changes. The smallest open models score 52.5–70.6% on semantic changes but only 8.7–33.3% on low-level ones.
The dominant failure mode is picking “no difference” even though a real change exists. For these four models, 48.1–80.9% of low-level questions get the no-difference answer. Grok 4.3 is the most striking case: it scores 96.4% on motion and 82.0% overall on semantic changes, but only 5.3% on noise and 15.3% on texture. Meanwhile the much larger open-weight Kimi K2.5 handles low-level changes at 88.8%. The authors read this as: scale raises the ceiling, but training data, objectives, and visual encoding determine whether cross-image comparison actually works. They do not run ablations that isolate any single factor.
One more note: appearance/disappearance stays hard even for the best models (category max 79.6%), likely because it requires scanning both images for small missing entities.
If you use an MLLM as a judge for image editing or generation, do not assume that strong VQA scores imply reliable difference detection. The paper shows this assumption breaks specifically on global color, noise, texture, and illumination. Worth testing your own judge on the released benchmark before trusting its verdicts on edit fidelity.
If you are picking an off-the-shelf model for a diff-detection feature, the paper’s evidence favors Gemini 3.1/3.5 Pro/Flash, Kimi K2.5, or Doubao Seed 1.6 Vision over Grok 4.3 for low-level sensitivity. Note this is a benchmark ranking on 1,756 curated items, not a production deployment study; category performance varies enough that you should re-test on your own change distribution.
If you are training or fine-tuning an MLLM, the finding that a 7–8B model can score under 10% on texture while doing fine on motion suggests low-level comparative perception is a distinct capability worth targeting with dedicated data, not something that emerges from general multimodal scaling. The benchmark’s category breakdown gives you a diagnostic surface to check whether an intervention helped.
Data and demo are released on Hugging Face: dataset and project page.
The hard-negative distractors were generated by Gemini 2.5 Pro and GPT-5.5, then human-filtered. Models from the same family may have a subtle advantage or blind spot on the resulting phrasing; the paper does not analyze generator-evaluator overlap.
Many low-level items come from programmatic augmentations on FFHQ faces (Gaussian noise, RGB shifts, smoothing). Results on this slice may not transfer to naturally occurring noise or texture variation in the wild.
The “scale is not sufficient” claim rests on cross-model comparison of Grok 4.3 vs. Kimi K2.5/K3. The authors do not have access to training data or architectures for these closed and semi-closed systems, so the attribution to “data, objectives, or visual encoding” is a hypothesis, not a controlled result. Invalid outputs count as incorrect, which slightly penalizes models that refuse or produce unparseable answers (mostly a small effect except for Kimi K3 at 5.2% invalid).