Should a visual-question-answering benchmark use one standardized prompt or test several equivalent versions? It should do both—but for different purposes. Keep one canonical prompt so results are reproducible, then run a prespecified sensitivity panel that separates answer-option formatting from question-and-image ordering. One study found rank reversals across 48 multiple-choice formats; another found that moving and repeating the question could repair a specific answer-time access failure. Variation can therefore reveal either an unwanted measurement confound or a useful correction, not just generic prompt noise.
The same task can become a different computation
Imagine a benchmark item with an image, a question, and four answer choices. The intended task stays constant if the choices are labeled A–D rather than 1–4, or if a colon replaces a parenthesis. Yet the model does not receive “the same meaning” directly. It receives a serialized sequence of text and image tokens. Changing that sequence can alter which tokens are created, where they sit, and which earlier information is easiest to use when the answer is generated.
That means a reported score measures a model together with its prompt serialization and evaluation procedure. It does not isolate an abstract quantity called visual reasoning. The practical question is not merely whether prompts matter, but why a particular change matters. Two recent papers identify different causal paths.
Tiny option changes can reverse a leaderboard
First consider the last step of a multiple-choice task. Even if a model has extracted useful information from the image, it must map that information onto an option identifier such as A or 3. Punctuation can affect how the tokenizer splits or combines that identifier with neighboring characters. The resulting representation can then make the model’s option-selection step more or less reliable.
The reported result is not just movement in absolute scores: the study finds frequent rank reversals even under order-invariant evaluation. Its mechanistic analyses attribute the instability to low-level language-model effects, including option-ID tokens being fused or removed during tokenization and identifier choices changing the reliability of attention used for option selection. The authors also report weak correlation between these multiple-choice rankings and open-ended evaluation.
That weak correlation does not make open-ended scoring ground truth. It establishes a narrower warning: the tested multiple-choice rankings reflect option-selection dynamics in addition to whatever multimodal reasoning the models perform. The abstract supplies neither one aggregate reversal rate nor a correlation coefficient, so it does not support estimating how frequently any arbitrary format will reverse a ranking.
Putting the question first can make it harder to answer
The second paper investigates a different stage. Suppose the sequence is question, then hundreds of image tokens, then the answer position. Putting the question first can help shape the processing of the image: the model knows what concepts to seek. But by the time it generates an answer, the question is now far back in the sequence, behind the image tokens. A model can therefore gain guidance during perception yet lose effective access to the question during response generation.
“Ask Twice, Look Twice” tests question-first against image-first ordering across five open vision-language models. The researchers use internal representation and attention probes to examine what information is emphasized, then suppress selected attention links to test whether answer generation depends on access to the question. They report that question-first prompts shift image-patch representations toward question-relevant concepts, while the answer token barely attends to the earlier question. Their causal attention intervention indicates that the answer reads the question when it follows the image. This diagnosis motivates a targeted change: place one copy of the question before the image and another after it. The first copy can guide image processing; the second remains nearby when the model generates its answer. This is called question echoing, but the causal sequence matters more than the label: early guidance, intervening image tokens, restored late access.
Across the five tested open models, question-first ordering cost up to 17.5 group-accuracy points. Echoing recovered most of that gap and, on NaturalBench and Winoground, beat the best single-pass ordering—by as much as 19 group-accuracy points on Winoground—without training, fine-tuning, or an architecture change. Those maxima describe the reported settings, not an expected gain for every model or benchmark.
This result qualifies the idea that all prompt sensitivity is merely nuisance variance. Here, a particular serialization is tied to a diagnosed information-access problem and is designed to repair it. The evidence does not show that echoing is optimal for closed models, every tokenization scheme, or every VQA benchmark; its mechanistic findings concern the tested models’ token and attention configurations.
One average would hide two different problems
The papers alter different parts of different systems. One varies multiple-choice labels and punctuation across seven models and five datasets. The other varies question-image order and echoing across five open models, including tests on NaturalBench and Winoground. Their scores should not be pooled or compared as if they measured one common effect size.
Reading them together nevertheless changes the reporting decision. A benchmark with one fixed prompt is reproducible, but it can conceal whether its leaderboard depends on an arbitrary delimiter or identifier. Averaging every variation into one score is not a complete remedy either: it could mix option-selection failures with a prompt that repairs answer-time access, obscuring a useful diagnosis.
A defensible VQA report should preserve one reproducible canonical serialization while also reporting prespecified prompt sensitivity. This is an editorial protocol inferred from both studies, not a protocol the papers jointly evaluated. The sensitivity panel should keep at least two result groups separate:
•
Option serialization: vary identifiers, delimiters, and separators while preserving option content and order. This reveals whether comparative rankings depend on low-level answer-selection machinery.
•
Question-image placement: compare question-first, image-first, and—where the architecture and access permit it—question echoing. This tests whether early perceptual guidance conflicts with late access to the question.
For a multiple-choice benchmark, I would also include an open-ended or otherwise format-distinct check. Disagreement would not identify which score is correct, but it would warn against treating one prompt’s multiple-choice rank as a stable measure of multimodal reasoning.
The resulting report answers two questions instead of collapsing them. Which model performs best under the canonical, reproducible setup? And how much does that conclusion depend on formatting choices or on a serialization that changes access to task information? The first supports comparison. The second tells builders whether the comparison survives contact with an equivalent interface—and whether a deliberate prompt change fixes a specific computational bottleneck rather than merely shopping for a better score.