ActiveVision is a 17-task benchmark showing that frontier Multimodal Large Language Models cannot iteratively re-inspect an image the way humans do: the best model solves 10.6% of items where three humans average 96.1%, and neither more reasoning nor coding agents close the gap.
You’re shipping a product that asks a vision-language model to read a chest X-ray, count parts on a factory line, or trace a wiring diagram in a scanned PDF. Today’s leaderboards (MMMU-Pro, CharXiv) are near saturation, so you assume the model can look as well as it can talk. This paper argues that assumption is wrong for any task where the answer requires the model to keep glancing back at the pixels while it reasons. The dominant prior framing, exemplified by BLINK, measures a generic human-model gap but does not isolate which visual capability is missing. ActiveVision names that missing capability and puts a number on it.
The design principle: build tasks whose visual content carries more information than any single language caption can preserve, so a model that encodes the image once and then reasons over its own verbal summary must lose the answer. Three task families operationalize this:
•
Distributed Scanning: count many small signals (dots, regions, graph faces) spread across the canvas, beyond the subitizing range.
•
Sequential Traversal: follow a curve, arrow chain, or tangled loop step by step, maintaining position and running tally.
•
Visual Attribute Transfer: extract a fine-grained property (length, curvature, dot pattern) from one region and match it against candidates elsewhere.
Each task is generated in two stages. A deterministic Python scaffold places items at continuous coordinates, with shape boundaries drawn as random Fourier harmonics curves and traces as random splines. Then GPT-image-2 re-renders the scaffold as a photorealistic scene (aerial fields, ropes on driftwood, stones with footprints) while preserving positions, counts, and topology. Only the rendered image is shown to the model. Ground truth stays attached to the scaffold.
for task in 17_generators:
scaffold, question, answer = task.generate(seed)
photo = gpt_image_2.render(scaffold, task.prompt)
# positions, counts, topology preserved
model_out = mllm(photo, question) # <answer>...</answer>
score += exact_match(model_out, answer)
Evaluation is pure chain-of-thought at every reasoning-effort tier the API exposes, plus a separate agentic track where Codex agent harness and Claude Code can write and run their own vision code.
The prevailing story about frontier model weaknesses is that they need more deliberation: raise the reasoning-effort dial and the gap closes. This paper shows the opposite. The bottleneck is perceptual, not deliberative: the model encodes the image once and then reasons over its own summary, so no amount of extra thinking retrieves visual evidence that was never fetched. The load-bearing evidence is not the headline accuracy but the reasoning-effort sweep and the failure-mode analysis showing counts, traces, and comparisons break in the same way regardless of budget.
•
Scaling GPT-5.5’s reasoning tier from none to xhigh raises per-item cost roughly 100× but accuracy only from 2.4% to 10.6%. Spending 31× more per item on Claude Fable 5 does not move accuracy beyond noise. The models do not run out of reasoning steps; they run out of visual evidence.
•
Headline: best pure-CoT model is GPT-5.5 at 10.6% (9 of 85), scoring zero on 11 of 17 tasks; Claude Fable 5 solves 3.5%; three humans average 96.1%, about nine times the best model.
•
Failure signatures are consistent. On counting, fitted slopes of predicted vs. true count fall below the diagonal and worsen as scenes grow more crowded, as if the model glimpses rather than scans. On ordered walks, prefix survival collapses in the first one or two steps, and no walk in 18 runs is completed exactly. On difference-spotting, models default to answering “same” or “none,” a response bias rather than perception.
•
Agentic tool use helps unevenly. The Fable 5 coding agent reaches 50.6%, Codex 37.6%, the Opus 4.8 agent 24.7%. Gains concentrate on Visual Attribute Transfer (43–66%), where template cropping admits clean code. Traversal stays near zero for two of three agents, and all three score 0 on Tangled Loop Counting. Agents cost $2.74–$7.63 and 12–15 minutes per item, vs. a human’s roughly half minute unaided.
•
Two agent failure modes recur: classical vision primitives fragment on photorealistic textures, and the model cannot visually verify its own code’s output. A glance at the overlay would falsify the answer, but the glance itself requires the active perception the model lacks.
Reach for this when you’re evaluating a vision-language model for a deployment where the operator would naturally look twice: a radiology triage assistant, an inventory-count sanity checker, a schematic-tracing tool for field techs, a document layout parser on phone snapshots. The paper’s practical warning: benchmark scores from MMMU-Pro or CharXiv do not tell you whether the model can keep looking, and neither does raising reasoning effort. If your task decomposes into a reliable script (segment, threshold, connected components), an agentic wrapper may help substantially. If it requires following a structure or spotting a subtle difference, agents currently fail expensively.
Everything is released: website, GitHub, and a HuggingFace dataset. Because each task is generated from a deterministic seed with attached ground truth, you can produce fresh unseen instances for internal evaluation rather than worry about contamination. The 85-item public split is small enough to run cheaply against your own model.
More thinking does not fix a model that only glanced once. If your product needs the model to revisit visual evidence as reasoning unfolds, evaluate that capability directly. Do not assume it comes bundled with reasoning-effort tiers or agentic scaffolding.
•
Images are synthetic-then-re-rendered, not sampled from the natural image distribution. External validity rests on the elemental operations (scan, trace, compare) generalizing to real deployments, not on the pixel statistics matching.
•
The design premise is that no short language description carries the answer. As models get better at dense image description, this property erodes and the benchmark will need tightening.
•
The agentic track is reported as an ablation, not a full agentic-vision benchmark. A purpose-built agent stack with visual verification loops might close more of the gap than the three tested agents did.