ProVisE lets image-generation models answer spatial reasoning questions by drawing on the image instead of emitting coordinates, then parses the drawing back into the benchmark’s original answer format, revealing that visual answering rescues ~30–37% of the failures made by the same team’s text answers.
Suppose you’re building a robotics or UI-automation agent that needs to point at a graspable region, sketch a path, or mark which of two objects is closer. Today, you either force a VLM to emit [x1,y1,x2,y2] boxes and direction strings, or you ask an image-generation model to draw the answer and then hire another model to read the drawing. The first option distorts continuous spatial judgments into brittle text. The second option (VLM-as-judge) means your final score reflects the judge’s reliability, not the model’s spatial cognition. This paper argues neither is a fair test of image generators, which natively express location by marking pixels, and it builds the missing evaluation harness.
The core move is to freeze a per-task visual protocol before any model is evaluated. A protocol is a pair: (1) a generation prompt telling the image model exactly how to encode its answer visually (mark instances in green, draw a yellow trajectory, fill one cell of a 3×3 direction grid, output a binary mask), and (2) a deterministic parser that reads that specific visual convention back into the benchmark’s native answer type. For example, on a relative-depth question, the model outputs a depth-shaded image; the parser samples a 5×5 grayscale patch at each of the benchmark’s two query coordinates and picks the brighter one as closer. No second LLM is in the loop. Because manually writing a protocol per task doesn’t scale, they add an Agentic builder: given a new benchmark’s answer schema, it either reuses an existing protocol, composes one from ~41 registered parsing operators (color thresholding, contour extraction, CLIP similarity, OCR, geometry), or falls back to a constrained VLM parser that sees only the generated image (not the source question, so it can’t just re-solve the task). Every candidate passes a smoke test on a few examples before being locked in and applied uniformly to all models.
for task in benchmark.tasks:
protocol = agent.route(task) # Reuse | Build | Fallback
if not smoke_validate(protocol, task.samples[:3]):
protocol = agent.revise_once(protocol)
for model in image_generators:
img = model.generate(sample.image, protocol.prompt)
pred = protocol.parser(img) # deterministic when possible
score = task.native_metric(pred, sample.answer)
They also build SpatialGen-Bench, 470 samples across 14 subtasks organized into four levels (perception, understanding, reasoning, interaction), specifically to stress-test heterogeneous answer forms.
The prevailing move when you want to compare an image generator to a text VLM on spatial reasoning is to bolt on a VLM-as-judge that reads the picture and produces a label. This paper shows the opposite. Freeze a deterministic visual convention per task before evaluation, and the generator’s drawing becomes directly scorable under the original metric, with no black-box judge in the loop. The load-bearing evidence is the parser-sensitivity ablation: swapping in three different universal VLM parsers over the same saved images shuffles both scores and rankings, which means judge-based evaluation was never measuring the generator alone.
•
Parsers are not interchangeable. On the same fixed set of generated images, ProVisE ranks GPT Image 2 first; Qwen3-VL-8B agrees; Qwen2.5-VL-72B puts JoyAI-Image first; Llama 4 Scout puts Seedream 4.5 first. Rank correlations with ProVisE are only 0.60, 0.31, and 0.60, with mean absolute score shifts of ~5–6 points. Valid-parse rates stay above 92%, so this isn’t a formatting issue; the universal parsers genuinely reinterpret the pictures.
•
Interfaces are complementary, not ordered. Pairing GPT-5.4 (text) with GPT Image 2 (visual) on identical items: the visual model correctly answers 71 items the text model missed, a 37.0% rescue rate on text failures. The same-family SenseNova-Vision-7B-MoT comparison shows 30.6% rescue in its visual mode.
•
Where showing wins vs. telling. Visual answering gains most on Depth (+18.85 pp over the text-model mean) and Relationship (+6.74 pp), where the answer is a spatial state. Text answering wins on Size, Feasibility, Prediction, and Counting, where you must transform evidence into a discrete judgment; text leads Reasoning overall by 17.63 pp.
•
Failures are reasoning, not plumbing. Of visual-answer failures, 88.03% are valid-but-wrong predictions; only 8.46% are protocol noncompliance, 3.45% parser failures, and 0.06% generation failures. The pipeline works; the models just get spatial questions wrong.
•
Everyone is far from human. Best text model (GPT-5.4) hits 61.04 overall; best visual (GPT Image 2) 54.49; humans 87.79.
•
Agentic transfer works. The builder successfully constructed protocols for all 23 task units across six external spatial benchmarks (EmbSpatial-Bench, OmniSpatial, Q-Spatial+, RoboSpatial-Home, SAT, RoboAfford), with a 95% valid-parse rate on the 300-sample pilot.
Reach for this when you’re evaluating a multimodal system whose answer is a location, region, or path rather than a label. Instead of the usual pipeline (model emits [x1,y1,x2,y2] → string match → arguments about tokenizer quirks), you define a visual convention up front (“mark the target with a cyan dot on a copy of the input”), write a small CV parser to extract the dot’s coordinates, and score with your existing point-in-mask metric. The generator and a text VLM now share a scoring path. This is especially valuable if you’re benchmarking internal image-edit models where a judge VLM would be a confounder.
The paper positions ProVisE and SpatialGen-Bench as an open-source suite with a project repository (link on the paper’s first page; the paper doesn’t specify a license in the main text). The 41-op parser library, the Agentic builder, and the 470-sample benchmark are the reusable pieces; the six external benchmarks are configured but not re-released. Directly usable if you already ship image-generation models and want a metric-compatible spatial testbed.
When your model’s answer is spatial, score the picture with a fixed convention, not with another model’s opinion of the picture. The judge you skip is the confounder you were about to introduce.
•
The Agentic builder itself runs on GPT-5.4 and GPT Image 2 as backends. Protocols it invents may implicitly favor visual encodings that GPT Image 2 can execute cleanly, disadvantaging weaker generators. This isn’t a neutral referee.
•
Text-answering and visual-answering pools differ in architecture, scale, and training data, so “visual rescues 37% of text failures” is a system-level observation, not a controlled modality effect. The one same-family split (SenseNova text vs. visual) is a single data point.
•
Roughly a third of external-benchmark tasks (8 of 23) still route through a constrained Fallback VLM parser because deterministic operators can’t express them. The framework is only as “judge-free” as its parser library is complete, and the paper flags library coverage as ongoing work.