Vision-language models score well on depth questions partly by cheating: they treat “higher in the image” as “farther away.” The paper shows this Vertical-Distance Entanglement is baked into the embeddings themselves, producing 30+ point accuracy gaps when that heuristic is reversed.
You’ve shipped a multimodal assistant that answers “which object is closer to the camera?” for a robotics or AR pipeline. It passes your eval at 85% and you move on. This paper says: re-check whether your eval images happen to put farther objects higher in the frame. If they do, your model is likely riding a perspective shortcut from natural-photo statistics, and it will silently fail on ceiling-mounted cameras, top-down views, or any scene where the heuristic flips. Prior probing work like AdaptVis looked at attention patterns; this work instead asks how the spatial axes themselves are laid out inside the model.
The core trick is a contrastive probe. For each image, the authors ask two questions that differ only in object order (“is A left of B?” vs “is B left of A?”), pull the hidden state from a mid-network layer for each, and subtract them. That difference vector isolates how the model encodes the spatial relation itself, stripped of object identity and scene content. Aggregating these delta vectors across many images for each category (left/right, above/below, far/close) gives a geometric portrait of the model’s spatial axes.
Two metrics fall out. Axis coherence is the average cosine similarity among delta vectors for one axis (after sign-flipping opposing categories so they point the same way). High coherence means the model encodes “farther” as a consistent direction in embedding space. VD-Entanglement Index measures whether the “above” direction looks like the “far” direction (and “below” like “close”): positive means the model has fused vertical position with depth.
To confirm this isn’t just a quirk of photo datasets, they build SpatialTunnel, a Blender-rendered corridor where objects sit at fixed depths but slide around the tunnel’s cross-section. An object can be high in the frame yet close to the camera, breaking the natural correlation.
for image, (obj_a, obj_b) in dataset:
q1 = f"Is {obj_a} closer than {obj_b}?"
q2 = f"Is {obj_b} closer than {obj_a}?" # swap
h1 = model.hidden_state(image, q1, layer=L_star)
h2 = model.hidden_state(image, q2, layer=L_star)
deltas[category].append(h2 - h1)
coh_distance = mean_pairwise_cosine(sign_align(deltas["far/close"]))
vd_ei = cos(mean(above), mean(far)) - cos(mean(above), mean(close)) + ...
The field’s default story is that more spatial training data fixes spatial reasoning. This paper shows the opposite. Scaling spatial fine-tuning often sharpens the vertical-distance shortcut rather than dissolving it, because the new data inherits the same perspective statistics as the old data. The evidence that pins this down is not the headline benchmark lift but the gap between consistent and counter examples on the SpatialTunnel synthetic set, where the photographic correlation is engineered away.
The load-bearing finding is the consistent-vs-counter gap that survives scaling. On SpatialTunnel, base Qwen2.5-VL hits 77.6% on consistent examples but 36.0% on counter examples, a ~40 point gap that fine-tuning at 2M samples does not close. On real benchmarks like EmbSpatial-Bench, the same Qwen at 2M scores 60.9% consistent vs 24.0% counter, a 36.9 point gap. The skew matters because consistent examples make up 80.9% of EmbSpatial-Bench and 60.5% of CV-Bench-3D, so headline accuracy mostly measures the easy half.
The representation probes explain why. Across all model families, distance-axis coherence is the weakest of the three axes. Fine-tuning Molmo pushes vertical coherence from 0.23 to 0.57 but barely moves distance coherence. Distance coherence measured on the synthetic set correlates with counter-example accuracy on real benchmarks at Spearman 0.76–0.80, so it generalizes. Two outliers escape the trap: RoboRefer-2B-SFT, trained on 20M+ samples including RGB-D supervision, has the highest distance coherence (0.182) and lowest VD-EI (0.362) in its family, and Qwen3-VL-235B reaches 90.8% mean accuracy with only a 6.8 point gap. PCA confirms the picture: weaker models show “far” and “close” vectors collapsed into the vertical cluster, while RoboRefer shows three clean orthogonal clusters.
Reach for this when you’re evaluating or shipping a VLM for any task involving depth, occlusion, or 3D placement: bin your eval set by whether the farther object is higher or lower in the frame, and report the two numbers separately. If the gap is big, your benchmark is flattering the model. For training, the takeaway is that piling on more spatially-annotated photographs may not help. You likely need explicit depth supervision (the RoboRefer recipe) or counter-perspective synthetic data to force the axes apart.
The authors release the project page with code, the SpatialTunnel renders, and the probing pipeline. The probe itself is cheap: it’s just hidden-state extraction plus cosine similarity, applicable to any open-weight VLM. You can compute distance coherence on your own model as a sanity check before trusting a benchmark number.
A VLM that scores well on spatial benchmarks may have learned camera angles, not geometry. Before trusting depth-related accuracy, split your eval by whether the easy perspective heuristic agrees with the answer. The gap between those two slices tells you more about real 3D understanding than the headline number does.
•
The probe needs hidden-state access, so it doesn’t work on closed API models. The authors fall back to exact-match Yes/No accuracy for GPT-5.2 and Gemini 2.5 Pro, which is a coarser signal.
•
Distance-coherence values differ in absolute magnitude between synthetic and real images, so the metric is reliable for ranking models trained or evaluated under the same conditions, not as an absolute score.
•
RoboRefer’s win conflates several variables: 10x more data, RGB-D supervision, and a different training curriculum. The paper cannot isolate which ingredient cures the entanglement, so “add depth supervision” is suggestive rather than proven.