S-Space is a three-axis linear subspace inside multimodal language models where object-name tokens carry readable left-right, up-down, and near-far coordinates. Editing those coordinates flips spatial answers while leaving unrelated judgments intact.
If you build an agent that uses an MLLM to answer “is the mug left of the laptop?” or “from behind, which chair is closer?”, you’ve probably seen it fail on questions a child solves instantly. The usual response is to swap models or add Chain-of-Thought, but you can’t tell where the failure lives: did the model misperceive the scene, or did it perceive correctly and then reason badly?
Prior interpretability work on vision-language models has shown that spatial information gets bound to object-name tokens (the paper cites a line of work on linear mechanisms for spatial reasoning in VLMs). This paper pushes further: it claims there is a single, fixed three-dimensional subspace per model that acts as the model’s internal spatial scratchpad, and that you can read, edit, and diagnose through it.
The intuition: when the model sees “apple” in an image, the hidden vector at the apple token quietly contains an (x, y, z) coordinate for where the apple is, written along three fixed directions in activation space. Find those three directions once, and you have a coordinate reader that works everywhere.
To find the directions, the authors ask the model many relative questions like “Is the apple to the left or right of the cup?” and look at the gradient of the logit difference (right minus left) with respect to the hidden state at the apple and cup tokens. They subtract the two gradients (query minus reference) so that effects which push both tokens the same way cancel, and average across prompts and swapped object orders so prompt-specific noise averages out. What survives is a unit vector per axis. Stack the three vectors into a matrix V and the coordinate of any object token is just V^T h. That’s it. No training, no probe fitting beyond the gradient average.
To answer a spatial question without generating text, compare two objects’ coordinates along the queried axis: positive margin means right/above/close, negative means left/below/far. To test causality, edit the coordinates in-place and let generation continue.
# Identify S-Space (once per model, per layer)
for axis in ["x", "y", "z"]:
grads = []
for (img, prompt, q_tok, r_tok) in spatial_questions:
s = logit(pos_word[axis]) - logit(neg_word[axis])
grads.append((grad(s, h[q_tok]) - grad(s, h[r_tok])) / 2)
v[axis] = unit(mean(grads))
V = stack([v["x"], v["y"], v["z"]])
# Use it
coord = V.T @ h[object_token] # read
h_new = h + V @ inv(V.T @ V) @ (c_new - V.T @ h) # edit
The axis-identification layer is picked by sweeping validation accuracy; the paper reports it lands in the middle of the network, not early (too close to pixels) or late (too close to next-token prediction).
Readout accuracy. On CV-Bench, EmbSpatial, and SpatialTunnel, answering relative-position questions by comparing S-Space coordinates matches the models’ own verbal answers, across Molmo2-ER, Qwen3.5-4B, and Qwen3.6-27B. Importantly, the axes identified on COCO val2017 transfer to these benchmarks without retuning.
Causality, not just correlation. Editing the coordinates (shifting one object, swapping two objects along an axis) flips the model’s spoken spatial answer, while an energy-matched random edit in the orthogonal complement does not. A sanity check: shifting both objects by the same amount preserves their relative position and the answer stays put. Critically, the same edits barely perturb a color question about the same object, so the edit is spatially targeted rather than generally destructive.
Persistence. The coordinates stay readable even when the prompt asks about colors or queries a single object with no comparison. The authors read this as evidence that S-Space is an intrinsic organization of the representation, not an artifact of the identification prompts.
Content beyond visible pixels. Case studies show the ball token sits in the direction a goalkeeper is diving even when no ball is visible, the token you projects toward the close end of the distance axis, and changing “moving past you” to “moving toward you” pulls you closer to the robot. More strikingly, socialist projects left and conservative projects right on the horizontal axis, mirroring the political metaphor. The authors frame this as spatial structure inherited from language pretraining.
Perception vs. computation on SpinBench. For perspective-taking questions (“from the back view, which object is closer?”), direct answers are weak and chain-of-thought helps but leaves errors. If instead you read S-Space coordinates from the original view, apply the viewpoint rotation yourself with a fixed matrix, and threshold the result, accuracy is consistently higher across all three models in Table 1. When the prompt also supplies the correct source-view relation as a premise, the S-Space pipeline reaches near-perfect accuracy while the model’s own verbal answer still often misses. The authors’ interpretation: perception is largely fine, the bottleneck is the geometric transformation step inside the model.
Reference-frame coupling. When Qwen3.6-27B reasons about cardinal directions on MMSI-Bench, the tokens east, north, northeast project to the same S-Space quadrants as right, above, and upper-right. The model represents cardinal and image-relative directions through partly shared axes, which the authors suggest comes from map-aligned data in pretraining.
•
If you’re debugging spatial failures in an open-weights MLLM, S-Space gives you a decomposition the paper actually demonstrates: identify the axes with the gradient recipe on a few thousand COCO pairs, then on each failing example read coordinates and check whether perception or transformation is the broken step. The method needs activation access, so hosted-API-only models are out of scope here.
•
If you’re building an agent that handles viewpoint changes, the SpinBench result is worth testing in your own setting: do the geometric transform in code against coordinates read from the model, rather than asking the model to do the rotation in natural language. The paper shows this works on three specific models and one benchmark, not that it generalizes to arbitrary 3D reasoning.
•
If you care about interpretability, the political-semantics and you-token cases are a concrete demonstration that linear spatial directions carry non-literal content. Worth probing in your own model before assuming a direction is “just” geometric.
•
If you want to reproduce or extend, code is at GitHub and the project page is here.
The horizontal and vertical axes are image-plane coordinates, not world coordinates, so depth-dependent perspective effects aren’t disentangled. The axes are identified from relative-comparison prompts on COCO; generalization is shown on three spatial benchmarks and a few case studies, not on arbitrary scenes or tasks. All causal interventions use Molmo2-ER at one layer, so the “spatial edits don’t affect color answers” claim is a single-model demonstration. The viewpoint-rotation improvement on SpinBench uses a fixed, hand-specified rotation matrix matched to the question; this is a diagnostic tool showing the information is present, not a drop-in deployment recipe. And the political-semantics finding is a case study on specific words in specific models, not a systematic claim about bias.