Spatial-Interactor teaches vision-language models to track how scenes change by training on before/action/after triplets from interaction trajectories, then uses a privileged-context teacher to distill long-horizon integration, closing much of the gap between recognizing a single view and reasoning over a whole trajectory.
Say you point a VLM at a robot’s video feed and ask “how far did the camera move to the right across these 32 frames?” Current models flub this badly. The paper shows a diagnostic: if you randomly shuffle those 32 frames before asking, Qwen2.5-VL-7B’s accuracy on VSTI-Bench drops by less than 1 point. The model wasn’t really using temporal order in the first place. It was pattern-matching on order-invariant visual cues.
The deeper problem: on a local viewpoint-change question from SAT-Real, GPT-5.5 scores 88.7, but on the long-horizon camera-displacement version in VSTI-Bench it drops to 23.9. Even when a model can read one local change, it can’t stitch consecutive changes into a coherent trajectory.
The authors argue this is a training-data mismatch. Most spatial supervision is static QA: “what’s left of the chair?” That never teaches a model what changed between two frames or how to accumulate changes across a walk through a room. Interaction trajectories, which naturally pair (observation, action, next observation), give that supervision directly.
The contribution has two parts: a curriculum of interaction data, and a distillation trick for long trajectories.
The curriculum (LSI-108K, 108K QA pairs) organizes supervision by what changed and over how long a horizon:
•
L1, passive world-state transitions: viewpoint roughly fixed, objects move or get manipulated. “What action was taken between these two frames?”
•
L2, active self-state transitions: world fixed, the camera moves. “Did the camera rotate left or right?”
•
L3, long-horizon trajectory integration: full camera paths. “What was the total displacement?”
Data comes from simulators (AI2-THOR, ProcTHOR, HSSD) where an agent executes scripted actions and the ground truth falls out of the simulator, plus real videos with camera poses (ScanNet, ScanNet++, RoomTour3D) and robot trajectories (BridgeData V2) where geometry gives the labels. Deterministic templates turn each (observation-pair, measured-change) record into a verifiable QA pair. A frozen VLM checks whether the answer is visually recoverable, but never writes labels.
Stage 1 is plain SFT on L1 and L2. This teaches local transition reading.
Stage 2 is the interesting piece: On-Policy Distillation. The problem it solves: on long trajectories, the only reward you can compute is “was the final answer right?” That signal can’t tell you which intermediate step of the reasoning went wrong. OPD gives the model a teacher that has extra help.
Here is the setup. For each long video, a frozen strong VLM writes a short description of each of four segments (what’s visible, what changed, coarse camera motion). Concatenated in order, these become a privileged trace $z$. Then during Group Relative Policy Optimization (GRPO) training:
for batch in data:
rollouts = student.sample(x=(video, question), n=8)
for y in rollouts:
for t in reasoning_positions(y):
p_student = student(x, y[:t]) # no trace
p_teacher = stop_grad(prev_student(x, z, y[:t])) # sees trace
L_process += KL(p_teacher || p_student)
L_answer += grpo_loss(y, reward(y))
update(L_answer + beta*L_kl_ref + lambda*L_process)
Two details make this work. First, teacher and student score the same student-generated prefix, so the KL isolates “what would you predict next if you also knew the transition trace?” Second, the teacher is a stop-gradient snapshot of the policy right before the update, not a separately trained model. The trace $z$ is the only asymmetry. Process weight $\lambda$ (a scalar controlling how much the KL term matters) starts at 0.05 and decays to zero by step 1,200, so the trace guides early learning and fades out.
Across four backbones (Qwen2.5-VL 3B/7B, Qwen3-VL 4B/8B), Spatial-Interactor lifts the overall average across VSI-Bench, MindCube, VSTI-Bench, and SPBench-MV by 16.4 to 25.0 points over the base model. The 8B version reaches an overall of 65.9, which the paper reports as the best in its comparison table.
More telling than the totals are the axis-specific gains. Camera Displacement, the hardest long-horizon task, improves by 5.1 to 28.0 points depending on backbone. On four held-out benchmarks (MMSI, ViewSpatial, SAT-Real, SAT-Syn) not in training, gains are 4.3 to 10.6 points, suggesting transfer rather than benchmark memorization.
The ablation on Qwen2.5-VL-7B is where the story gets crisp:
•
SFT on external spatial QA only: 56.0
•
Add L1+L2 interaction data: 58.4
•
Add plain GRPO on L3: 59.2
•
Add OPD instead of plain GRPO: 60.1
So interaction-derived local transitions add ~2.4 points on top of a strong SFT baseline, and OPD’s privileged process supervision adds another ~0.9 over vanilla GRPO with identical data, rollouts, and rewards. OPD’s edge concentrates where you’d predict: +5.1 on Route Planning, +3.1 on Camera Displacement versus GRPO.
A nice sanity check: after training, frame-shuffling now hurts Spatial-Interactor by 6.6-7.0 points (versus <1 point for the base model). The trained model actually depends on temporal order. Separately, an inference-time experiment shows that giving a frozen SFT model an ordered 4-segment trace raises Camera Displacement from 32.6 to 35.4, while shuffling the same trace text drops it to 33.8. Order of the transition descriptions matters more than adding more segments.
Closed-loop tests on WalkerBench and ESI-Bench show gains too (WalkerBench success 7.0 → 14.0, mean actions per task 17.71 → 6.52), though absolute numbers stay low.
If you’re training a VLM for embodied or video-heavy tasks, the takeaway is that static spatial QA leaves capability on the table. Interaction triplets (before, action, after) with programmatically verifiable labels are cheap to synthesize from simulators or pose-annotated video, and the paper’s ablation suggests they help even on top of a large external spatial-QA mixture. The project page is the pointer for artifacts; the paper mentions released code, rewards, and configs but doesn’t link them directly in the excerpt. If you’re doing RL post-training and your reward is only on final answers, OPD is a pattern worth borrowing when you can construct a “privileged view” of the problem cheaply. The recipe: (a) generate side-information $z$ that makes the task easier, (b) let a snapshot teacher condition on $z$ while the student doesn’t, © same-prefix KL on the student’s own rollouts, (d) decay the KL weight. This is different from standard distillation because you never need a stronger model. The privileged context is the entire teacher advantage.
A hedge worth naming: the OPD gain over plain GRPO in the main ablation is about 0.9 overall points, with larger effects (3-5 points) on the specific long-horizon tasks. The framework’s benefit here rests on those targeted gains and cross-benchmark transfer (+1.0 to +1.4 over SFT-only checkpoints), not on a dramatic overall lift. Worth testing on your task before committing engineering.
If you just want better spatial reasoning at inference, the trace experiment implies a lightweight recipe you can try today: have a strong VLM write ordered per-segment descriptions of your video, then feed them alongside the original video and question to your working model. The paper shows this gives real gains without any training. Order matters, so keep segments chronological.
A few things to keep in mind:
•
The visual encoder is frozen throughout. Improvements come from language-model and projector updates, so if your bottleneck is perception rather than reasoning, this recipe won’t help as much.
•
The comparison to other spatial reasoning systems in Table 1 mixes architectures and training data, so the “best overall” claim is a reference point, not a controlled comparison. The controlled comparison is against each backbone’s own base model.
•
Closed-loop success rates are still low in absolute terms (14% overall on WalkerBench). The method makes models better at spatial reasoning, not competent embodied agents.
•
OPD needs a strong frozen VLM to write privileged traces during training. That’s a real cost, and the paper doesn’t quantify how trace quality affects the student.
•
The Camera Displacement gap the paper frames as “local vs long-horizon” (~60 points for GPT-5.5) is illustrated with one model on two related but different benchmarks (SAT-Real vs VSTI-Bench). Take it as a directional claim about difficulty, not a calibrated measurement.