FactoSR teaches a vision-language model to reason about 3D scenes and motion by splitting one intractable “4D consistency” reward into three checkable pieces (cross-view point matching, depth ordering, and forward/backward motion agreement), trained with RL with verifiable rewards on top of Qwen3-VL.
You’ve shipped a multimodal assistant that takes a few photos of a room, or a short walkthrough video, and needs to answer questions like “is the chair closer than the table?”, “which way did the camera turn?”, or “is this the same mug from the earlier frame?”. Today’s VLMs handle these badly. They read each frame as a flat 2D picture and guess spatial relations from appearance cues (bigger = closer, similar color = same object), which breaks the moment viewpoint or lighting shifts.
The dominant response in the literature has been to throw more supervised spatial question-answer data at the model, or bolt on explicit 3D representations. This paper argues both routes plateau, because a static-image loss can’t teach a model that a scene has depth and time behind the pixels.
The framing: real spatial understanding needs three things a single camera image throws away, namely which pixel in view A corresponds to which pixel in view B (the XY plane), which object is in front of which (the Z axis), and how the camera moved between frames (the T axis). Optimizing one giant “be 4D-consistent” loss is intractable, so FactoSR breaks it into three checkable rewards and trains with reinforcement learning.
Training runs in two stages on top of Qwen3-VL. Stage 1 is supervised fine-tuning on a mix of general instruction data (LLaVA-OneVision) plus a new 1.2M-sample spatial dataset, first with short answers then with longer “anchor, transfer, verify” reasoning traces for cross-frame alignment. Stage 2 is RL with Group Relative Policy Optimization (GRPO) where the reward is a weighted sum of four verifiable signals, gated on the output being in the right format.
The three geometric rewards are the contribution:
•
XY reward. The model picks which of four candidate pixels in view 2 matches a reference pixel in view 1. Because the authors have ground-truth depth maps and camera poses, they can [[reprojection|reproject]] the reference pixel into view 2 and check the model’s pick against the true location. A visibility mask zeros out the reward if the target is occluded, so the model can’t get credit for a lucky guess in an unseen region.
•
Z reward. The model outputs 3D bounding boxes; predictions are matched to ground-truth boxes via Hungarian assignment, then the ordering of their depths is scored with the Kendall-tau rank correlation rank correlation. So the model is rewarded for getting front-to-back order right, not for hitting exact metric distances.
•
T reward. Each sample has a forward question (“how did the camera move from frame 1 to frame N?”) and its inverse (“how would you undo that motion?”). The model rolls out both and only gets reward if both answers are correct. This punishes shortcut answers that happen to match the forward label but don’t imply a physically reversible camera path.
for query in batch:
rollouts = policy.sample(query, n=8)
for y_hat in rollouts:
if not format_ok(y_hat): reward = 0; continue
r = l1*acc(y_hat, y)
if task == "xy": r += l2 * reproj_score(y_hat) * visible_mask(y_hat)
if task == "z": r += l3 * (kendall_tau(depths_hat, depths_gt)+1)/2
if task == "t": r += l4 * acc(y_hat,y) * acc(y_hat_inv, y_inv)
grpo_update(policy, rollouts, rewards)
The prevailing move in spatial VLM work is to either scale up spatial question-answer data or add a single outcome-correctness reward during RL. This paper shows the opposite. Don’t ask the model to learn “be 4D-consistent” from one final-answer reward. Factor the objective into the specific geometric properties a 2D camera destroyed (cross-view correspondence, depth order, time-reversibility) and reward each one with a check the physics allows you to verify. The load-bearing evidence isn’t the headline benchmark lift, it’s the ablation where plain Group Relative Policy Optimization (GRPO) with an accuracy-only reward barely moves the needle while each factorized reward moves its matched sub-metric.
The finding that carries the paper is the ablation. Plain Group Relative Policy Optimization (GRPO) over the SFT model gives only +0.2 average accuracy across nine spatial benchmarks, meaning generic RL on final-answer correctness barely helps. Turning on the three factorized rewards each move their targeted axis: the XY reward lifts correspondence tasks by +2.7, the Z reward lifts depth tasks by +1.3, and the T reward lifts camera-motion tasks by +7.9. A naive “grounding” reward that supervises box localization actually hurts overall accuracy by -0.2, which is why they switched to rewarding depth ordering via Kendall-tau rank correlation rather than metric depth.
Those mechanisms then show up on headline benchmarks. FactoSR-8B-RL scores 61.5 on VSI-Bench (+5.9 over the base Qwen3-VL-8B-Instruct) and 55.4 on All-Angles-Bench (+4.5), the best among open-source VLMs the authors compare against. General multimodal benchmarks (MMBench, MMStar, OCRBench) stay roughly flat, so the spatial training doesn’t visibly cannibalize general skills.
Reach for this recipe when you’re building a VLM-based agent that has to reason about physical scenes across multiple views or across time, and you already have (or can generate) synthetic scenes with ground-truth depth and camera poses. The pattern to steal: instead of rewarding “did you get the final answer right?”, write a reward per geometric invariant your data lets you verify. Reprojection consistency for correspondence, rank correlation for depth ordering, forward/inverse agreement for motion. Each one is cheap to compute and hard to game.
Code and models are released at GitHub. The 8.2M-sample SFT mix and the 32K RL set are described in aggregate in the paper; the exact release scope of the datasets isn’t spelled out in the text provided here.
When a single end-to-end reward can’t teach a physical invariant, split the invariant into pieces your simulator can check, and reward each piece. This works because RL with verifiable rewards is only as smart as the verifier; a factored verifier lets the model learn the mechanism instead of pattern-matching the label.
•
The rewards need ground-truth depth maps and camera poses at training time. If your domain doesn’t have a simulator or annotated RGB-D data, you can’t compute the XY or Z rewards as defined.
•
Gains on general multimodal benchmarks are flat to slightly negative (MMStar -1.0, OCRBench -2.7), so spatial specialization isn’t free even though the drop is small.
•
All results are on one 8B backbone (Qwen3-VL). Whether factorized rewards keep helping at larger scale, or on non-Qwen backbones, isn’t tested here.