GST-Bench tests whether vision-language models can build a globally consistent map from a long egocentric walkthrough video, then answer spatial questions from a fresh off-trajectory viewpoint. The strongest model scores 42.68 vs. 79.08 for humans, and controlled ablations show the bottleneck is cross-frame integration, not single-image perception.
Imagine you’re building a household or warehouse robot that watches a walkthrough of a room, then gets asked “from where I’m standing now, which direction is the kettle?” The kettle isn’t in view. The robot has to remember it from earlier video and reason about its own new position. That’s the workload this paper isolates.
Prior spatial benchmarks mostly test single images or a handful of views, or when they use video, they mix in questions answerable from one lucky frame. The closest recent effort is VSI-Bench, which does use egocentric video but doesn’t cleanly separate questions requiring cross-frame reasoning from ones a single frame can solve. GST-Bench is built so every question forces cross-frame integration.
The benchmark is constructed in the OmniGibson simulator over 50 indoor scenes from BEHAVIOR-1K. For each scene the pipeline renders an egocentric exploration video, then samples a “current view” from a point that is not on the video’s camera path. That off-trajectory query view is the anchor for questions.
To force global reasoning, the pipeline enforces two hard constraints. The target object must appear somewhere in the exploration video but must be invisible from the current view. And single-frame-solvable questions (like object size) are excluded entirely. So the model can’t shortcut by matching the query image to a video frame or by recognizing the target locally.
Questions fall into three competencies with 12 subtasks total: self-localization (place yourself on a top-down map), object-localization (where is object X relative to you now, in degrees and meters), and scene structure (pick the top-down map that matches the video). Top-down maps come in three abstraction levels: photo-real, occupancy footprint, bare floor plan. Answers are graded numerically (angular error, relative distance error, pixel distance on the map) rather than left/right/front/back categories.
for scene in behavior_1k_scenes:
video = render_egocentric_walkthrough(scene)
query_view = sample_offtrajectory_viewpoint(scene)
for target in objects_in(scene):
if visible_in(target, video) and not visible_in(target, query_view):
angle, dist = geometry(query_view, target)
qa_pairs.append((video, query_view, target, angle, dist))
qa_pairs = human_verify(auto_filter(qa_pairs))
Every sample is human-verified for answerability, yielding 2,762 questions over 6,790 minutes of video. The authors also release GST-Train, built by the same pipeline on disjoint scenes, for supervised fine-tuning.
The common assumption when a VLM stumbles on spatial video tasks is that it needs sharper single-frame perception, more 3D pretraining, or embodied post-training on affordances. This paper shows the opposite for frontier models. The frontier bottleneck is not seeing a frame, it is stitching many frames into one persistent map; embodied fine-tuning as currently practiced doesn’t fix this and open models fail at both stages at once. The evidence is the Local-Video and Local-Image ablations that hold the spatial question fixed while removing the need for cross-frame integration.
The load-bearing finding is the ablation. On egocentric-direction questions, when the target is made visible in the current view so no video integration is needed, Gemini 3 Pro jumps from 22.11 to 61.20 (+39 points) and GPT-5 from 31.09 to 65.61. On egocentric-distance, the two Gemini variants gain +18 to +28 points under the same relaxation. Proprietary models can see; they cannot integrate a video into a consistent scene.
Open-source models tell a different story. Under the same local relaxation, two of four tested open models actually get worse on direction, and even the best local scores stay far below proprietary. Their failure is at both perception and integration.
Secondary evidence supporting the mechanism:
•
Headline gap: best zero-shot model Gemini-3-Pro at 42.68, human at 79.08, gap of 36.4 points.
•
Open-source ceiling is InternVL3.5-38B at 30.71 and Qwen3-VL-32B at 30.43, and even those leads come mostly from the easy top-down-selection subtask. On the other ten subtasks they sit within a few points of random guessing (20.01).
•
Embodied-tuned models (RoboBrain2.5, Robix, Cosmos-Reason2) do no better than their general-purpose backbones at matched scale. RoboBrain2.5-8B (24.61) trails Qwen3-VL-8B (25.89).
•
Fine-tuning Qwen3-VL-8B on GST-Train lifts it from 25.89 to 53.52, surpassing every zero-shot proprietary model, though still 25 points shy of humans.
Reach for this benchmark when you’re building an embodied or navigation agent, or evaluating a VLM you plan to put behind one. If your product needs the model to remember where it saw something and reason about a new viewpoint (robot fetch tasks, AR wayfinding, drone inspection recall), GST-Bench is a targeted probe. The Local-Video / Local-Image variants are especially useful as a diagnostic: run all three settings and you learn whether your model’s failure is perception or memory.
The paper says it releases GST-Bench (2,762 verified questions), GST-Bench-Local variants, and GST-Train for fine-tuning. The paper does not include a public repository URL in the provided text, so the release location is not specified here.
Frontier VLMs can see a room; they cannot yet remember one. Before adding more embodied post-training, check whether your failures survive when the target is placed back in view. If they vanish, you have a video-memory problem, not a perception problem, and no amount of local affordance tuning will close it.
•
Everything is rendered in simulation. Real robot video has motion blur, exposure shifts, and imperfect odometry that could change which failure mode dominates.
•
Human baselines are computed on only 20 samples per task, and on egocentric absolute distance humans themselves score ~41, so “human-level” is not a uniformly high bar.
•
The fine-tuning demonstration uses a single 8B model on data from the same pipeline as the eval (scenes disjoint, but distribution shared), so the 25.89 → 53.52 gain partly reflects distribution match and may not transfer to real footage.