RoTA-Bench tests whether vision-language models can rotate a 3D object to match a target 2D silhouette through repeated action-and-feedback steps, and finds frontier models top out around 17% task success while undergraduates hit 100%.
If you want a vision-language model to drive a robot arm, align a part in a CAD tool, or help a user orient something on screen, the model has to look at a 2D image, infer the 3D shape and its current orientation, pick a physical action, then look again and adjust. That closed loop is different from answering a question about a static picture.
Most spatial benchmarks test the static case: show one image, ask a question. The interactive benchmarks that do exist mostly measure web navigation, maze pathfinding, or sliding-tile puzzles, where success is dominated by planning and text reasoning rather than reading 3D orientation off a picture. The authors argue there’s a gap: no benchmark isolates “turn 2D observations into 3D control.” RoTA-Bench is their attempt to fill it.
Each task gives the model a target silhouette (a black-and-white shape) and a 3D object that starts rotated away from the pose that would produce that silhouette. The model picks one rotation at a time, sees the new view, and repeats until the current silhouette overlaps the target by at least 0.90 Intersection over Union or it runs out of steps (budget: 48 decisions).
The action menu is small and discrete: 24 rotations total, built from 6 directions (pitch up/down, yaw left/right, roll clockwise/counter-clockwise) times 4 step sizes (1°, 5°, 10°, 30°), plus a STOP action. At each step the model sees three panels side by side: the target silhouette, a depth-shaded auxiliary view of the current object, and the current silhouette.
To keep the task honest, the authors do not pull shapes from a 3D model library (where common objects might be memorized). Instead they take 2D vector icons from the open-source Game-icons collection and extrude each one into a 3D blob inside a sphere. The front view of this blob reproduces the original icon, which becomes the target. Initial poses are made by applying a random sequence of 6–16 reversible 30° rotations, which guarantees every task is solvable (just invert the sequence). The pipeline yields 674 tasks, with a frozen 100-task test set stratified by shortest-known solution length.
obs = render(target, current_pose) # 3 panels
history = []
for step in range(48):
action = model(obs, history) # e.g. (YAW_POS, 10)
if action == STOP: break
current_pose = rotate(current_pose, action)
obs = render(target, current_pose)
history.append((obs, action))
if iou(project(current_pose), target) >= 0.90: break
Success is scored on silhouette overlap, but the authors also log the true 3D 3D pose error (accounting for object symmetries) as a diagnostic. A model can improve silhouette overlap while actually rotating further from the correct pose, and vice versa.
Big human-model gap. Across 9 vision-language models plus two agent wrappers, the best single model is GPT-5.6-sol at 17% success, followed by Gemini 3.6 Flash at 10%. Claude Opus 5, Claude Sonnet 5, and Qwen 3.7 Plus score 0%. Twenty undergraduates ran the 100-task set four times (400 runs total) with 100% success and no practice trials. The ReAct and Magentic agent scaffolds built on GPT-5.6-Sol did not help (16% and 2%).
Two oracle baselines frame the gap. An oracle that greedily picks the rotation maximizing immediate silhouette overlap only solves 39% of tasks, so chasing visible overlap step by step is not enough. An oracle that picks the rotation minimizing true 3D pose error solves 100% in about 8 steps. The task is solvable with the given action space; the models just are not estimating 3D pose well enough to drive it.
Models over-weight silhouette, humans use both. On 211 states that a human and GPT-5.6-Sol both visited, the model improved silhouette overlap while making 3D pose worse in 19.4% of actions, versus 10.9% for humans. The Spearman correlation between chosen rotation angle and current overlap is stronger for the model than its correlation with true pose error; for humans both are strong.
Fine control depends on seeing history. With full inputs on a 30-task ablation subset, GPT-5.6-Sol uses the 1° step 12% of the time and the 30° step 63% of the time. Remove the interaction history (prior images and actions, keeping only the current panel) and 96.8% of actions collapse to 30°, with 1° adjustments effectively disappearing. Success drops from 5/30 to 0/30. Removing just the depth-shaded auxiliary view hurts less than removing history.
One model collapses entirely. SpatioLM-Understanding-InternVL3.5, a spatial-reasoning fine-tune, emits the same action (YAW_POS 10°) in 91% of its moves, scoring 0%.
•
If you are evaluating a VLM for an interactive 3D control use case (robotic manipulation, AR alignment, CAD assistance), the static spatial benchmark score is not a reliable proxy. Task success here is near-zero for most frontier models despite reasonable scores on static VQA, so running something closed-loop before committing is worth it.
•
The ablation suggests interaction history is load-bearing for fine-grained control, not just for high-level planning. If you are building an agent harness around a VLM and summarizing or dropping prior turns to save tokens, test whether that pushes the model toward coarse, uniform actions.
•
The gap between the two oracle baselines (39% vs 100%) means a plausible research direction is giving the model an explicit pose estimate rather than only pixels. The paper does not test this, but it is a natural next experiment: the action space is already expressive enough to solve every task.
•
The authors release task construction code, the evaluation pipeline, and the 674-task dataset. The specific repository URL is not given in the provided text.
•
The 3D shapes are extruded 2D icons, not real meshes. A side experiment on 30 Poly Haven meshes with matched difficulty gave GPT-5.6-Sol a higher mean final overlap (0.72 vs 0.57), suggesting familiar object categories are easier and that icon-blobs may understate performance on realistic geometry.
•
Human participants are 20 undergraduates with a mouse-and-button interface. They are not experts, but they are also not representative of all users, and the interface is more ergonomic than JSON.
•
Models are evaluated at temperature 0 with reasoning effort “medium” where applicable. The paper does not test whether higher reasoning budgets, chain-of-thought variants, or fine-tuning close the gap.
•
The 100% human success rate means the benchmark cannot resolve differences among skilled humans; it is calibrated to expose the model floor, not the human ceiling.