HumanCLAW tests whether a frozen vision-language model can pilot a full human body by decoupling decision from motor control: the model picks atomic skills, a motion generator realizes them, and a half-physics simulator applies consequences. The best of nine frontier models finishes only 16.8% of find-navigate-sit episodes.
You’ve shipped an agent that drives a robot or avatar through a house from egocentric video. It has to find a couch and sit on it. When it fails, you can’t tell whether the model made a bad choice or the low-level controller lost balance on the rug. Every failure is ambiguous, so you can’t improve the right layer.
Existing work sits at two extremes. Symbolic benchmarks like ALFRED let the simulator resolve actions as scripts, so the VLM never has to deal with body dynamics. End-to-end Vision-Language-Action model policies learn motor control from robot data, but that entangles reasoning with a fitted controller. HumanCLAW is the middle: physical consequences are real, but balance and motor-tracking failures are factored out by construction.
Every half-second, the VLM sees an egocentric RGB frame plus a compact text history of what it just did. It’s prompted to describe visible objects and distances, pick a mid-level objective (approach, align, sit), then emit one atomic skill with parameters: walk(x,z,\psi), turn(\theta), sit(h), and so on. Atomic means unambiguous. “Sit on the sofa” is banned because it hides finding, approaching, and orienting inside one fuzzy command. Composition lives in the VLM’s reasoning, not in the skill.
A short skill-specific verifier then re-checks the proposal against the current view (would this walk hit a wall? is a stair actually visible?) and can swap the skill. This catches long-context drift where the planner hallucinates progress.
The skill call goes to a motion generator: a frozen Motion Base DiT trained on AMASS gives a general prior over natural human motion, and one ControlNet adapter adapter per skill steers it to the requested magnitude. New skills plug in without retraining the base or the VLM.
Finally a half-physics simulator on AI Habitat plays the motion as kinematic velocities against a rigid-body world. Walls block, objects get knocked over, gravity applies, but the body cannot fall over from bad motor control.
while not done:
obs = sim.render_ego()
visible = vlm.describe(obs, history)
mid_goal = vlm.plan(visible, instruction)
skill, params = vlm.pick_skill(mid_goal)
if not verifier.check(skill, params, obs):
skill, params = verifier.repair(...)
motion = generator(skill, params, body_state)
body_state, obs = sim.step(motion) # half-physics
history.append((visible, skill, params))
The usual read on VLM-driven agents is that the bottleneck is perception or planning: better grounding, better chain-of-thought. This paper shows the opposite. Once you factor out motor failures, current VLMs recognize targets fine but lose track of the body they’re driving. They can’t tell where their pelvis is, whether they’ve arrived, or that their leg is jammed against a chair. The load-bearing evidence is not the headline success rate but the funnel: recognition holds up while termination, arrival-detection, and body placement collapse.
The single most load-bearing result: once a target is genuinely rendered in the egocentric view, the strongest model acknowledges it within 5 percentage points of objective visibility. Recognition is essentially solved. Everything downstream breaks.
Secondary findings, all evidence for the same thesis:
•
Headline benchmark: on 1,218 find-navigate-sit episodes across 41 HSSD houses, the best model (Gemini-3.1) hits 16.8% full success. Four of nine models sit on the target in ≤0.2% of episodes.
•
Stage funnel: FindSR 32.6–64.9%, NavSR 0.8–42.4%, InteractSR 0–16.8%. Most of the task is lost between finding and reaching, then again between reaching and sitting.
•
Navigation failures: of episodes where the agent actively sees the target, 68% still fail to navigate. Root cause: 34% body-awareness (arrived but didn’t notice, or jammed but kept issuing forward), 30% distance hallucination (stopped and declared arrival while >0.2 m away). Only 9% are perception lapses.
•
Interaction failures: 81% are body-placement errors. 58% sit into thin air, lowering the pelvis where the body already stands with no seat under it.
•
Collisions cluster on unwatched parts: legs and arms 20–45% of steps, head under 7%.
•
Ablation: removing the verifier drops NavSR from 27% to 2% and InteractSR to 0, while FindSR barely moves. Removing the mid-level objective leaves navigation intact but collapses InteractSR to 0, since sitting is a several-second compositional routine.
Reach for this framing when you’re evaluating an embodied agent and can’t tell whether failures are decision-level or control-level. The recipe is: define an atomic skill vocabulary with deterministic text-to-motion realization, put a per-skill verifier between the planner and the actor, and run motion through half-physics so contact and gravity matter but balance doesn’t. Now every failed episode reads as a decision.
The paper releases HumanCLAW-Bench: 1,218 episodes over 41 HSSD houses with staged progressive metrics, difficulty tiers based on route geometry (distance, turns, clutter), and automated root-cause labels from deterministic rules over the logs. No repo URL is provided in the text; the paper doesn’t specify a release location for code or the trained motion prior.
A VLM can describe a scene it’s standing in without knowing where its own legs are. Perception and planning benchmarks miss this because they never make the model live with the physical consequence of its last action. If you want to measure embodied reasoning, you have to close the loop through a body and score whether the model tracks that body, not just the world.
•
The interaction vocabulary is tiny (walk, turn, side-step, step-back, stairs, sit). Richer manipulation would stress finer placement and might redistribute where failures land.
•
Half-physics factors out balance by design. A competent HumanCLAW decision maker is not yet a competent humanoid policy on real hardware, and the paper doesn’t claim otherwise.
•
The agent gets only egocentric RGB and text history, no proprioception or contact signal. The “embodied self-awareness” gap may be partly an interface problem (missing input channel), not purely a model limitation. The authors flag this explicitly.