Show-Harness lets a general-purpose VLM drive a robot arm by choosing among about a dozen Semantic action units like MV_LEFT or GRASP, which an embodiment-specific interpreter turns into small, bounded Cartesian moves. The model stays in the physical control loop without learning continuous motor outputs.
Say you want a robot arm to pick up a banana and put it on a plate. Today you have two unpleasant options. Option one is a Vision-Language-Action model model like π0.5 or GR00T action head: you fine-tune a big VLM to regress joint targets or motor tokens, and it works, but the model now speaks robot-specific numbers instead of language, and you re-collect trajectories for each new arm or task. Option two is hierarchical: the VLM emits high-level calls like place(banana, plate), and a separate controller executes them. That preserves the VLM’s smarts but hands physical execution to an opaque skill library you have to engineer per system.
Show-Harness sits between these. The VLM keeps deciding physical moves at every step, but only in a small vocabulary it can already reason about. The claim is that most of a foundation model’s embodied competence is bottlenecked by interface design, not model capacity.
The action vocabulary has three flavors: six translational moves (MV_FWD, MV_BACK, MV_LEFT, MV_RIGHT, MV_UP, MV_DOWN), rotations about x/y/z axes, GRASP/RELEASE, and DONE. Directions are defined relative to a chosen camera view, so “left” means left in the image. Each unit triggers a fixed small displacement (2 cm by default) or a 15° rotation. An embodiment-specific interpreter maps that unit into an actual pose setpoint for whichever arm you have, and enforces workspace safety limits. Swap arms, swap interpreters; the model-facing interface is unchanged.
Around the action layer sits a perceive-reason-act loop with plugins: multi-view camera guidance, textual Proprioception (gripper height, contact state), subtask decomposition, an action-chunking mode that lets the model queue a few moves open-loop when the target is far, an adaptive step-size switch (4 cm coarse, 2 cm fine near the target), a short action history to prevent oscillation, and a grasp-failure detector that rewinds and retries.
The same interface supports two deployment modes. ZS mode wraps a frontier model like Gemini 3.1 Pro with no fine-tuning. FT mode takes a small open VLM (default Qwen3.5-2B) and applies LoRA adapters (rank 64) to the language layers, freezing vision. Training is next-token cross-entropy over action symbols using the model’s normal vocabulary, so no custom action head. The authors report it runs in under 2 hours on one H200.
while not done:
obs = capture_multiview() + proprioception()
ctx = apply_plugins(instruction, obs, history)
action = vlm.predict(ctx) # e.g. "MV_DOWN" or "GRASP"
setpoint = interpreter[embodiment](action, setpoint)
robot.execute(setpoint)
history.append(action)
Demonstrations for FT mode are collected through GUMI, a GUI where a human presses keys (or a computer-use agent clicks buttons) mapped to the same semantic units, so one demo trains both semantic-action and continuous-control policies.
Evaluation is on a real 7-DoF Franka arm and a bimanual AgileX bimanual rig rig, with 10 pick-and-place tasks (five objects × two receptacles) plus targeted scenarios. Baselines are two VLAs (π0.5, GR00T action head), two VLA-plus-agent-wrapper systems (Harness-VLA, Goal-VLA), and two code-as-policy agents (CaP-X, RATS). All trainable methods use the same 164 real-robot episodes (~7.8K steps).
Headline: both ZS and FT beat baselines across cross-task, cross-environment, and cross-embodiment shifts (Table 2, exact numbers in the paper). More informative are the targeted ablations:
•
Fine-grained control: shrinking the interpreter step from 2 cm to 1 cm, with no retraining, lifts ZS on stacking/insertion from 60% to 82% and FT from 40% to 65%. π0.5 on the same demonstrations gets 18%, reaching 62% only after extra fine-grained training data.
•
Rotation extrapolation: trained only on 0° and 45° carrot orientations, FT reaches 70% at an unseen 90° via composition of 15° rotation units; π0.5 reaches 20%.
•
Reasoning tasks (find a block hidden under one of three cups; arrange letters into “SHOW”): ZS with situated planning hits 85%, FT hits 10%, π0.5 hits 0%. Feeding Gemini-generated subtask instructions raises FT to 70% but π0.5 only to 5%.
•
Action-space ablation: when action names are replaced with arbitrary symbols but the written conventions describing their physical effects are kept, performance nearly matches the default. Strip the conventions and let the agent probe symbols to infer their effects, and success collapses to 1/20 with only 23% of inferred mappings correct. The takeaway the authors offer: explicit written conventions carry most of the grounding; semantic names are a nice prior but not the load-bearing piece.
•
Plugin leave-one-out: removing subtask planning drops success to 60%, removing failure recovery drops it to 72%, removing action history increases oscillation timeouts. Visual Prompt and Situated Planning are off by default and only help on their targeted scenarios.
One honest observation: increasing frontier-model “thinking effort” (reasoning budget) mainly cuts redundant steps rather than lifting success, and can cost 3.4× wall-clock on one tested backbone.
•
If you have a manipulation setup with parallel-jaw grippers and a decent camera rig, and you want to skip VLA fine-tuning entirely, the ZS recipe (frontier VLM + this harness + an interpreter you write for your arm) is worth trying before committing to a trained policy. The paper’s evidence for zero-shot Franka and AgileX control is direct.
•
If you already have a small open VLM in your stack, the FT recipe is a low-cost baseline: ~8K decision-step demos, LoRA on the language layers only, a couple of GPU-hours. Worth testing against a VLA trained on the same trajectories, especially if you care about out-of-distribution objects or a different arm.
•
The action-representation ablation is the most portable lesson even if you never touch a robot: when giving an LLM a custom action vocabulary, the written description of what each action does matters more than whether the names are semantically suggestive. Symbolic labels with clear conventions worked; semantic labels without conventions were weaker.
•
Data collection through a keyboard GUI, without teleoperation hardware, is the artifact most likely to be immediately reusable. Demos and code are released (data).
All evaluation is on parallel-jaw grippers doing pick-and-place, stacking, peg insertion, and a few dual-arm tasks. Nothing here tests dexterous hands, humanoids, contact-rich assembly, or tasks needing force feedback; the authors flag this explicitly. The step-size and rotation-increment choices (2 cm, 15°) are calibrated for these tasks, and finer manipulation needed a manual step-size change. Trial counts are modest (10 per task, 20 for targeted scenarios), so treat single-percentage-point comparisons cautiously. The strongest ZS numbers depend on a frontier API model; costs, latency, and provider stability are real deployment factors the paper does not price out. Finally, the baseline VLAs are trained on the same 164-episode set as FT, which is small for continuous-action regression, so the VLA comparisons show what happens in low-data regimes rather than at VLA-native data scale.