ZimaBlue trains a robot policy by predicting future video before predicting actions, and scaling action-free egocentric video from ~300 hours to 120,000 hours lifts zero-shot real-robot success from 36.1% to 77.8%, most sharply when scenes are perturbed.
You’ve shipped a robot manipulation policy. It works on the tasks you demonstrated, but the moment lighting changes, a distractor appears, or the object sits differently, success collapses. Today the answer is: collect more teleoperated demonstrations on your robot. That data is slow and expensive, and the diversity ceiling is low.
The dominant alternative is the Vision-Language-Action model family: extend a vision-language model with an action head and train on labeled robot trajectories. It inherits semantic priors from the web but still learns physics only from action-labeled data. This paper argues the scalable substrate for physics isn’t more robot trajectories, it’s first-person human video, which is abundant and captures contact, tool use, and failure recovery. The question is how to convert action-free video into control.
The core object is a World Action Model (World-Action Model): a single generative network that jointly predicts future video frames and future actions. Because video prediction supervises every frame transition, the model learns physical dynamics from unlabeled footage, then only needs a smaller amount of action-labeled data to connect those dynamics to motor commands.
Training runs in three stages on a data pyramid. Stage 1 does causal video pretraining on ~120K hours of egocentric human plus robot video, initialized from the Wan2.2-TI2V-5B video generator. The model just learns to denoise future frames from past ones. Stage 2 (mid-training) adds ~6K hours of multi-robot trajectories with a unified 100-dimensional state-action space where each slot has a fixed physical meaning (left end-effector pose, right gripper, torso, hand joints, etc.) and inactive slots are masked. This lets a Franka arm and a bimanual robot share one model. Stage 3 specializes to the target robot.
All three stages use Flow matching as the training objective, applied to video latents in Stage 1 and to both video and action tokens jointly in Stage 2.
Running a 5B video diffusion model at 30 Hz is impossible, so inference uses a Slow-Fast dual system. The Slow branch (5B) runs infrequently, produces future-video predictions, and exports its per-layer video key/value cache. The Fast branch (0.5B) runs at control rate, takes the latest observation, and cross-attends to the cached Slow features to emit actions. This is trained with Real-Time Chunking so Fast can produce a coherent continuation while Slow is still thinking.
# Deployment loop (simplified)
slow_kv = None
prefix = []
while running:
obs, state = read_sensors()
if slow_ready(): # low frequency, async
slow_kv = slow_dit(obs, state, instruction) # 2 denoising steps
action_chunk = fast_dit(obs, state, slow_kv,
prefix=prefix) # 2 denoising steps
execute(action_chunk)
prefix = tail_of(action_chunk)
Distribution Matching Distillation (DMD) compresses each branch from 8 denoising steps to 2, and CUDA graph compilation drops end-to-end latency to 33 ms on an RTX 4090.
The prevailing route to a generalist robot policy is scaling action-labeled robot demonstrations, whether teleoperated or simulated, and hoping physical priors emerge from that. This paper takes the opposite bet. Physics should be learned from the largest available embodied data source, which is first-person video without action labels, and action supervision should only be asked to bind those learned dynamics to a control interface. The evidence that matters isn’t the leaderboard number, it’s the monotonic zero-shot scaling curve as the video corpus grows from 300 hours to 120K hours under a fixed target-robot post-training budget.
The load-bearing result is the controlled real-robot scaling ladder, all sharing identical DROID post-training. Target-robot only reaches 36.1% on 12 zero-shot tasks. Adding 6K hours of multi-embodiment trajectories lifts it to 46.1%. Adding 60K hours of egocentric video reaches 66.9%. Doubling that to 120K hours reaches 77.8%. The perturbed-scene subset gains most from video: going from 60K to 120K hours only adds 5 points on standard tasks but 22.5 points on perturbed ones (35.0% to 57.5%), which is the mechanism claim, video priors buy visual robustness.
•
On LIBERO-Plus, ZimaBlue reaches 86.7% zero-shot and 92.0% with fine-tuning, competitive with InternVLA-A1.5.
•
On RoboTwin 2.0, 94.5% average across 50 bimanual tasks, best among reported WAMs.
•
On RoboCasa365, 49.5% overall, second only to a VLA trained on 100,000 hours of real robot data while ZimaBlue used only 6K hours of robot data. On the unseen-composite split, ZimaBlue more than doubles the next WAM (16.5% vs 7.9%).
•
The Slow-Fast dual system alone lifts standard-suite success from 80.8% (Slow only) to 87.9%, and perturbed from 30.0% to 57.5%, showing the fast reactive loop matters most under distribution shift.
•
Acceleration: Slow-only latency 449.6 ms, async dual-system 145.6 ms, with distillation and compile 33.0 ms, a 13.6× overall speedup, at a 2.8-point success cost.
Reach for this design when you’re shipping a manipulation policy and your bottleneck is diversity of physical scenes rather than diversity of language instructions. The recipe: pretrain a video model on first-person footage of the domain you care about (kitchen work, warehouse handling), then add a modest slice of action-labeled trajectories with a shared state-action schema, then specialize to your robot. If latency is your constraint, the Slow-Fast split is a general pattern: run the heavy generative predictor asynchronously, let a small head cross-attend to its cached features at control rate.
The authors release code and a project page at GitHub and the project site. Training uses a mix of public datasets (EPIC-KITCHENS, EgoDex, DROID, AgiBot, plus proprietary egocentric video); the proprietary portion isn’t released. The unified 100-dim state-action interface is directly reusable for anyone mixing heterogeneous robot datasets.
When robot data is the bottleneck, scale the video corpus, not the demonstrations. Action-free first-person video teaches physics; a small amount of action-labeled data is enough to bolt on control. The catch: this only pays off if you can afford a generative predictor at inference, which is why the Slow-Fast dual system is not decoration but the price of admission.
•
The dramatic scaling curve is measured on a single robot platform (Franka arm on DROID) and 12 held-out tasks. Whether the 300-hour to 120K-hour trend holds on very different embodiments (mobile bases, dexterous hands) isn’t shown at that scale.
•
Camera-viewpoint robustness is the weakest axis in zero-shot LIBERO-Plus (58.1%) and only recovers with in-distribution fine-tuning. If your deployment involves novel camera placements, video pretraining alone doesn’t fix it.
•
The 120K-hour corpus mixes public datasets with unspecified proprietary egocentric video. Reproducing the top of the scaling curve from public data alone isn’t guaranteed, and the paper doesn’t quantify how much of the gain comes from the proprietary slice.