Reflex teaches a humanoid to catch thrown boxes from onboard RGB-D by splitting the problem into three stages linked by a compact object-dynamics code, letting the robot catch at 76% of human success at 2.5 m and 59% at 5 m on a real Unitree G1.
If you want a humanoid to work around people (catch a tossed carton on a loading dock, step aside from a cart, grab a falling cup), it needs to react to a moving thing with its whole body within about a second, using only cameras it carries. Prior demos that look similar cheated on at least one of those axes: Motion capture rigs tracked the object, retroreflective markers were glued on, or the arm sat on a fixed base so balance wasn’t at stake. A thrown box is harder than catching a ball with hands because the robot has to position its torso, form a forearm cradle, absorb momentum, and not fall over, all while the box occludes itself near contact and the camera is moving with the robot.
The closest reference point in the paper is the general recipe of training a privileged teacher in simulation (one that sees the true object state) and distilling to a vision-based student, as in RMA (Rapid Motor Adaptation) for legged robots. That recipe works for static terrain but hadn’t been shown for a fast-moving object seen from a moving camera on a humanoid.
The core idea: don’t train pixels-to-joints end-to-end, because rendering RGB-D inside reinforcement learning is slow and the reward (did you catch it?) only fires after contact, which is terrible supervision for learning what a moving box looks like. Instead, split into three stages joined by an 8-dimensional latent vector that summarizes everything the body needs to know about the incoming box.
•
Stage 1, body skill. Train a controller with PPO in simulation that sees the true box position and velocity. It learns to step into place, form a forearm cradle against the torso, absorb impact, and stay standing. Human motion clips shape the style via an AMP (Adversarial Motion Priors) reward so motions look natural and coordinated.
•
Stage 2, object dynamics as a latent. Freeze the Stage 1 teacher. Train a small encoder that takes a delayed, noisy, sometimes-missing box observation and squeezes it into an 8-dim code in [-1, 1]. The decoder (initialized from the teacher) consumes this code plus robot state and outputs joint targets. Initialization uses Behavior cloning against teacher actions, then PPO refines encoder and decoder jointly on the catching reward. Observation latency is randomized 20–60 ms so the controller learns to anticipate, not just react.
•
Stage 3, grounding the latent in vision. Freeze encoder and decoder. Train a vision module to predict the same 8-dim latent from a 25-frame history of RGB-D plus camera ego-motion. A CNN produces a box heatmap, a geometric layer turns that into image coordinates and a backprojected 3D point, and a GRU (Gated Recurrent Unit) integrates these over time to recover motion. DAgger is used: run the current visual policy, label the frames it actually visits with the privileged encoder’s latent, retrain.
Trained in NVIDIA’s Isaac Lab simulator. The payoff of the split: Stage 2’s PPO runs rendering-free at ~28k env steps/s, and Stage 3 only needs ~1.6M rendered steps instead of tens of millions.
# Deployment-time control loop (one step, 20 ms)
rgbd_hist = camera.last_k_frames(K=25)
ego_hist = imu.last_k_egomotion(K=25)
z_hat = visual_encoder(rgbd_hist, ego_hist) # 8-dim latent
robot_state = proprioception() # 160-dim
action = frozen_decoder(robot_state, z_hat) # 29 joint targets
robot.send_joint_targets(action)
•
Vision barely trails privileged input in sim. Visual Reflex catches 85.7% of throws; the same decoder fed the ground-truth box latent catches 87.9%. Most of the catching ability survives swapping true state for predicted state.
•
Temporal history is the single most important input. A single-frame visual encoder loses 35.6 points on centered throws and 39.8 points on 0.6 m-offset throws. Removing camera ego-motion costs 10–23 points. A single still frame cannot see motion, and the controller depends on inferred motion.
•
The latent target beats regressing raw state. Training the vision module to predict the 26-dim privileged box observation (position, velocity, orientation, size, etc.) instead of the 8-dim learned latent loses ~19 points. The latent discards whatever the body doesn’t actually use.
•
Behavior-cloning initialization helps. Starting Stage 2 by action-matching the teacher, rather than mapping teacher weights by coordinate, improves visual catch rate by 5.2 points on average across six seeds.
•
Latency randomization helps at every tested delay. A controller trained with randomized 20–60 ms observation lag is 5–11 points above an un-randomized one across simulated delays, and peaks at 60 ms instead of at zero, suggesting it learned to compensate rather than ignore delay.
•
Real Unitree G1 numbers. 20 trials per condition, so the error bars are ~±20 points. 65% catches at 2.5 m, 40% at 5 m, 15% at 7 m. Three human catchers hit 85.0 / 68.3 / 28.3% at the same distances; Reflex reaches 76% and 59% of human success at 2.5 and 5 m. 7 m is near the practical limit for both humans and the robot.
•
Generalization is uneven. Payloads 0.5–2 kg work (30–60% real, 78–91% sim). Boxes outside the trained size range are often intercepted but not retained: the robot gets a forearm on them but doesn’t hold on. The controller has no size input, so this isn’t surprising.
•
A learned latent beats detector+Kalman, within its training range. With the same CNN detector feeding either a tuned Kalman filter + state policy or the learned decoder, the learned latent wins on 13/13 condition cells inside the Stage 3 throw distribution. Beyond that range (5–7 m throws not in Stage 3 data) the filter wins, and adding far throws to Stage 3 closes the gap.
•
If you’re building a reactive humanoid skill and currently training vision end-to-end with RL, the staged decomposition is the actionable lesson: let a privileged teacher learn the motor skill with cheap simulation, then regress a small latent from vision while the body controller stays frozen. This is worth testing when your reward is sparse and rendering is your rollout bottleneck.
•
If your student is currently regressing the full physical state of the object (position, velocity, orientation, size), try regressing a learned task-specific latent instead. The ablation suggests the latent throws away what the body doesn’t need, which is easier to recover from pixels.
•
For any pipeline with real sensor delay, randomize observation latency during policy training across a range that brackets your deployed p50 latency. The paper’s hardware pipeline runs at a 69 ms median camera-to-control latency, and the latency-randomized controller is strictly better across the tested range.
•
Honest scope: this is one task (catching boxes), 20 trials per real-robot condition, no public code link in the provided text, and the controller is size-blind. If your downstream task needs size-conditioned behavior (grasping variable objects, say), you’d need to add that input and retrain. The approach needs a simulator in which you can train a privileged teacher; it isn’t a drop-in for pure imitation-from-video setups.
•
The real-robot numbers come from 20 trials per cell, giving 95% intervals up to ±20 points. Treat the human-success-ratio claims as directional.
•
Long-range throws are the weak spot. At 5 m, visual catch success drops to 43% in sim vs. 79.7% with privileged input, because Stage 3 data only covered 3.0–3.5 m throws. Fixing this is a dataset problem, but the authors only verified the fix in sim, not on hardware.
•
Rendering omits motion blur, auto-exposure dynamics, and realistic depth artifacts; sim-to-real for appearance leaned on color augmentation of the box.
•
The hardware latency measurements come from the detector-based baseline pipeline, not the full Reflex visual encoder. The 69 ms median actually exceeds the 20–60 ms range used during latency randomization, so the sim numbers at zero added delay are probably optimistic for the real robot.
•
“Catch success” is a chest-cradle criterion; downstream carrying and handover degrade quickly (65% → 35% → 30% → 25% over the four-stage pipeline in 20 trials).