CosmoH2G transfers human hand demonstrations to a two-finger robot gripper by training a diffusion model on paired hand-and-gripper videos, then splitting generation into keyframes first and a full trajectory second, so the gripper can follow rotations and flips instead of just planar pick-and-place.
Say you want a robot arm to pick up a bottle, rotate it 90 degrees, and slot it upright into a crowded shelf. You’d like to just record yourself doing it once and have the robot copy the motion. The cheap way people try this today: record video of a human hand, then either (a) map hand keypoints onto gripper keypoints with hand-written rules, or (b) track a 2D or 3D point flow of the object and have the robot follow that flow.
Both break on anything that involves real spatial reorientation. Rule-based retargeting (e.g. Phantom, which picks the midpoint between thumb and index finger as the grasp point) assumes a pinch grasp and a hand shaped like the gripper, so it fails on power grasps and flips. Trajectory-following methods rely on point trackers like CoTracker, and the authors show that when the object rotates, the tracked points get occluded and drift onto the wrong part of the object, corrupting the trajectory. The result: existing hand-to-gripper datasets and methods are stuck on flat pushing and pick-and-place with little reorientation.
The paper has two contributions that lean on each other: a paired dataset, and a two-stage model that only works because of the paired dataset.
The dataset. A human does a manipulation with their bare hand while an RGB-D camera records. Then the same human does the same manipulation holding a UMI (Universal Manipulation Interface), which is basically a parallel-jaw gripper on a stick with a camera, designed so the recorded motion of the stick is exactly what a real robot arm would need to reproduce. This gives you hand video and gripper video of the same task, in the same scene, roughly aligned in time. They then run off-the-shelf hand reconstruction (WiLoR) and 6-DoF pose tracking (FoundationPose dataset) to lift both streams into 3D: object point clouds, hand meshes, and gripper 6-DoF pose sequences. Pairs where the hand and gripper trajectories don’t align well (similarity below 0.9) are thrown out. Final size: 6,189 episodes across 1,254 objects, deliberately biased toward flips and rotations.
The model. Naively, you’d train one network to eat the hand video and emit the full gripper pose sequence. The authors tried this (see ablation) and it drifts: small per-frame errors compound over a long rotation and the gripper misses the object. Their fix is factorization.
•
Stage I predicts only two poses: where the gripper should be at the first frame (initial grasp) and the last frame (final placement). Just those two 6-DoF poses, conditioned on object points, hand points, and a coarse contact map indicating which region of the object the hand touched. This is a Diffusion Transformer (DiT) trained with a standard denoising loss.
•
Stage II fills in the sequence between those two keyframes, but only the orientation part of each intermediate pose. The position of the gripper at each frame is not learned. Instead, they take the hand’s contact-region center at each frame and reuse it as the gripper’s position, relying on the assumption that hand and gripper grasp the same spot on the object.
•
The resulting position sequence is then cleaned up with a three-step optimization: adjust start/end positions to avoid object penetration, smooth the intermediate positions, and run inverse kinematics so the arm can actually reach each pose.
One small but load-bearing choice: they represent each gripper pose as three 3D points (center, left fingertip, right fingertip) rather than a rotation matrix or quaternion. The condition (object points, hand points) and the prediction target then live in the same 3D coordinate space, which the ablation shows matters.
# Inference sketch
obj_pts, hand_seq, contact_map = extract_3d(rgbd_video)
pose_start = stage1(obj_pts, hand_seq[0], contact_map) # 6-DoF
pose_end = stage1(obj_pts, hand_seq[-1], contact_map)
orient_seq = stage2(obj_pts, hand_seq, contact_map,
pose_start, pose_end) # rotations only
pos_seq = hand_contact_centers(hand_seq) # borrow from hand
pos_seq = optimize(pos_seq, grasp_stability, smoothness, ik)
gripper_seq = combine(pos_seq, orient_seq)
Evaluation is on 186 held-out episodes with 40 unseen objects, run both in the GalaxeaManipSim simulator and on a real Galaxea R1 Lite robot. The four metrics: grasp orientation error in degrees (GOA, lower better), success rate (SR), trajectory similarity to the demonstration via Dynamic Time Warping (TS), and final placement orientation error (TOPA, lower better).
•
Against the strongest learning baseline Track2Act (retrained on the same dataset), CosmoH2G improves simulation success rate from 77.4% to 83.9% and real-robot success from 60.2% to 70.4%. Grasp orientation error drops from 10.67\u00b0 to 7.53\u00b0. Optimization-based baselines MimicFunc and 3DFlowAction do substantially worse because they inherit tracker failures under rotation.
•
The single-versus-two-stage ablation is the most informative. Swapping in a single-stage Diffusion Policy or Action Chunking with Transformers (both standard imitation-learning architectures) collapses success rate to 58\u201365% and roughly quadruples placement-orientation error (TOPA jumps from 10\u00b0 to 48\u201353\u00b0). This is the paper’s evidence that the keyframe-first split is doing real work, not just the dataset.
•
Removing the ending-frame keyframe conditioning (Stage II sees only the start pose) doubles TOPA from 10.3\u00b0 to 21.8\u00b0, consistent with the story that the terminal keyframe is what pins down the final placement orientation.
•
The keypoint pose representation beats matrix and quaternion representations by 4\u20136\u00b0 on GOA.
•
Generalization: works across adult male and female demonstrators (SR 83\u201392%), degrades sharply on children’s hands (SR 33%) because they fall outside the training hand-scale distribution. Stable across four camera viewpoints since the pipeline operates in 3D after reconstruction.
Worth noting what these numbers do not establish: the test set is pick-and-place with unseen hand motions and unseen objects, not fundamentally new task types. “Complex spatial movement” here means rotation and flipping during transport, not, say, assembly or tool use.
•
If you’re building an imitation-learning system for a parallel-jaw gripper and hitting failures on any task that involves reorienting the object mid-motion, the factorization here is worth stealing even without their dataset: predict the two boundary poses with one model, generate only orientations for the middle, and borrow positions from the human hand trajectory. The single-stage ablation suggests this split matters more than the specific architecture.
•
If you’re collecting your own hand-to-robot data, the UMI (Universal Manipulation Interface)-based protocol (human does task, then repeats with handheld gripper, keep only pairs whose trajectories align above 0.9 similarity) is a concrete recipe. The authors will release dataset, pipeline, and code at the project page.
•
If you were planning to use rule-based hand-to-gripper retargeting for anything beyond pinch grasps, the Phantom failure figure is a useful counter-example to point at. The paper’s own comparison excludes these methods entirely because they couldn’t even grasp the objects.
•
Worth testing before committing: whether the assumption “hand contact center = gripper contact center” holds for your object shapes. It works here for household objects grasped by two-finger grippers; it may not hold for very small or very large objects, or for multi-finger hands.
•
Open-loop only. The gripper executes the generated sequence without visual feedback or collision avoidance during execution. Real-robot failures the authors report (objects slipping during rotation, dragging on the table) are exactly the kind that closed-loop control would catch.
•
The position-borrowing trick assumes the hand and gripper contact the same object region. When that assumption breaks (e.g., a grasp the two-finger gripper geometrically can’t replicate), no amount of orientation modeling saves you.
•
Evaluated task family is pick-and-place with reorientation. Broader task types (insertion, articulated objects, bimanual coordination) are explicitly left to future work.
•
Hand-scale generalization is bounded by training distribution: children’s hands drop success rate from 84% to 33%.
•
The dataset pipeline depends on several off-the-shelf perception models (FoundationPose dataset, WiLoR, SAM2) working well enough on your scenes. Their failure modes propagate into training labels.