RoboTok retrieves human manipulation demos from internet video by embedding 3D hand trajectories in an actor-centered frame instead of matching pixels or captions, lifting retrieval mean Average Precision from ~0.007 (best prior) to 0.353 against a Dynamic Time Warping oracle.
You’re training a dexterous-hand robot policy and you need thousands of examples of, say, “turning a bottle cap.” Collecting that on real hardware is slow and expensive, and existing robot datasets don’t cover the long tail of objects and grips you actually care about. YouTube has millions of humans doing exactly these motions, but until now, finding the motion-relevant clips has meant either language search (which returns visually similar but kinematically unrelated clips) or optical-flow matching (which is dominated by camera and scene appearance). The dominant behavior-aware baseline, STRAP, aligns visual-foundation-model features over time; it still barely beats random on this task.
The core move is to stop comparing videos in pixel space and start comparing them in the space of how the hands moved relative to the person’s body. RoboTok processes web clips through a fixed pipeline: filter for near-static camera and visible hands, run WiLoR to get 3D hand keypoints at 5 fps, ground them in metric depth using MoGe-2, and infill gaps with HaWoR. Because the actor’s torso is often off-screen, they train a small model that predicts a static torso reference frame from only the wrist trajectories, giving an egocentric coordinate system without needing the body to be visible. This canonicalized 3D hand trajectory is what gets compared.
The similarity oracle is Dynamic Time Warping on 21-joint hand poses: it aligns two sequences allowing local speed differences, then normalizes by length. Computing Dynamic Time Warping between a query and every clip in a 100k corpus is infeasible online, so they train a lightweight cross-attention encoder to produce an L2-normalized embedding where cosine similarity approximates the Dynamic Time Warping ranking. Two tricks make training work at scale: batches are built around anchor clips with their top-20 Dynamic Time Warping neighbors plus a boundary negative just outside that set, and the loss combines a set term (top-K should score above negatives) with a rank term (positives should be ordered like Dynamic Time Warping orders them).
for clip in web_videos: # offline indexing
if not (static_cam(clip) and hands_visible(clip)): continue
hands_3d = metric_depth(wilor(clip))
traj = to_torso_frame(hands_3d)
index.add(encoder(traj)) # single forward pass
def query(q_clip): # online retrieval
q = encoder(to_torso_frame(metric_depth(wilor(q_clip))))
return index.top_k_cosine(q, k=20)
The prevailing approach to mining web video for robots is to match on appearance or language, or at best on 2D optical flow in the camera frame. This paper shows the opposite: manipulation similarity lives in 3D hand motion expressed relative to the actor, and once you canonicalize into that frame a tiny encoder trained against a Dynamic Time Warping oracle beats visual-foundation-model retrieval by orders of magnitude. The load-bearing evidence isn’t the headline retrieval number, it’s that the same ordering of methods holds on an out-of-distribution assembly dataset with sensor-grade hand poses.
•
Retrieval quality on the 100k held-out queries collapses the baselines. STRAP hits mean Average Precision@20 = 0.007; FlowRetrieval and HAND sit near random. RoboTok reaches mean Average Precision@20 = 0.353 and Recall@20 = 0.996, meaning almost every query recovers a true Dynamic Time Warping neighbor within its top 20. Mean Dynamic Time Warping cost of RoboTok’s top 20 is 1.333 m against 1.145 m for the oracle neighbors and 4.776 m for random.
•
The ranking generalizes to AssemblyHands, an external two-hand assembly corpus with ground-truth 3D hand poses. RoboTok leads on every metric (mean Average Precision@5 = 0.261 vs. 0.133 for STRAP), confirming the encoder isn’t just memorizing its training distribution.
•
Downstream PPO policies trained with RoboTok-retrieved demos as a reward-shaping signal win on the VTDexManip benchmark. On the original formulation, RoboTok beats the best pretrained baseline on 5 of 6 tasks. On a harder version the authors introduce (full 3D wrist control, dense rewards removed), the gap widens: on Lever Sliding, seen objects go from 19.5% (HAND) to 79.3% (RoboTok); Faucet Screwing goes from 6.8% to 44.8%.
•
Although the encoder was trained only on the top-20 Dynamic Time Warping neighborhood, the embedding organizes globally, and semantic categories emerge in the t-SNE without any label supervision.
Reach for this when you’re training a dexterous-hand or humanoid policy and you have a few reference clips of the target task but not enough demos to learn from. Instead of teleoperating hundreds more, use one clip as a query, pull the top-K similar human videos from a web-scale index, and feed the retargeted hand trajectories into your reward as a demo-tracking bonus, exactly the setup they use with PPO on VTDexManip. The retrieval never touches robot actions, so it slots in above whatever policy learner you already have.
The project site is linked from the paper; code, model weights, and index availability aren’t specified in the text. The pipeline depends on three released third-party components (WiLoR, MoGe-2, HaWoR) plus SMPL-H for the torso estimator, and the training corpus is drawn from Action100M. If you want to reproduce the index you’d need to re-run that stack on your own filtered video pool.
When you’re searching video for behaviors, embed the geometry that actually causes the behavior, not the pixels that happen to depict it. For manipulation that means 3D hand trajectories in an actor-relative frame; for other domains the analog will differ, but the pattern is the same: canonicalize away the nuisance axes (viewpoint, appearance, speed) before you learn the similarity metric.
•
The pipeline assumes near-static cameras and visible hands; the authors filter aggressively for both, and explicitly leave moving-camera and egocentric video to future work. A lot of web video fails these filters.
•
The whole system inherits errors from WiLoR, MoGe-2, and HaWoR. If 3D hand estimation drifts on your target domain (unusual grips, heavy occlusion, low light), the embedding degrades and there’s no obvious recovery path.
•
Downstream gains are shown only in simulation on VTDexManip with PPO reward shaping. Whether retrieved human trajectories transfer as cleanly to real-hardware behavior cloning or diffusion policies isn’t tested here.