TL;DR
ScaleNav learns navigation priors from Internet egocentric videos by pretraining a joint video-and-action Diffusion Transformer (DiT) to predict how a scene evolves as a camera walks toward a described target, then aligns those priors to robot actions using a small simulated dataset.
Why It Matters
You want a robot (say a quadruped) that follows instructions like “walk to the second door on your left” using only its camera. End-to-end policies that map pixels and text directly to motion need lots of diverse scenes to generalize, but real-robot data is expensive and simulation leaves a sim-to-real gap. YouTube has endless first-person walking footage that covers real rooms, lighting, and clutter, but it has no robot actions attached.
Prior work tried to recover action labels from these videos indirectly: estimate the camera trajectory from the pixels, or infer latent actions from frame-to-frame changes, then train a policy on those pseudo-labels. If the pose estimator is wrong, the policy learns from wrong supervision. ScaleNav sidesteps that: it uses the videos only to learn what the world looks like when you approach a target, and gets action supervision from a separate simulated dataset.
How It Works
The core model predicts two things at once from the same backbone: the next few camera frames, and the next few robot actions. Because video prediction and action prediction share parameters, visual priors learned from action-free YouTube footage can inform action generation later. This joint video-action architecture follows DreamZero. The backbone is a Diffusion Transformer; both future frames and action chunks are generated by denoising from noise, conditioned on the instruction, the current frame, and a short history.
Two context tricks matter. First, an initial-frame anchor: tokens from the robot’s starting view are kept in context throughout the episode, because phrases like “second door on your left” are grounded in that initial perspective. Second, a sliding window of the last couple of chunks (frames plus executed actions) provides short-term memory. Current-chunk tokens attend to these via chunk-wise causal attention.
Training has three stages:
# Stage 1: video pretraining on ~18K Internet egocentric clips
# objective: predict future frames given instruction + current frame
# no action tokens; initialized from Wan2.2-TI2V-5B
for clip, instruction in internet_videos:
loss = flow_matching(predict_video, clip, instruction)
# Stage 2: action alignment on ~2.6K simulated episodes
# unfreeze action head + action embedding + shared self-attn only
for obs, instr, action_chunk in sim_episodes:
loss = (1-lam)*video_loss + lam*action_loss # lam auto-balanced
replay_internet_video_batch() # keep video priors alive
# Stage 3: joint fine-tune, unfreeze remaining DiT + video head
At deployment the model denoises an action chunk (9 actions at 4 Hz), the robot executes it, then re-predicts from the new observation. Stopping is implicit: trajectories are padded with zero-motion actions so the model learns to output near-zero motion once it arrives.
What They Found
On a single-view adaptation of Short-Horizon OVON (Open-Vocabulary Object Navigation), ScaleNav reaches 86.5% success rate; on the authors’ new HabitatGS-LN benchmark (157 held-out indoor episodes with instructions referring to specific objects), it reaches 94.9%. Both beat the four evaluated baselines, including fine-tuned StreamVLN and Uni-NaVid, two recent vision-language navigation models.
The most informative comparison is the ablation that drops Stage 1 (no Internet video pretraining, same simulated data). Success drops by 5.3 and 4.5 percentage points on the two benchmarks respectively. That isolates the contribution of learning from the YouTube clips.
Two other ablations are worth noting:
•
Deriving pseudo-action labels from Internet videos via Depth Anything 3 pose estimation and mixing them into training actually hurt performance (73.9% SR, worse than skipping the videos entirely). The authors attribute this to noisy pose estimates. This supports their choice to use the videos only for visual prediction, not fake actions.
•
Removing the initial-frame anchor dropped SR from 94.9% to 89.8%, confirming that keeping the starting view in context helps resolve referring expressions.
On a real Unitree Go2 quadruped across three indoor spaces (12 tasks, 60 trials), ScaleNav hit 83.3% overall SR, highest among evaluated methods in every environment.
What’s Useful
If you’re building a language-conditioned navigation stack and considering whether to mine Internet video, the useful signal here is: pretraining a world model on action-free video and then aligning actions on a small simulated set outperformed deriving pseudo-action labels from the same videos. Worth testing on your own data if your pose estimator is noisy.
The anchor-plus-recent-window memory design is a cheap, portable idea for any policy that follows instructions grounded in an initial view. It’s not tied to this specific backbone.
Amortized per-action latency is 76 ms on an RTX 4090 with their inference optimizations (cached history KV, batched classifier-free guidance branches, skipping RGB decoding of predicted latents since actions don’t need the pixels). That’s in the range of real-time control for a walking robot but assumes a beefy GPU in the loop, not onboard compute. Code and videos are at the project page. Caveats
Evaluation is indoor and focused on reaching an object visible in the initial view. Outdoor settings, region-level goals (“go to the kitchen area”), and targets not initially visible are explicitly out of scope. The real-robot study is 60 trials across one robot in three spaces, which supports plausibility of transfer but not a strong generalization claim. The HabitatGS-LN benchmark is introduced by the same authors; the Short-Horizon OVON result uses their single-view adaptation and is “not directly comparable” to the original three-view protocol per the paper. Finally, the simulated training set is small (~2.6K episodes), so the ablation deltas between configurations are modest in absolute size, and some component ablations (removing Stage 2, removing video replay) move SR by only ~1 point.