DriveZero trains a camera-only self-driving planner without any human driving examples by first learning behavior in a fast structured simulator with reinforcement learning, then distilling that teacher’s rollouts (queryable under many goals) into a vision student that reaches 95.3 PDMS on NAVSIMv1, above the human driver.
Most end-to-end driving stacks are trained by Behavioral Cloning on recorded human trips. That has a hard ceiling: the log shows one action per scene, rarely contains near-crashes or recoveries, and never shows what happens after the policy itself makes a mistake. Once the deployed car drifts off the demonstrated distribution, errors compound.
The obvious alternative, letting the car practice in closed-loop simulation with RL, runs into a different wall: rendering photorealistic camera frames for millions of RL steps is prohibitively expensive. Prior privileged-teacher work like ROACH (a CARLA-trained coach distilled into a camera policy) and GigaPixel (self-play distilled through a simplified renderer) showed the general recipe: train the RL policy on cheap privileged inputs, then hand its behavior off to a camera model. DriveZero, from Xiaomi’s L3 team, takes that split further by pretraining perception and action completely independently, each in its native regime, then bridging them with distillation.
The system has three separately trained pieces that only meet at distillation time.
1. DriveRL, the action teacher. A small (~5.7M-parameter) policy sees a structured scene (ego, up to 96 nearby agents’ recent history, a local vector map, traffic lights, two goal points) and outputs a distribution over two continuous controls: longitudinal jerk and steering-angle rate. It’s trained from scratch with PPO inside a mixed-agent simulator built from real nuPlan logs. Each background actor gets its own behavior provider: some replay their log, most follow IDM car-following, some occasionally emergency-brake, and optionally some are controlled by the same learned policy (self-play). The simulator runs ~196k worlds in parallel on 96 GPUs. Reward combines hard events (collision, off-road terminate the episode), a one-time goal-arrival bonus, and six soft driving-quality scores multiplied together, so the policy can’t trade safety for progress.
2. Value-guided test-time search. PPO also produces a critic. At inference DriveRL uses it like a mini-MCTS:
mode = policy.mode(obs)
candidates = [mode] + policy.sample(obs, n=N-1)
scores = []
for a in candidates:
traj = rollout(a, policy, horizon=L) # bg agents extrapolated
scores.append(discounted_reward(traj) + gamma**L * V(traj[-1]))
best = argmax(scores)
return candidates[best] if scores[best] > scores[0] + delta else mode
The modal action is only replaced if a sampled alternative beats it by a margin, making search a conservative local improvement.
3. DriveVFM, the perception backbone. Instead of training a shared encoder against labeled detection/segmentation/depth heads, DriveVFM distills four frozen vision foundation models into one ViT: DINOv3 for spatial structure, SigLIP2 for semantics, Segment Anything Model for object boundaries, and Depth-Anything-V2 for geometry. Each teacher supervises where it’s strongest (some on a summary token, some on patch tokens). Because supervision is just “match the frozen features,” no driving-specific labels are needed, and the training mix can freely combine web images with driving footage. Following RADIO’s agglomerative distillation, they use PHI-S to normalize each teacher’s feature statistics so no single teacher dominates the gradient.
4. DriveZero, the student. A camera-only planner: DriveVFM (LoRA-finetuned) encodes four surround cameras, register tokens compress the visual tokens, and a decoder emits 64 candidate trajectories plus per-candidate quality scores. It’s trained open-loop on logged frames, but the target for each frame is not the human trajectory. Instead the frozen DriveRL teacher is rolled out from that frame’s structured state, and its trajectory becomes the label via a winner-takes-all loss (only the closest of the 64 proposals gets pulled toward the target). The scoring head is trained against PDM Score components.
Goal augmentation is the twist. Because DriveRL is goal-conditioned, the authors re-query it on the same scene with different navigation commands, producing multiple valid futures per frame, something a human log physically cannot contain.
On nuPlan closed-loop (structured inputs, DriveRL only). Mean score across the three community splits in reactive and non-reactive modes: 93.01, rising to 93.57 with value-guided test-time search. This beats Log-Replay (playing back the actual human trajectory) on every single split, and beats prior RL-only methods CaRL and GigaFlow where they report numbers. Test-time search helps most on the harder non-reactive splits (+1.16 and +1.43 points) and scales roughly monotonically with the number of sampled candidates from 8 to 64.
On NAVSIMv1 navtest (camera-only student). DriveZero hits 94.8 PDMS, matching the human driver (94.8), without ever seeing a human trajectory as a training target. Scaling training data with simulation scenes from SimScale pushes DriveZero-Scale to 95.3 PDMS, ahead of every prior camera and camera+LiDAR method in their table. The gain over the human comes mostly from ego-progress (91.5+ vs the human’s 87.5): the model is willing to go when the human hesitates.
On NAVSIMv2 navhard and HUGSIM. DriveZero-Scale reaches 57.1 EPDMS on navhard and 46.6 HD-Score zero-shot on the true closed-loop HUGSIM benchmark (no HUGSIM-specific finetuning), 8.1 points above the previous best RL-teacher method GigaPixel.
Ablations that matter. In the supervision study, distilling DriveRL trajectories without goal augmentation is slightly worse than cloning human trajectories (93.61 vs 93.92 PDMS). Adding goal augmentation flips this to 94.41, the highest of the three. So the paper’s claim isn’t “RL trajectories beat human trajectories” in the naive sense: it’s that the RL teacher’s ability to answer counterfactual “what if the goal were different” queries is what makes teacher supervision surpass human supervision. That capability is structurally unavailable to log-based training. Separately, adding SAM and Depth Anything V2 on top of a DINOv3+SigLIP2 base gives cumulative +0.41 and +0.31 PDMS, supporting the multi-teacher design.
•
If you’re building an end-to-end driving stack on top of imitation logs and hitting a ceiling on rare or recovery behaviors: the concrete lesson is that you don’t need a photorealistic RL loop to escape it. A structured-state RL teacher plus offline distillation onto camera inputs is enough to match or beat human supervision on public benchmarks. The heavy lift is the parallel structured simulator, not visual rendering.
•
If you’re picking a vision backbone for driving perception: multi-teacher distillation from frozen foundation models is a label-free alternative to multi-head auxiliary training. The ablation shows each teacher contributes; if you can’t afford all four, DINOv3+SigLIP2 alone is already close. Worth testing on your own perception tasks before committing to labeling budgets.
•
If you’re already running a policy that outputs an action distribution and a value function: the value-guided test-time search is essentially free to bolt on: sample N first actions, roll each out a few steps with the same policy, keep the best only if it beats the mode by a margin. The paper shows monotone gains up to N=64 with no retraining. Useful as a cheap knob when you have inference budget to spare.
•
A caution before generalizing the “beat the human” headline: the human comparison is on NAVSIMv1’s PDMS aggregate, which is a pseudo closed-loop scoring rule, not real driving. And the score gap comes largely from ego-progress; DriveZero moves through scenes where the human hesitated. Whether that’s “better driving” or “more aggressive driving that the scoring rule happens to reward” is a judgment call the benchmark alone doesn’t settle.
The project page is linked from the paper (xiaomiautol3.github.io/DriveZero); the paper does not mention a code or weights release.
•
Real-world validation is thin. The paper reports a small-fleet urban demo with DriveRL controlling the vehicle, but no quantitative on-road results, no intervention rates, no comparison to a shipped baseline. All numeric claims are simulator or pseudo-closed-loop.
•
DriveRL uses privileged inputs. Its 95.8 PDMS is with ground-truth symbolic scene state. Real deployment needs an upstream perception stack to produce that state, which the paper handles by mixing auto-labels with noisier onboard perception during training but doesn’t quantify separately.
•
Goal augmentation is doing real work. Without it, teacher supervision underperforms cloning humans. The mechanism-level story only holds because the teacher is goal-conditioned; a non-conditioned RL teacher would not obviously beat imitation here.
•
HUGSIM zero-shot numbers are strong on Easy/Medium but drop sharply on Hard and Extreme tiers (HD-Score 24.7 and 27.6 for DriveZero-Scale). State of the art on that benchmark still leaves a large absolute headroom.
•
The nuPlan reactive evaluator itself drives background traffic with IDM, which the authors note limits how much self-play helps. Gains from more naturalistic interaction training won’t show up until benchmarks reward it.