SimWAM trains a driving planner alongside a pretrained video generator, then throws the video branch away at inference. Video co-training lifts planning by +3.7 points PDMS (Predictive Driver Model Score) without paying video-synthesis latency at deployment.
You’re shipping a self-driving planner that takes camera frames and outputs a trajectory. A recent line of work (World-Action Models, e.g. DriveWAM and DriveLaW) improves such planners by pretraining on video generation: the model imagines the next few seconds of the road, then plans against that imagined future. The trouble is that in these systems the video generation happens at inference on every planning tick. Synthesizing future frames on the critical path is expensive, which is a hard constraint in a real vehicle. SimWAM keeps the benefit of learning from video and drops the cost.
The setup is two networks that only talk through shared attention: a heavy pretrained video expert (initialized from Wan2.2-5B, a general video generator) and a lightweight action expert that outputs waypoints. Both are trained with Flow matching against their own targets. The video expert learns to denoise the next 8 future frames from the current frame. The action expert learns to denoise the next 8 waypoints from the current observation, the ego vehicle state, and a navigation command.
The key structural trick is an isolated attention mask. Both the future-frame tokens and the action tokens are allowed to attend to the current-observation tokens, but they cannot see each other. So the action expert never reads any predicted future frame. What it does absorb, through gradient flow during joint training, is a shaped observation representation: the current-frame tokens have been sculpted by the pressure to also predict plausible futures. After training you delete the video expert entirely. Deployment runs only the small action expert.
A second stage applies reinforcement learning on top of the imitation-trained action expert. The deterministic flow-matching sampler is replaced with a stochastic equivalent (from Flow-GRPO) that preserves the same distribution but permits sampling multiple candidate trajectories. Each candidate is scored by NAVSIM’s composite driving reward, and Group Relative Policy Optimization (GRPO) updates only LoRA adapters on the action expert, focusing on hard scenes where imitation already scores below 90 PDMS.
# training
for obs, state, nav, future_frames, gt_traj in loader:
z_obs = encode(obs)
# attention mask: future_frames <-> action_tokens BLOCKED
# both can see z_obs
loss_vid = flow_match(video_expert, future_frames, cond=z_obs, nav)
loss_act = flow_match(action_expert, gt_traj, cond=[z_obs, state, nav])
(loss_act + lam * loss_vid).backward()
# inference: video_expert is gone
traj = ode_solve(action_expert, noise, cond=[encode(obs), state, nav])
The assumption in recent driving world-action models is that anticipated futures help planning most when the planner literally conditions on generated future frames at inference time. This paper argues the opposite. The training pressure to predict futures is what installs a traffic-dynamics prior in the shared observation encoder; once that prior is baked in, generating the frames at inference is dead weight. The evidence that isolates this claim is the attention-mask ablation, where blocking the action branch from ever seeing future-frame tokens matches or beats letting it see them.
•
The load-bearing result is the attention-mask comparison. Letting the action expert attend to future-frame tokens (bidirectional or one-way) does no better than the isolated mask that forbids it. The isolated mask actually reaches the top PDMS of 90.3 among the three, with the best no-collision and time-to-collision numbers. So the video branch’s contribution is entirely in shaping the shared observation, not in providing future context at inference.
•
Component ablation: action-only baseline is 86.6 PDMS; adding video co-training brings it to 90.3; adding RL brings it to 91.5. Video co-training is the larger single lift.
•
Headline: 91.5 PDMS on NAVSIM navtest with a single front camera, reported as better than the strongest VLM-based planner SGDrive by 0.4 and better than the DriveWAM and DriveLaW imagine-then-act planners by 1.4 and 2.4 respectively, at substantially lower inference latency (Fig. 1 in the paper; exact latency numbers not reproduced in text).
•
Video backbone is swappable. Wan2.1-1.3B, Wan2.2-5B, and a driving-pretrained Cosmos-Predict2.5 all land within 0.2 PDMS of each other; a lighter LTX-Video drops to 88.7, so backbone quality matters but the design is not tied to one choice.
•
Zero-shot transfer to nuScenes open-loop planning: lowest average collision rate at 0.04% and competitive L2 error of 0.96 m without any nuScenes training.
•
RL works better when restricted to hard scenes (imitation PDMS < 90) than when applied to all of navtrain, because easy scenes give weak reward signal.
Reach for this shape when you have a small deployment-time policy network and a much larger generative model of the same domain that is too expensive to run online. Instead of distilling the big model into the small one, or conditioning the small one on the big one’s outputs at inference, co-train them with a hard attention wall between the target tokens: the small network attends only to shared context, never to the big model’s outputs. At deployment you delete the big model. The isolated-mask trick is the transferable idea, not the driving specifics.
Code and weights are released at GitHub. The training pipeline depends on a pretrained video generator (they use Wan2.2-5B); the NAVSIM and nuScenes datasets are the evaluation harness. No new dataset is released.
If a big auxiliary model helps a small model learn, wall it off during training and delete it at inference. Don’t let the small model learn to lean on outputs it won’t have in production.
•
All main results are on NAVSIM navtest, which is open-loop, non-reactive, and front-camera-only. Closed-loop behavior with reactive agents is not evaluated, and safety-critical driving claims should not be read into a 0.4-point PDMS gap.
•
The action expert still inherits a traffic prior from a general video generator trained on internet video. If your domain has no such pretrained generative model available, the whole premise (“we get the prior for free at training time”) does not apply.
•
The isolated-mask ablation shows attending to future frames doesn’t help in this architecture with this joint objective. It does not prove future-frame conditioning is useless in general, only that once the shared-observation pathway exists, the extra channel adds nothing measurable here.