PhysisForcing makes a video diffusion model generate physically plausible robot manipulation by adding two training losses that fire only on interaction regions: one matches predicted point tracks to a tracker’s trajectories, the other matches token-similarity matrices to a frozen video encoder. Closed-loop robot success climbs from 16% to 24%.
You’re shipping a robot policy that uses a video model as a simulator: feed it the current camera frame plus an action, get a predicted future clip, then decode actions from that prediction. The problem is that today’s Diffusion Transformer (DiT) video generators, even ones fine-tuned on robot footage, hallucinate. The gripper teleports, the cup deforms mid-grasp, the pushed block sits still while the arm moves through it. Visually plausible, physically nonsense. That breaks the simulator’s value because the predicted frames aren’t a valid consequence of the action.
Prior fixes either bolt on geometric signals like depth and tracking after the fact, or do post-hoc preference alignment such as Direct Preference Optimization against a physics discriminator. The authors argue these are either too local (geometry only constrains per-pixel motion) or too sparse (preferences don’t say where the violation is).
The insight is that physical errors cluster in specific places: where the gripper touches things, where objects move. So supervise there, not everywhere, and supervise at two levels.
First, find the physics-informative region. Run CoTracker3 on the ground-truth video to get dense point trajectories. Multiply each point’s motion magnitude by an inverse-depth weight from Depth-Anything-V2 so foreground beats background jitter. Threshold to get a binary mask of “where interesting physics happens.”
Second, two losses pull on a single middle layer of the Diffusion Transformer (DiT) during fine-tuning:
•
Pixel-level trajectory loss. Take the first-frame DiT feature as a query, treat later-frame features as keys, do a soft-argmax over the similarity map to predict where each tracked point went, then MSE against CoTracker3’s actual trajectory. Only points inside the mask count.
•
Semantic-level relational loss. Pass the same clip through frozen V-JEPA 2 to get teacher tokens. Project the DiT feature to the same shape. For tokens inside the mask, build a K×K cosine-similarity matrix on each side and L1-match them. This transfers “how regions co-move” without forcing absolute feature matching.
mask = build_phys_mask(video, tracker, depth) # foreground + high motion
h = dit.intermediate_block(noisy_latents) # one middle layer
pred_traj = soft_argmax_match(h[0], h[1:]) # query frame 0 vs frames 1..T
L_pix = masked_mse(pred_traj, cotracker3(video))
teacher = vjepa2(video)
R_hat = cosine_gram(mlp(h)[mask])
R = cosine_gram(teacher[mask])
L_sem = l1(R_hat, R)
loss = L_flow_matching + lam_pix * L_pix + lam_sem * L_sem
All auxiliary models are training-only, so inference cost is unchanged.
The usual move for physics-aware video generation is to add a global signal (depth everywhere, a preference reward over the whole clip) and hope the model figures out where to apply it. This paper inverts that. Localize supervision to the pixels where contact actually happens, and split it into per-point motion plus pairwise region relations, because uniform supervision dilutes the signal background pixels don’t carry. The cleanest evidence is the region-focus ablation: applying the same two losses everywhere gains +1.2 points, restricting them to the mask gains +2.7, with the largest lift on task-level scores.
•
Region focus is what makes it work. On R-Bench with Wan2.2-TI2V-5B, vanilla fine-tune scores 44.8. Adding both losses without the mask gets 46.0. Adding them with the mask gets 47.5, and the gain concentrates on task-oriented dimensions (35.4 → 38.9). That delta is the load-bearing finding because it isolates the region-focus design from the losses themselves.
•
The two losses are complementary, not redundant. Pixel loss alone is the bigger single-loss gain (suppresses local trajectory discontinuity); semantic loss alone fixes global contact and coupling errors; stacking is best on both backbones tested.
•
Headline benchmarks. On R-Bench, PhysisForcing lifts Wan2.2-I2V-A14B by 22.3% over base and 7.1% over vanilla fine-tune, and Cosmos3-Nano by 9.2% and 3.7%. PF-Cosmos hits the top overall score (63.8), beating Wan2.6 (60.7) and Cosmos3-Super (58.1).
•
It transfers to control. Under the WorldArena action-planner protocol, closed-loop success goes from 16.0% to 24.0%, beating WoW at 20.5%. Used as the video backbone inside Fast-WAM on RoboTwin 2.0, average policy success goes 68.2% → 72.8%, with the biggest jumps on contact-rich tasks (place_empty_cup +21.5 pp).
•
Layer choice matters: middle DiT block (layer 15 of the smaller model) wins; shallow blocks lack semantic structure, deep blocks are too specialized for noise prediction.
Reach for this if you’re training a video world model for a robot stack and your downstream actor keeps tripping over contact errors that the visual loss never penalizes. The recipe assumes you have ground-truth video and can afford running a point tracker, monocular depth, and a frozen V-JEPA 2 encoder on each training clip. Inference is unchanged, so deployment cost is the same as your base Diffusion Transformer (DiT). The interesting design choice to copy even outside robotics: when your loss is uniform but your errors aren’t, build a cheap relevance mask from off-the-shelf perception and gate the auxiliary loss with it.
The project page is at dagroup-pku.github.io/PhysisForcing with code promised. Training data is a 500K-clip filtered subset of RoVid-X; the paper doesn’t say whether that subset is released. No new benchmark is introduced; evaluation is on existing R-Bench, PAI-Bench, and EZS-Bench.
•
Ceiling is the backbone, not the loss. The authors say so directly: PhysisForcing is fine-tuning, and the base Wan2.2 and Cosmos3 models still have limited world knowledge and long-horizon reasoning. If your application needs minute-long coherent rollouts, this doesn’t fix that.
•
Three frozen auxiliaries during every training step. CoTracker3, Depth-Anything-V2, and V-JEPA 2 ViT-L all run on each ground-truth clip. The paper doesn’t quantify training overhead, but the memory and throughput hit is real, and the quality of your physics signal is bounded by the tracker’s failure modes (occlusion, thin objects, fast motion).
•
Domain-score gains, quality-score flat. On PAI-Bench and EZS-Bench, the lift is concentrated on the physics-plausibility axis; visual quality stays roughly at vanilla-fine-tune level. If your bottleneck is rendering fidelity rather than physical consistency, this isn’t the right lever.