PhiZero is a video world model that predicts the future in two stages: first it emits a short sequence of learned discrete tokens describing how the scene changes, then a diffusion decoder renders those tokens into pixels, letting it compress a 4-second future into 256 discrete symbols instead of ~44,800 continuous visual tokens.
Suppose you’re building a robot-simulation product or a driving-scene generator, and you want the model to actually respect gravity, collisions, and cause-and-effect, not just look plausible. Today’s Video world model approach is to train a huge pixel-space video generator (Sora, Veo, Wan) and hope physics emerges from scale. It often doesn’t: a tennis ball hits a rubber duck and the duck sits there. PhiZero’s pitch is that dynamics should live in their own representation, separate from appearance, so the model can “reason” about what happens before it draws it.
The design splits future prediction into two learned pieces. First, a Physical Language Tokenizer watches a video and, for every pair of adjacent latent frames, uses a shared Q-Former to emit a small bundle of tokens describing the transition between them. Those tokens get discretized with Finite Scalar Quantization (FSQ) into a fixed vocabulary of about 25K symbols. A 33-frame clip becomes 8 transitions × 32 tokens = 256 symbols total, and that’s the “physical language.”
Second, a diffusion decoder (a fine-tuned Wan2.2-5B video model with LoRA rank 32) is asked to reconstruct the video given only the first frame plus those 256 symbols. Because the first frame carries all the static appearance, the bottleneck is forced to encode change, not looks. A “pure-noise warm-up” stage prevents the decoder from cheating by leaking future pixels through its own denoising prior.
Third, a Physical Language Reasoner, initialized from Qwen3-VL-4B, has its vocabulary extended with one new token per FSQ code. It’s trained to autoregressively predict the physical-language sequence from a first frame plus a text prompt. At inference:
def predict_future(first_frame, action_text):
# Stage 1: reason in transition space
z = reasoner.generate(first_frame, action_text) # 256 discrete symbols
# Stage 2: render pixels from the plan
video = diffusion_decoder(first_frame, z)
return video
Training used roughly 10K hours of filtered web video for tokenizer pretraining, 5M curated 4-second clips for tokenizer supervised fine-tuning and reasoner pretraining, and 1M motion-rich clips for the final reasoner stage.
The prevailing bet in physical world models is that if you scale pixel-space video generation hard enough, physics will fall out as a side effect. This paper argues the opposite. Dynamics deserve their own compressed, discrete representation, separate from appearance, so a language-model-style reasoner can plan the transition before any pixels are drawn. The evidence that isolates this claim is not the headline generation scores. It’s the transfer demos: encode a human pouring liquid, swap the first frame for a different bottle or a robot hand, decode the same token sequence, and the motion carries over.
The load-bearing result is the compression-versus-quality tradeoff of the tokenizer. Reconstructing 4-second videos at 512×896, PhiZero uses 256 discrete symbols versus 44,800 continuous tokens for the Wan2.2 VAE baseline, and still leads among highly-compressed tokenizers on reconstruction quality. That’s what makes the reason-then-render story credible: the intermediate representation is small enough for an autoregressive VLM to actually predict, but rich enough to rebuild the video.
•
On three generation benchmarks (Physics-IQ Verified, PhyGround, WorldModelBench), PhiZero reports the best headline score in each: top IQ-Score, top Physics Score and Overall on PhyGround, top Physics Adherence and Total on WorldModelBench. Absolute numbers are in the paper’s tables but not enumerated in the running text.
•
On three understanding benchmarks (IntPhys 2, LikePhys, YoCausal), the same reasoner scores real-vs-fake video pairs by comparing their physical-language log-likelihoods, and is reported as competitive across all three.
•
Ablations: replacing the diffusion decoder with a deterministic one causes the largest drop; removing the transition-level Q-Former (using a global one) hurts; skipping pure-noise warm-up hurts; skipping simulator data in the reasoner hurts; a prompt-enhanced Wan2.2-5B baseline stays well below PhiZero, which the authors read as evidence that natural language is too coarse to describe fine-grained transitions.
•
Qualitatively, PhiZero produces collision displacement, gravity-driven deformation, and chain reactions where the Wan2.2-5B baseline shows visually plausible contact but no physical consequence.
•
The tokens cluster meaningfully: UMAP projections of transition features on nuScenes driving clips and AGI-Bot manipulation clips separate by kinematic pattern (left turn, right turn, gripper closing) rather than by visual content.
Reach for this design when you’re building a controllable simulator, say a driving-scene generator conditioned on ego-trajectories, or a robot data-augmentation pipeline where you have simulation demos and want realistic-looking versions. The recipe: train a small discrete tokenizer that captures how things move given a start frame, put a VLM on top of it to translate action intents into transition tokens, and reuse a strong pretrained video diffusion model as the renderer. Zero-shot motion transfer falls out naturally: encode a source clip, edit the first frame with an image editor (they use GPT-Image 2.0), decode with the unchanged tokens, and you get sim-to-real or human-to-humanoid retargeting without paired data.
On artifacts: the paper points to a project page but does not, in the text provided, promise a code or weights release. The training corpus is a mix of public datasets (OpenVid-1M, Kinetics-710, SSV2, Moments in Time, HOIGen-1M, nuScenes, AGI-Bot RealRobot, LIBERO, CLEVRER, Physion, and others) plus in-house data. Training used 128 A100 GPUs for both the tokenizer and reasoner stages.
Give dynamics their own vocabulary before you ask a model to plan them. Pixel space is a bad substrate for reasoning about physics, and natural language is too coarse. A compact discrete transition code, learned self-supervised from raw video, is small enough for an autoregressive model to predict and rich enough for a diffusion decoder to render.
•
The “physical language” is empirical, not symbolic. The 25K FSQ codes are not interpretable physical variables, so you can’t audit why a predicted transition is what it is; you can only sample from it and look.
•
Coverage is bounded by what’s visible in training video. Tactile forces, occluded contacts, and anything sub-visual are effectively outside the representation.
•
The strong headline numbers rely on a pretrained video diffusion prior (Wan2.2-5B) doing the heavy lifting on appearance. How much of the benchmark win comes from the transition tokens versus that prior is not fully separated in the ablations reported here, and the paper’s clips are capped at 4 seconds, so long-horizon consistency is unproven.