H3-World turns a frozen 33B video generator into an interactive world model by writing per-timestep character and camera actions as text prompts bound to matching video latents, training only 0.199% of parameters via LoRA on 8,000 clips.
You’re building an interactive game demo or a simulator for robot policy training. You want the user to press W-A-S-D and have a video model respond with the right character motion and camera pan, at the right moment. The dominant recipe is to bolt on a new action-conditioning module: encode the keyboard state into a vector and inject it via feature modulation like FiLM or additive bias into the video backbone. That means new parameters, action-video paired data at scale, and risk of clobbering what pretraining already knew. This paper’s bet is that a strong text-to-video model like MiniMax-H3 already half-understands motion instructions in natural language, so you should route control through the text pathway it already has, not build a parallel one.
The base model MiniMax-H3 is a bidirectional Diffusion Transformer (DiT) that denoises a full video clip at once, with text, image, and video tokens packed into one self-attention stream. Vanilla H3 can already follow a single coarse motion instruction for a whole clip, but not a schedule that changes mid-clip. H3-World’s fix has three moves.
First, actions become text. Each keyboard state (character key + camera key + a speed flag) is templated into a short sentence like “the man walks backward and strafes left, camera pans right slowly.” One such sentence is generated per video latent interval, so a 124-frame clip produces 37 short prompts.
Second, each prompt is bound to its matching video latent by position. The prompts are encoded independently, placed just before their paired video latent in the packed sequence, and given a temporal position offset by a fixed gap. This gives every action-latent pair the same relative position, a cue the pretrained model can pick up.
Third, Single-Egress Routing restricts attention so each action span can only write into its own matched video latent. Video-to-video attention stays fully bidirectional, so motion still propagates naturally. Then LoRA adapters (rank 32) on the attention projections learn the mapping from these constrained action tokens to visual dynamics.
for k in range(K): # K action intervals per clip
prompt_k = f"{char_template(u_k)}, {cam_template(c_k)}"
A_k = refiner(text_encoder(prompt_k))
pos(A_k) = pos(V_k) - delta # mirrored temporal position
X = [S, A_1..A_K, C_0, V_1..V_K, padding]
mask = single_egress(A_k -> V_k only) # A_k cannot see other V_j
X_out = h3_backbone_with_lora(X, mask)
The prevailing move when you want a video model to obey actions is to introduce a fresh action-conditioning pathway, learned embeddings, FiLM modulation, dedicated camera-geometry heads. This paper shows the opposite: the pretrained text pathway of a strong video generator is already most of the way to being an action interface; you just need to give it per-timestep prompts and enforce that each prompt only speaks to its own time slice. The cleanest evidence is not the headline gameplay generations but a switched-pan ablation isolating what LoRA adaptation actually adds on top of the frozen prior.
The load-bearing experiment is a scheduled camera pan that goes hard left for 15 latents then hard right for 22. They measure cumulative horizontal optical flow before and after the switch. Three conditions: frozen H3 with one global text instruction, the full per-latent interface with LoRA weights zeroed, and trained H3-World. Frozen H3 with a global prompt gives 0.0 flow before the switch and -17.3 after: it responds a bit but ignores the schedule. The per-latent interface without LoRA is essentially inert (-0.1 and 0.0). Trained H3-World produces +52.7 then -106.0, following both directions in their assigned intervals. Reversing the schedule reproduces the pattern. So the per-timestep text tokens are necessary but not sufficient. LoRA is what teaches the backbone to actually route them.
•
Against learned direct action-conditioning baselines (additive-bias, in the style of ReactiveGWM, and FiLM-style modulation after Adaptive Layer Normalization (AdaLN) modulation), the text interface produces visibly more coordinated character-plus-camera changes on held-out clips. The paper reports this qualitatively.
•
On a coarse constant-direction command, frozen H3 and H3-World produce nearly identical directional separation (301.8 vs 300.5). So the pretrained model already had the coarse motion prior. The adaptation is buying temporal precision, not motion competence.
•
Compositional generalization: training covers 83 of 135 valid character-camera pairs. On unseen pairs whose parts appeared separately, the model still follows both clauses.
•
Visual generalization: the same interface works on first-person and third-person scenes, indoor and outdoor, fantasy and sci-fi styles far from the ABot-World-Explorer-500h training distribution.
Reach for this pattern when you’re building an interactive video-model demo (game preview, driving sim, embodied agent rollout) on top of an existing strong text-to-video backbone, and you don’t want to burn a full fine-tune or design a fresh action head. Convert your discrete control state into a compact templated sentence per output chunk, pack one sentence per latent interval, mask attention so each sentence only reaches its own chunk, and adapt with LoRA. Roughly 8,000 clips got them working control here.
Artifacts: code at GitHub, weights on HuggingFace, project page linked in the paper. Training data is the ABot-World-Explorer-500h gameplay set. Note the base model is 33B parameters, so inference is not cheap, and the paper only demonstrates fixed-length short-horizon clips, no streaming, no persistent state, no policy loop.
•
Quantitative evaluation is thin. The controlled optical-flow test is on one switched-pan schedule; most other claims (compositional pairs, visual transfer, comparison to FiLM and additive-bias baselines) are shown as picked examples, not aggregated metrics. The paper explicitly flags this.
•
The approach assumes the base model already has a usable coarse language-motion prior. On a weaker text-to-video backbone that doesn’t respond to “camera pans right,” packing per-latent prompts probably won’t bootstrap control from nothing. The zero-LoRA condition already shows the prompts alone do nothing.
•
Fixed-length short-horizon generation only. No persistent world state, no real-time streaming, no planning or policy learning on top. Extending to long-horizon rollouts with memory is left open.