Programmable World Model keeps the authoritative game state in a small rule engine and uses a video model only as a renderer, driven by projected 3D bounding boxes plus identity/semantic/motion overlays, so entities persist even off-screen and across long rollouts.
If you build an interactive video world today with something like Genie 3 or a similar generative video engine, the model both imagines the next frame and implicitly remembers what exists in the world. That second job is where it breaks. A character you killed reappears alive. An NPC that walked off-screen forgets its inventory. Prompting “the guard is dead now” produces a plausible frame but doesn’t bind that fact to future frames.
The scenario the paper actually evaluates: a combat scene where the engine records who is alive, and you want the rendered video to keep showing the right number of alive characters and to keep dead ones dead. Baseline video world models like LingBot-World-V2 and YUME have no external hook for “entity 3 is now dead”, so state has to be smuggled in through updated text prompts, and they drift.
The authors’ claim is that visual generation should not be responsible for remembering world facts at all. Split the two jobs.
The pipeline has three parts that talk through a narrow interface.
1.
A rule engine holds the world state. Each entity is a 3D 3D Oriented Bounding Box (OBB) plus attributes (identity, category, health, faction, velocity) plus rules (what actions are legal, what damage does, what triggers death). A VLM-based coding agent reads a reference image and a natural-language description, runs an off-the-shelf 3D detector to seed the entities, and writes an executable program defining the rules. Player actions run through this program deterministically. No learned transition model, so state updates are verifiable.
2.
A state compiler turns state into pixel-aligned control maps. For the target camera at frame t+1, it projects each entity’s OBB into the view and rasterizes three overlays stacked as channels: an identity map (a learned embedding from a fixed bank of K slots, so the renderer knows “this blob is the same instance as before” even after occlusion), a semantic map (a text-encoder embedding of the category label), and a direction map (the entity’s world-space velocity, rotated into camera coordinates and quantized to one of 7 directions like forward/left/static). The direction map matters because it lets the renderer distinguish “the car is moving right” from “the camera panned left”.
3.
A frozen video model renders, conditioned on those maps. They take a pretrained camera-controlled video backbone (LingBot-World-v1) and attach a trainable ControlNet adapter that consumes the stacked control maps. The backbone stays frozen; only the control branch trains. For long rollouts, generation is chunk-autoregressive: each new chunk sees a multi-scale temporal history of past latents plus a Geometry-aligned spatial memory borrowed from AlayaWorld that reprojects earlier RGB frames into the current view using estimated depth.
Training data is scraped from HUD-free gameplay footage (Cyberpunk 2077, Forza Horizon 6, GTA V). An automatic pipeline recovers what the control maps need: VIPE SLAM estimates camera pose and metric depth, Qwen3-VL proposes the category vocabulary, SAM 3 (Segment Anything 3) does instance segmentation and tracking, and WildDet3D fits per-frame 3D OBBs. Those get rasterized into the same identity/semantic/direction maps used at inference, so training and inference see the same interface.
Rough control flow per interaction step:
s = engine.init(reference_image, agent.write_rules(user_desc))
for action in player_actions:
s = engine.step(s, action) # deterministic rule execution
ctrl = compile_maps(s, camera_next) # project OBBs, stack id/sem/dir
frame = renderer(ctrl, camera_next,
temporal_history, spatial_memory)
temporal_history.append(frame)
spatial_memory.update(frame, depth, camera_next)
Evaluation is on CombatStateBench, the authors’ own 50-clip benchmark of combat scenarios with scripted death events. A VLM judge (Qwen3.6-27B) sees only rendered RGB frames, with no ground-truth boxes, and answers two global questions:
•
Count Accuracy: does the number of visibly-alive characters in a sampled frame match what the engine says? Their method hits 94%, versus 40.75% for LingBot-World-V2 and 32% for YUME.
•
State Accuracy: after a scripted death event, does at least one of three sampled post-event frames actually show a dead character? Their method hits 98%, versus 8% for LingBot-World-V2 and 58% for YUME.
On VBench perceptual metrics (imaging quality, subject/background consistency, temporal stability), their method is best or tied-best across all four, so the structured control doesn’t cost visual fidelity.
The qualitative results are the more telling evidence for the mechanism: in one clip the camera rotates far enough to reveal characters that were behind the initial viewpoint, and the renderer produces them consistent with the engine’s stored positions. They also show an 897-frame autoregressive sequence with NPCs progressively entering the scene. Both suggest the OBB scaffold plus spatial memory is actually carrying persistence information that pure video prediction loses.
One caveat on interpretation: both baselines were given state changes through prompt switching only, because they don’t expose an instance-level state API. So the comparison shows “structured spatial control beats text-prompt-swap”, not “structured spatial control beats an equally-resourced baseline with its own state interface”. The authors are up-front about this.
If you’re building an interactive video demo where object permanence matters (NPCs, inventory, kill/respawn, doors that stay open), the takeaway is architectural: don’t ask the video model to remember. Put the source-of-truth state somewhere you can print, diff, and unit-test, and hand the renderer per-frame spatial hints. You get verifiability and editability for free, because the state is a data structure and the rules are code.
The specific interface they chose, entity-level 3D OBBs plus identity/semantic/direction overlays, is worth copying even if you’re not using their renderer. It’s the sweet spot they argue for in Section 2: coarser than G-buffers or articulated meshes (so you don’t have to author limb trajectories), but stronger than 2D boxes or text (so you get view-consistent world coordinates). Worth testing whether a smaller ControlNet adapter on top of a video model you already have can consume the same three-channel stack.
If you want to reproduce the training-data path without buying game footage, the recipe of ViPE + Qwen3-VL + SAM3 + WildDet3D on unlabeled video is spelled out. The paper doesn’t mention releasing code, weights, or the benchmark.
Worth testing before committing: whether the approach holds up when entities interact in ways an OBB can’t express (grappling, deformable objects, fluids). The evidence is for combat with rigid characters and vehicles.
The benchmark is built by the same pipeline that generates training data, and both baselines are handicapped by lacking a state interface, so the headline gap partly measures “having any structured control channel” rather than the specific OBB design. The evaluation metrics are deliberately coarse (character counts and “is anyone visibly dead”) and don’t check instance-level correspondence, which is what the identity map is actually for. Generalization is shown qualitatively on unseen scenes but not measured. And the whole approach assumes an off-the-shelf 3D detector can seed the initial layout from one image, which is a real dependency for open-domain scenes.