HappyWorld-Bench tests whether generative world models stay reliable when you actually poke them: it runs World Model systems through video rollouts, 3D scene exports, and robot-egocentric predictions under a shared W1–W6 capability ladder, and finds that the best spatial system still hits only 70.14% plate-placement success on generated geometry.
A World Model is supposed to let an agent imagine consequences before acting: rotate the camera, push the block, edit gravity, and get back a plausible next frame or scene. That only works if the imagined world stays consistent. A video that looks gorgeous for two seconds but forgets where the couch was, or a generated 3D room whose floor a physics engine falls through, is useless as a substrate for planning.
The problem the paper attacks is that evaluation of these models is scattered. Video-generation benchmarks like VBench and WorldScore measure visual quality and camera controllability. Spatial benchmarks like SceneEval check object counts and support relations on generated 3D. Embodied benchmarks score action-conditioned videos separately again. Each community uses its own scenes, prompts, and rubrics, so you cannot tell whether a system that renders pretty video would also give you a walkable room or predict what happens when the robot gripper closes.
The authors want one framework where all three model families are graded on the same underlying question: does the generated world hold up under interaction, memory, and intervention?
The organizing move is a capability hierarchy W1–W6, from W1 (Perceptual: coherent scene from an image) up through W2 (Interactive: action-conditioned transitions), W3 (Persistent: state survives revisits), W4 (Programmable: language-edited rules propagate correctly), W5 (Scalable: multi-agent shared worlds), to W6 (Universal). Each track evaluates the subset of levels its model form can express.
Three tracks share this vocabulary. The video track runs 1,138 prompts where a model must generate rollouts under keyboard/mouse-style controls or natural-language edits. The spatial track takes 300 scenes and asks systems to export renderable geometry (Gaussian splats or meshes), then probes it: can a Habitat-style NavMesh find a connected walkable region? Can a physics simulator drop a plate on the annotated table without it falling through? Can you edit “add a lamp” without wrecking the rest of the room? The embodied track gives 254 cases where a video model receives one egocentric frame plus an action prompt, and its rollout is judged on whether the prescribed contact, state change, and terminal condition actually occur.
Scoring combines two things. Automated metrics use frozen tools: CLIP and DINOv2 for appearance similarity, SAM 3 (Segment Anything 3) for masking, DA3 for depth and pose, TAPIP3D for 3D point tracking, HPSv3 for learned human preference, plus a VLM judge (Qwen3-VL, Gemini) that answers atomic yes/no assertions about scene facts, causal order, and action execution. In parallel, HappyWorld-Arena collects human A/B votes across models and fits Elo rating ratings per track. For paired W4 tests, the same initial frame is fed under two different edited conditions (say, normal vs low gravity) and the metric checks whether the two branches actually diverge in the prescribed way.
The embodied assertion pipeline roughly looks like:
for case in cases: # 254 embodied tasks
video = model(case.ref_image, case.action_prompt)
frames = sample(video, fps=4)
scores = {}
for dim in ["perception","consistency","causality","controllability"]:
assertions = case.assertions[dim] # frozen, task-specific
judgments = vlm.judge(frames, assertions) # binary + weights
scores[dim] = weighted_mean(judgments)
overall = 0.2*perc + 0.2*cons + 0.3*caus + 0.3*ctrl
Across 14 video, 9 spatial, and 8 embodied systems, visual quality and reliability come apart hard.
•
Video track. HappyOyster leads Arena Elo at 1263, Genie 3 next at 1206. But per-level scores reshuffle: Genie 3 wins W1 (perception) at 82.8, HappyOyster wins W2 and W3. On W4 (rule editing), only HappyOyster and Lingbot-World-v2 even support the interface, scoring 66.7 and 59.3. The authors highlight that strong perception scores routinely coexist with weak state consistency: models can render a beautiful frame while quietly changing what objects exist.
•
Spatial track. After adjusting Elo for unsupported capabilities, Marble leads at 1308, then GPT-6-Astra at 1252. But FlashWorld and the Lyra variants produce renderable scenes that score 0.00–1.36 on navigable ratio, meaning their exported geometry has almost no connected walkable region. Best plate-placement rate is 70.14% (HYWorld-2.0); best edit-success rate is 73.33% (also HYWorld-2.0), but GPT-6-Astra wins the joint edit score because it preserves untouched content better. Editing success and side-effect containment are separately measured and separately failed.
•
Embodied track. MiniMax-H3 tops all three levels and the Elo. But controllability decays as tasks lengthen: models that execute a single grasp cleanly at W2 often fail to carry state across multi-stage sequences at W3 (grasp the shirt, move it, hang it: the middle transfer drops). At W4, edited physical rules (change gravity, change material) frequently leave no observable trace, and edited action conditions are honored in direction but not in manner (a “push” becomes a grasp-and-drag).
The consistent observation, which the authors frame as their headline finding, is that perceptual quality is largely saturated while action grounding, state persistence, and condition-dependent physics are not.
•
If you’re picking a video world model for an interactive demo, the Arena Elo and W1 scores will mislead you about longer rollouts. Look at the W3 consistency columns (BC, GC, SC, SuC) for the specific system, because two models with matching W1 can differ sharply once revisits are involved.
•
If you’re using a generated 3D scene as a simulation asset, do not assume renderability implies usability. The paper shows a 0–70% spread in navigable-area and support-surface metrics among systems that all look fine in a screenshot. Running a NavMesh build and a few plate-drop trials on any candidate export is worth doing before you wire it into a policy loop.
•
If you’re evaluating an embodied video model as a planner or data engine, the W4 paired-branch protocol is the transferable idea: generate rollouts from the same initial frame under two edited conditions and check that the outputs diverge where the conditions diverge. Non-response to physical-rule edits is common enough that you should test for it explicitly rather than assume the prompt was honored.
•
For research framing, the W1–W6 hierarchy gives you a shared vocabulary when talking across video, 3D, and embodied subfields; the paper argues this is the main deliverable alongside the arena.
The paper does not link a public code or data repository in the supplied text, so availability of the harness itself is unclear from what’s provided.
•
W2 and W3 in the video track use different case pools, so the authors explicitly warn that a lower W3 score is not a pure “long-horizon degradation” signal.
•
Many automated metrics rely on frozen VLM judges and learned preference models (HPSv3, Qwen3-VL, Gemini checklists). These inherit those models’ biases, and the paper’s own “F1” composites use inverse dispersion as a stand-in for precision, which is not true-positive precision.
•
W4 video results cover only two systems that expose the intervention interface, so “programmable world” comparisons are thin.
•
W5 and W6 are defined but largely unscored: the spatial track scores W5 for four systems, and the embodied track skips W5 and W6 entirely. Claims about “scalable” or “universal” world modeling are not evaluated here.
•
Arena Elo is adjusted by adding synthetic wins/losses for unsupported capabilities in the spatial track. This mixes human preference with capability coverage, so the Elo number is not a pure preference statistic.
•
Several evaluated systems are accessed through hosted interfaces (Genie 3 via the Project Genie web app) where seeds, sampling steps, and checkpoints are not exposed; the authors note results characterize the deployed system rather than an isolated model.