Generating coherent video from text, images, or other video — covering temporal consistency, long-form synthesis, and the world-model framing of recent video models.
Video generation: keeping motion, control, and state coherent over timeVideo generation becomes difficult when a clip must do more than look plausible frame by frame. Camera motion needs a stable spatial reference, earlier frames must remain useful to later ones, and commands must affect the intended interval rather than leak across the sequence. Interactive systems add another requirement: generated changes should correspond to user actions while preserving appearance and geometry.This guide separates four practical problems: anchoring camera-controlled scenes, extending causal generation over long horizons, aligning language or scripts with time, and evaluating and serving interactive streams. Start with the group that matches your failure mode; the mechanisms address different bottlenecks and should not be treated as a single proven architecture.
Puffin-World: Scaling a Unified Multimodal Model with Native 3D World StatesStart here for a concrete account of why ray-only camera control can drift and how a propagated gravity anchor stabilizes rotation-heavy trajectories. It also separates camera-oriented “physics” from dynamics such as contacts and materials.The first-frame gravity estimate can be ambiguous; camera errors for some general generators were re-estimated by Puffin-World itself.
What are you trying to solve?
Anchor camera-controlled scenes to a stable worldRelative ray geometry can describe how views move yet provide no absolute notion of upright. Puffin-World adds a gravity-based orientation field to camera rays, propagates that anchor through known rotations, and couples generated appearance with depth. This targets horizon drift and unstable novel views, not general rigid-body physics or contact dynamics.
Puffin-World: Scaling a Unified Multimodal Model with Native 3D World StatesStart here for a concrete account of why ray-only camera control can drift and how a propagated gravity anchor stabilizes rotation-heavy trajectories. It also separates camera-oriented “physics” from dynamics such as contacts and materials.The first-frame gravity estimate can be ambiguous; camera errors for some general generators were re-estimated by Puffin-World itself.
Why can relative camera conditioning drift during long rotations?Relative rays encode motion between views but not which direction is physically up. Small orientation errors can therefore accumulate without an absolute reference. A gravity anchor supplies that reference, while relative rotations carry it forward along the requested trajectory.Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States
When is gravity-aware conditioning a useful design choice?Use it for walkthroughs, fly-arounds, novel-view synthesis, or world exploration where roll and pitch errors make horizons tilt or walls lean. It is less informative for yaw around the gravity axis and depends on reliable orientation perception from the reference image.Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States
Teach causal generators to survive their own historyLong rollouts fail when training and inference expose a model to different histories, or when old states are poorly encoded for future use. Self Gradient Forcing makes future losses train the historical key-value representations. SolarWM addresses a complementary engineering problem: normalizing heterogeneous video and camera data while preserving each backbone’s native representation.
Self Gradient Forcing: Native Long Video ExtrapolationRead for the context-gradient mechanism behind long autoregressive rollouts: future losses teach earlier key-value memory to encode more useful history without backpropagating through the full serial rollout.The abstract reports minute-scale extrapolation from 5-second windows but does not give implementation or compute details.
SolarWM: Open Data and Scalable Training for Long-Horizon Video World ModelsUseful when reproducibility across datasets and video backbones is the main obstacle. It identifies which data fields and interfaces to normalize and describes a staged route from bidirectional models to causal interaction.Results cover four 5B–33B models based on Wan2.2, LTX-2.5, and MiniMax-H3.
What is the historical context-gradient gap in autoregressive video?Earlier generated latents become cached keys and values for later frames, but a frozen rollout cache prevents future losses from improving how that history was written. Self Gradient Forcing uses a no-gradient rollout followed by parallel reconstruction so future prediction can supervise those context representations.Self Gradient Forcing: Native Long Video Extrapolation
How can short training windows support much longer causal rollouts?The two papers offer different pieces: improve the usefulness of causal memory, and use staged adaptation from bidirectional training through autoregressive initialization and distillation. SGF reports several-minute extrapolation from a 5-second window; SolarWM reports minutes-to-hours rollouts after training on 5-second sequences.Self Gradient Forcing: Native Long Video ExtrapolationSolarWM: Open Data and Scalable Training for Long-Horizon Video World Models
What must be standardized when training across incompatible datasets and backbones?SolarWM uses a frame-aligned contract for observations, metric camera geometry, captions, quality metadata, selection decisions, and provenance, while separating source processing from mixture construction. Shared interfaces then adapt several native backbone families without forcing one internal representation.SolarWM: Open Data and Scalable Training for Long-Horizon Video World Models
Route commands to the frames where they belongA model may understand an instruction yet apply it too early, too late, or beyond its intended interval. H3-World aligns structured character and camera actions with video latents and restricts their attention by time. Temporal Context Routing applies the same broad principle to joint audio-video generation by mapping script events onto the shared media timeline.
H3-World: Turning Language Understanding into World ControlRead for a lightweight route from pretrained language-conditioned generation to temporally bounded character and camera control, using structured instructions and temporal attention routing rather than a dedicated action module.Evidence comes from MiniMax-H3 adaptation with 8,000 gameplay samples and 0.199% trainable parameters.
The Missing Temporal Link: Temporal Context Routing for Script-Driven Audio-Video GenerationUseful for diagnosing why joint audio-video synchronization does not guarantee compliance with script timing. It maps shot and dialogue intervals onto the modalities’ shared temporal coordinates.Reported timing results use 200 test scripts in the described joint audio-video setting.
How can interactive video prevent one action from leaking into later actions?Represent character and camera commands as structured, time-bounded instructions, align them with the corresponding video latents, and restrict attention routing to that interval. H3-World reports doing this through lightweight adaptation of a 33B pretrained generator using 8,000 gameplay samples.H3-World: Turning Language Understanding into World Control
Why can synchronized audio and video still miss a script’s timing?Audio and video may share a temporal axis while shot and dialogue timing remains buried in an unaligned text embedding. Temporal Context Routing projects structured script timing onto that shared axis and sends each prompt segment only to its assigned positions in both modalities.The Missing Temporal Link: Temporal Context Routing for Script-Driven Audio-Video Generation
Judge local action outcomes, then schedule persistent sessionsInteractive video has both model-quality and systems bottlenecks. Long-video scoring can dilute brief evidence that an action was followed, so WorldReward compares action-aligned chunks before aggregating separate action and visual-quality preferences. TurboServe treats persistent generation as stateful scheduling across GPUs, including migration, batching, offloading, and autoscaling.
WorldReward: Reward Modeling for Camera-Conditioned World ModelsRead when whole-video metrics hide brief action failures. Its chunked pairwise method preserves local action evidence and produces separate preferences for execution and visual quality; it also reports use as an RL reward.Human-agreement and RL results are for the reported camera-conditioned models, benchmark, and preference pipeline.
TurboServe: Serving Streaming Video Generation Efficiently and EconomicallyUseful after generation becomes a multi-user service problem. It frames chunk delivery as joint session-placement and GPU-provisioning control rather than ordinary request batching.Latency and cost reductions were measured on production traces and clusters up to 64 NVIDIA B300 GPUs.
How should an interactive video be scored for both action adherence and coherence?Segment paired videos into chunks aligned with commands, judge local action execution and visual evidence separately, then aggregate those decisions at video level. This keeps short-lived action evidence visible while retaining distinct appearance and motion-quality judgments.WorldReward: Reward Modeling for Camera-Conditioned World Models
Why is streaming video serving different from offline generation?Sessions are long-lived, alternate between active and idle periods, retain state, and repeatedly need a new chunk before a latency deadline. Placement degrades as sessions evolve, while bursty demand changes the GPU budget needed to meet those deadlines.TurboServe: Serving Streaming Video Generation Efficiently and Economically
Which mechanisms help control latency and cost across multiple GPUs?TurboServe combines migration-aware placement with load-driven autoscaling. It batches active sessions, offloads suspended state to CPU, and migrates sessions between GPUs. Its reported gains come from production traces and clusters of up to 64 NVIDIA B300 GPUs.TurboServe: Serving Streaming Video Generation Efficiently and Economically