Get Started
Topic · 36 recaps
Video Generation
Generating coherent video from text, images, or other video — covering temporal consistency, long-form synthesis, and the world-model framing of recent video models.
Play all
...
Posts
Questions
Home
Topics
Search
Library
Questions researchers are working on
Follow a question through Rcap’s explanations and the latest papers addressing it.
Search
How can audio-video diffusion models preserve intended conditioning when biased cross-modal attention reroutes semantics?
In audio-video diffusion generation, cross-attention among text, audio, and video can route semantics bidirectionally rather than respecting intended conditioning. Learned biases may cause one modality to override prompts, producing visually canonical but semantically incorrect outputs.
How can camera-controlled video generation preserve spatial consistency over long horizons despite noisy 3D memory?
Long camera trajectories require generated frames to remain aligned with the scene over time. Multi-view 3D reconstruction can place the same surfaces at inconsistent locations, making history-based geometric conditioning noisy and degrading spatial consistency.
How can causal streaming video editing remain real-time while preserving backgrounds and unedited regions over long sequences?
Each frame must be edited without access to future frames, while backgrounds and untouched regions remain stable over time. Meeting that consistency requirement without sacrificing interactive responsiveness is difficult.
How can diffusion image and video generators be preference-aligned without inefficient training exploration or inference-time search?
Policy-gradient preference optimization can explore inefficiently, become trapped in local optima, and exploit reward signals in ways that reduce semantic faithfulness or visual realism. Alignment must also avoid requiring extra search during generation.
How can diffusion-transformer generators follow spatial controls without large task-specific adapters?
A generator may need to satisfy spatial or motion constraints while retaining general-purpose image or video synthesis. Separate adapters for each control can increase storage and adaptation costs and reduce reuse across tasks.
How can few-step generative video rendering preserve camera control and geometric consistency?
Reducing sampling steps can make camera control depend on discretization and amplify errors during self-rollout. The resulting renders must remain geometrically faithful while being generated with much lower sampling cost.
How can joint audio-video generators follow script-specified timing for shot transitions and dialogue?
Audio and video can remain synchronized while both occur at the wrong times relative to the script. This disrupts narrative structure when shot changes or dialogue are tied to specified moments.
How can language-controlled video generators make character and camera actions temporally precise in interactive worlds?
Natural-language control may specify behavior and camera movement, but interactive worlds require each action to occur at the intended time without leaking into other actions. The challenge is achieving this precision while retaining the generator’s visual quality and ability to handle unseen scenarios.
How can legacy SDR video be converted into convincing HDR while recovering detail in shadows and highlights?
A single nonlinear SDR video lacks the exposure range needed to represent both dark and bright regions faithfully. Conversion must infer missing scene information while preserving coherent video appearance and detail across the tonal range.
How can long-horizon interactive video world-model training remain reproducible across heterogeneous datasets and incompatible backbones?
Interactive video world models combine datasets with different temporal scales, camera geometries, visual quality, motion, and captioning. Backbone-specific representations and processing pipelines can make supervision inconsistent and results difficult to reproduce or compare.
How can mask-free video virtual try-on maintain garment consistency under motion, occlusion, and changing viewpoints?
Mask-based localization can fail during large body motions or severe clothing occlusions. Sparse keyframes and limited multi-view data further make it difficult to preserve garment details consistently across frames and viewpoints, while video-level pseudo-data construction is expensive.
How can multi-GPU serving meet chunk-latency targets for stateful, bursty video-generation sessions?
Streaming video generation keeps sessions active while producing output progressively, so the system must preserve state and repeatedly serve each session. Uneven session durations and bursts in demand make it difficult to maintain timely chunks across GPUs without wasting capacity.
How can multimodal reasoning guide diffusion models for controllable video generation and editing?
Multimodal models can interpret complex visual and textual intent, while diffusion models produce detailed video pixels. Coordinating these capabilities without losing semantic control, visual fidelity, or training efficiency is difficult.
How can multimodal world models maintain physically coherent 3D scenes during closed-loop interaction?
Interactive world models must keep geometry, appearance, physical behavior, and viewpoint changes mutually consistent over time. Modeling these elements separately can cause future views or reconstructions to drift.
How can omni-modal models generate speech and temporally coordinated 3D facial animation?
Semantic reasoning in language models produces relatively discrete representations, while facial animation requires dense, temporally precise motion coordinated with speech. Bridging these different granularities is difficult when the model must generate both modalities together.
How can on-policy diffusion RL reuse rollout computations without trading redundant recomputation for excessive memory use?
Diffusion RL methods often recompute selected rollout timesteps with gradient tracking during the update. Reusing rollout activations can remove that computation, but retaining them or tracking gradients during rollout can substantially increase memory use.
How can scalable synthetic video datasets preserve temporal alignment between actions and resulting scene transitions?
Action-conditioned video models need each control signal to correspond to the visual transition it causes, but ordinary video rarely records that relationship. Synthetic production must therefore coordinate simulated actions, scene evolution, and rendered observations while scaling across many environments and assets.
How can sparse-view 3D scene representations avoid cross-view artifacts without costly per-scene reconstruction?
With few input views, NeRF and 3D Gaussian Splatting can produce geometry and appearance that disagree across viewpoints. Existing artifact correction may require paired corrupted and clean renders plus costly reconstruction for each scene, limiting scalable training.
How can sparse-view novel-view synthesis maintain geometric and temporal consistency along a camera trajectory?
A few observed views leave unseen surfaces and regions underconstrained, so generated content can drift from the scene’s geometry. Errors in intermediate scene representations can propagate into the video, making consistency across camera motion difficult.
How can synthetic driving videos preserve physically plausible vehicle trajectories after collisions when real crash footage is scarce?
Real collision footage is rare and difficult to collect for autonomous-driving research. A video can appear realistic while still depicting physically implausible post-collision motion.
Previous
1 / 2
Next