Diffusion models: preserve, accelerate, and repurpose the denoising processDiffusion and flow-matching models learn a transformation from noise toward structured data. Their iterative refinement supports high-fidelity generation and editing, but it also creates practical tensions: fewer denoising steps can reduce latency while narrowing diversity, causal video processing can drift over time, and edits can overwrite content that should remain unchanged.Use this guide by starting with your failure mode. For interactive editing, focus on causal processing, selective recomputation, and end-to-end hardware measurements. For model compression, examine which teacher distribution and capabilities survive distillation. For depth, robotics, or 4D scenes, treat a generative prior as a reusable representation—but check whether the adaptation preserves the geometry or physics your downstream task needs.
LiveEdit: Towards Real-Time Diffusion-Based Streaming Video EditingStart here for a mechanism-level account of causal streaming editing: three-stage distillation, chunk-wise attention, and selective self-attention reuse. Ablations explain why caching FFN features fails, making this more useful than the FPS headline alone.Tested on Wan2.1-T2V-1.3B with three-frame chunks and A100-class hardware; no released LiveEdit checkpoint or mobile latency result is reported.Read or listen · 7 min audio
What are you trying to solve?
Preserve source content while cutting editing latencyInteractive editing is not ordinary generation: most pixels should remain stable while a localized region changes. Streaming systems must also operate causally, without future frames. The key mechanisms are distilling a bidirectional editor into a few-step causal student, anchoring each generated chunk to the source, and avoiding recomputation where prior features remain valid. Latency claims only become actionable when paired with resolution, hardware, and preservation tests.
LiveEdit: Towards Real-Time Diffusion-Based Streaming Video EditingStart here for a mechanism-level account of causal streaming editing: three-stage distillation, chunk-wise attention, and selective self-attention reuse. Ablations explain why caching FFN features fails, making this more useful than the FPS headline alone.Tested on Wan2.1-T2V-1.3B with three-frame chunks and A100-class hardware; no released LiveEdit checkpoint or mobile latency result is reported.Read or listen · 7 min audio
JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive DiffusionRead for a second causal-editing design that targets indefinite duration, two-step generation, source anchoring, and long-horizon drift. It adds a concrete 720p throughput point and a code release, but the abstract does not expose implementation or ablation detail.Reports about 30 FPS on one Nvidia B200 for its 16B system; use that hardware-specific result rather than treating it as general deployment speed.
Mage-Flow: An Efficient Native-Resolution Foundation Model for Image Generation and EditingUseful when latency comes from the whole image pipeline, not just denoising steps. It connects tokenizer cost, native-resolution packing, kernel fusion, and four-step distillation, while not addressing causal or long-horizon video editing.The abstract reports 1024² generation and editing times on one A100 for the 4B Turbo variants.
Which computations can be reused without erasing detail or changing the background?LiveEdit reports that reusing self-attention outputs on unedited tokens preserves its measured quality, while caching FFN features sharply damages detail and motion. Its mask is derived from the prior chunk’s source-to-edit latent difference, so the system recomputes edited regions and reuses context elsewhere. Fast-moving or newly appearing targets may lag because the mask looks backward.LiveEdit: Towards Real-Time Diffusion-Based Streaming Video Editing
When is denoising-step reduction alone insufficient for interactive image editing?When tokenization, variable-resolution handling, kernels, or memory dominate end-to-end time. Mage-Flow co-designs a lightweight latent tokenizer, native-resolution transformer, fused kernels, and four-step Turbo variants; it reports 1.02-second 1024² editing on one A100. Read it for stack-level optimization rather than as evidence about causal video consistency.Mage-Flow: An Efficient Native-Resolution Foundation Model for Image Generation and Editing
Distill speed without collapsing diversity or capabilityDistillation asks a student to reproduce a stronger model in fewer steps. Matching attractive teacher outputs can still discard low-probability modes, and combining generation with local and global editing can make capabilities interfere. Two useful diagnostics follow: evaluate coverage as well as visual precision, and train on states produced by the student’s own rollout so supervision reflects the distribution encountered at inference.
DistillAlign: Coordinating Mode Covering and Mode Seeking in Autoregressive Video DistillationRead to understand why visually strong distillation can lose teacher-distribution coverage. It introduces a precision/coverage diagnostic and a joint mode-seeking, mode-covering objective, but the abstract does not provide a deployment recipe.The reported evidence concerns autoregressive video distillation, including experiments with Wan-family teachers.
DanceOPD: On-Policy Generative Field DistillationUseful for composing text-to-image, local-editing, global-editing, realism, and guidance fields in one flow-matching student. It motivates on-policy state queries; it does not establish the method for non-flow generative formulations.Evidence in the abstract covers the listed generation, editing, realism, and classifier-free-guidance absorption experiments.
Why can strong visual scores hide a weak distilled video model?A mode-seeking objective can concentrate the student on high-probability teacher outputs, improving apparent precision while reducing coverage and diversity. DistillAlign proposes measuring student–teacher precision and coverage in a shared latent space, then combines a mode-seeking DMD objective with a consistency-based mode-covering constraint.DistillAlign: Coordinating Mode Covering and Mode Seeking in Autoregressive Video Distillation
How can one student absorb generation and editing capabilities that interfere?DanceOPD represents text-to-image, local editing, global editing, realism, and guidance as velocity fields over a shared flow state space. It routes each sample to one field and queries supervision at a low-noise state induced by the student’s rollout. This targets capability composition while preserving an anchor generation capability in the reported flow-matching experiments.DanceOPD: On-Policy Generative Field Distillation
Turn generative priors into geometry and physics predictorsA pretrained visual diffusion model contains representations useful beyond pixel synthesis. Adaptation can map those representations to a dense estimate, supervise motion-sensitive regions, or expose final video latents to a 4D decoder. The right interface depends on the output: per-pixel depth needs sharp boundaries, robotic simulation needs plausible contact and trajectories, and dynamic 3D generation needs geometry that remains stable over time.
Marigold V2: Revisiting Diffusion Transformers for Monocular Depth EstimationRead for a concrete example of converting a multi-step generative DiT into a single-step dense predictor using representation alignment and specialized fine-tuning. It supports depth and related dense regression, not image synthesis quality claims.Depth gains are reported on KITTI and ETH3D, with additional results for normals and intrinsic decomposition.
PhysisForcing: Physics Reinforced World Simulator for Robotic ManipulationUseful when a diffusion video model must support manipulation rather than merely look realistic. It identifies deformation and bad contact relations as failure sources and supervises trajectories plus semantic relations; real-robot validation is not described.Evaluated on R-Bench, PAI-Bench, EZS-Bench, and the WorldArena action-planner protocol.
Beyond Pixels: From Video Priors to 4D WorldsRead for a reusable latent interface between video diffusion and explicit 4D decoding. It avoids RGB reconstruction and reports transfer across compatible generators, but that transfer requires a shared VAE family.One checkpoint was trained on roughly 1,000 reconstruction clips and evaluated on Text4D-200 and I4D-200.
How can a multi-step image generator become a cheap depth estimator?Marigold V2 targets one-step inference from pretrained flow-matching DiTs. It attributes naive-training artifacts to poor adaptation and uses semantic representation alignment plus a two-stage Sinkhorn-loss fine-tuning protocol. Read it when out-of-distribution depth and thin structures matter; the reported improvements cover KITTI, ETH3D, and related dense regression tasks.Marigold V2: Revisiting Diffusion Transformers for Monocular Depth Estimation
What should a video world model learn beyond plausible-looking frames?For manipulation, supervise the events that expose physical failure: moving-object trajectories and relations between interacting regions, especially at contact. PhysisForcing aligns DiT features with reference point trajectories and semantic inter-region relations, reporting gains in embodied-video benchmarks and closed-loop planning success.PhysisForcing: Physics Reinforced World Simulator for Robotic Manipulation
When can video latents serve as an interface to dynamic 3D generation?Latent-to-4D aligns final denoised video latents directly with a pretrained 4D decoder, avoiding an intermediate RGB reconstruction cascade. A checkpoint trained on roughly 1,000 reconstruction clips transferred across video diffusion transformers sharing the same VAE family. Compatibility therefore depends on the latent interface, not merely on both models processing video.Beyond Pixels: From Video Priors to 4D Worlds