LiveEdit turns an offline bidirectional video-editing diffusion model into a causal streaming editor by distilling it in three stages and reusing self-attention features on unedited regions via a mask cache, hitting 12.66 FPS (~79 ms/frame) while preserving backgrounds.
Imagine you’re shipping an AR filter or a live-streaming tool where a user types “turn the red currants into frosted purple grapes” and expects the change to appear on their webcam feed in real time. The dominant recipe today (offline video editing with a Bidirectional Diffusion Transformer (DiT)) needs the full clip in memory before it emits frame one. Meanwhile the streaming video generation systems that do run causally, like StreamDiffusion and its successors, were built to synthesize whole scenes, so they trample the pixels you wanted to leave alone. LiveEdit targets the missing middle: causal, chunk-by-chunk editing that keeps unedited pixels bit-stable.
The backbone is Wan2.1-T2V-1.3B, a text-to-video diffusion transformer. LiveEdit adapts it in three passes. Stage 1 fine-tunes it as a normal offline editor: concatenate the source-video latent and noisy latent along channels (not sequence length, to avoid quadratic attention blowup) and train with standard noise-prediction loss. Stage 2 introduces Teacher Forcing (chunk-wise causal) with a chunk-wise causal attention mask (3 latent frames per chunk), so each chunk can only attend to itself and earlier chunks. This step matters because the authors show that naively truncating a bidirectional model’s attention to be causal makes attention weights flatten out uniformly over history, losing the local-neighbor bias the pretrained model relied on. Stage 3 applies Distribution Matching Distillation (DMD) to compress inference to 4 sampling steps and drops classifier-free guidance. Crucially, they skip the expensive ODE initialization that Self-Forcing uses, initializing the student directly from the Stage-2 causal weights.
At inference, the AR-oriented Mask Cache decides per-token whether to recompute or reuse. For chunk k, they take the previous chunk’s edited latent and source latent, compute their L2 distance per spatial location, and threshold it (dynamically set to prune ~70% of tokens) to get a binary edit mask. Edited tokens run the full block; unedited tokens reuse the self-attention output cached from the prior chunk. Cross-attention and FFN still run everywhere, because ablations show FFN features change too fast between steps to reuse safely.
for chunk_k in stream:
diff = l2(z_edit[k-1], z_src[k-1]) # per spatial location
mask = diff > tau # tau prunes ~70% of tokens
for token in chunk_k:
if mask[token]:
feat[token] = full_block(token) # SA + CA + FFN
else:
feat[token] = sa_cache[k-1][token] # reuse SA only
emit(decode(feat))
The prevailing move in streaming diffusion is to accelerate everything uniformly: distill all steps, cache all layers, treat every frame as a fresh generation. LiveEdit argues the opposite for editing: split the model’s work by where the edit actually is, and split the layers by which ones tolerate temporal reuse. Self-attention over unedited backgrounds is pure redundancy and can be cached across chunks with zero visible cost; FFN and cross-attention cannot. The load-bearing evidence is the cache-location ablation, not the FPS headline.
•
Caching location is not a wash. Caching self-attention on unedited tokens holds all six metrics roughly flat versus the no-cache baseline (Text Alignment even rises 0.265 → 0.270). Caching FFN features instead collapses everything: Background Consistency 0.956 → 0.841, Imaging Quality 0.720 → 0.513, Dynamic Degree 0.282 → 0.017. This is the finding that justifies the whole design: self-attention holds redundant spatial context, FFN carries high-frequency detail that must be recomputed.
•
Speed. End-to-end latency for 81 frames drops from ~200s (Stages 1-2, 100 NFE (Network Function Evaluations) with CFG) to 7.89s at Stage 3 (4 NFE, no CFG), which the paper reports as 12.66 FPS or ~79 ms/frame.
•
Head-to-head on a 120-pair benchmark against three offline editors (LucyEdit, InsV2V, VideoCoF) and three streaming translators (StreamDiffusion, StreamDiffusionV2, StreamV2V), LiveEdit tops Text Alignment (0.270 vs next-best 0.259), Background Consistency, and Imaging Quality, while streaming baselines exhibit background flicker or fail to localize the edit.
•
User study with 20 raters: LiveEdit gets 75% of “Best” votes for background preservation and a 100% top-3 rate on instruction consistency.
Reach for this design when you’re building a live video effect, an AR try-on, or a webcam filter where a text prompt should touch only part of the frame. The pattern generalizes past this specific model: (1) distill a strong bidirectional editor into a causal, few-step student rather than training a streaming editor from scratch; (2) at inference, derive a cheap edit mask from the L2 gap between source and previously-edited latents, and (3) cache only the layer that tolerates temporal reuse, which for diffusion transformers appears to be self-attention.
The paper does not link a code repo or released checkpoints. The foundation model (Wan2.1-T2V-1.3B) and the training data source (Ditto-1M, filtered to 20K pairs) are named but the filtered subset and the benchmark of 120 evaluation pairs are not stated to be released. Treat this as a design blueprint you’d reimplement rather than a drop-in artifact.
In streaming diffusion editing, cache the layer that stores context, recompute the layer that stores detail. Self-attention over unedited regions is the free lunch; FFN and cross-attention are not. The same split (mask by edit activity, cache by layer sensitivity) is a template for any real-time diffusion pipeline where most of the frame is supposed to stay put.
•
The mask for chunk k is derived from chunk k-1’s edit region, so fast-moving or newly-appearing edit targets can get missed for one chunk before the mask catches up. The paper’s cases are largely object-recoloring and texture swaps on relatively stable subjects.
•
The 70% token-prune rate and the claim that self-attention caching is free are validated on a single 1.3B backbone with a specific 3-frame chunk size. Whether the SA-vs-FFN redundancy split holds at larger scale, longer chunks, or on non-Wan DiTs is not tested.
•
The 12.66 FPS number is on A100-class hardware after full three-stage training on 8 A100s. “Real-time on edge devices”, which the intro invokes, is aspirational; the paper does not measure mobile or consumer-GPU latency.