JoyAI-Video-Edit turns a bidirectional 16B video editor into a causal chunk-by-chunk streaming editor by distilling it to a two-step generator that is Classifier-free guidance-anchored to the aligned source chunk, achieving ~30 FPS 720p editing on one B200 while resisting long-horizon drift.
Imagine you’re shipping a live video pipeline. A creator points a camera and says “make it look like a Pixar scene, keep my dog, remove the couch.” Today you’d have to record the clip, run an offline editor over the whole file, and hand back a rendered result minutes later. Streaming edits require the model to emit each edited frame as source frames arrive, without ever seeing the future, without memory growing unbounded, and without slowly drifting off the source over a one-minute clip.
The dominant baselines here (offline diffusion editors like VACE) get high quality by attending bidirectionally over the whole clip, which is fundamentally incompatible with streaming. Prior streaming editors like SANA-Streaming and LiveEdit preserve causality but sacrifice a lot of quality and drift over long streams.
The recipe has three stacked ideas on top of a standard Multimodal Diffusion Transformer diffusion editor with an Multimodal Large Language Model condition encoder and a causal video Variational Autoencoder.
First, chunk-wise causal adaptation. The latent video is cut into small chunks (one latent frame = 8 pixel frames). Attention is bidirectional inside a chunk but causal across chunks, with a sliding window over recent chunks plus a permanent “sink” on chunk 0. That bounds memory and per-chunk compute regardless of stream length. To close the gap between clean-history training and messy self-generated history at inference, they use Resampling Forcing: during training, historical chunks are replaced by the model’s own one-step denoised estimates, detached from gradients.
Second, two-step distillation that stays faithful to the source. They start from standard Distribution Matching Distillation (DMD) (a student generator trained to match a frozen teacher’s score) and add Source-Anchored DMD. The teacher’s score is guided along two independent classifier-free axes: one for the text instruction, one for the aligned source chunk. Plain-English version of the guidance formula: take the conditional prediction, push it further away from a text-free prediction (to sharpen instruction following), and further away from a source-free prediction (to sharpen source fidelity). The source-anchoring lives only in the training target, so the deployed generator is a single forward pass.
Third, Long-Horizon Autoregressive Distillation: unroll long rollouts, but backprop through them in segments with graph clearing between segments and gradient accumulation, so the model actually sees the drifted late states without OOM.
for chunk_t in stream:
hist = kv_cache.window() + kv_cache.sink()
# two-step generator, source-anchored during training only
z_t = generator(noise, cond_tokens, source_t, hist)
frames_t = vae.decode(z_t)
emit(frames_t)
kv_cache.append(z_t) # bounded sliding window
The prevailing move when causalizing a video model is to train it to predict on clean history and then hope inference-time drift is small, and to distill to few steps using a single classifier-free axis for the text prompt. This paper argues the opposite. In streaming editing specifically, the source video is a second, equally important guidance axis, and it must be baked into the distillation target so the student stops drifting away from the pixels it was supposed to be editing. The ablation isolating Source-Anchored DMD from long-horizon distillation is what carries this claim, not the headline benchmark score.
The load-bearing evidence is the distillation ablation on LongV2VBench, their one-minute editing benchmark. Turning on Source-Anchored DMD alone lifts the overall score from 2.81 to 3.23, with global style jumping 3.61 → 4.24 and local change 3.43 → 4.00. Long-horizon distillation alone gets to 3.06. Both together reach 3.30. The pattern shows Source-Anchored DMD is what fixes appearance/source fidelity, while the long-horizon piece stabilizes late-rollout states (background change, local removal).
•
On LongV2VBench, the system scores 3.30 overall vs the strongest streaming baseline XMax-X2.0 at 1.71, and does so at 30.19 FPS at 720×1280, roughly 1.67× to 2.08× faster end-to-end than other streaming editors at lower resolutions.
•
On the short-clip benchmark OpenVE-Bench it scores 3.60, first among streaming editors on four of five categories, and competitive with strong offline systems like Kling-3.0 Omni (3.64) despite emitting causally.
•
Pairwise human eval prefers it over other streaming editors 81–90% of the time; against strong offline systems it lands roughly at parity (48% vs Bernini-R’s 44%; 44% vs Kling-3.0 Omni’s 56%).
•
End-to-end 266 ms per 8-frame chunk on a single B200, dominated by 185 ms of DiT denoising.
Reach for this design when you’re building a live video product: a virtual-camera filter, a real-time restyler for a streaming platform, a game-capture recolor pass, or a telepresence effect where edited frames must ship as source frames arrive. The concrete pattern to steal is the two-axis guidance during distillation: if your student generator needs to honor both an instruction and a fidelity source (a reference image, an input video, a control signal), don’t fold them into a single conditioning branch. Give the teacher independent classifier-free axes per source, and let the student absorb both.
Code is released at GitHub. The paper also introduces LongV2VBench, a 229-task one-minute streaming-editing benchmark across background change, global style, local add/change/remove, which is useful if you’re evaluating drift and throughput jointly rather than on the usual <10s clips. Training data details are partial: paired video-editing data is synthesized by propagating an edited keyframe with an image-and-video-to-video model or via latent-shared parallel I2V generation, but the underlying image-editing corpus JoyAI-Image is cited rather than fully described.
When distilling a conditional generator, treat every source of ground truth as its own guidance axis, not a single condition blob. In edit-style tasks the model has two masters (the instruction and the source pixels), and folding them together is what lets self-generated history quietly pull the output off the source. Give them separate CFG axes in the teacher and the student learns to hold both.
•
The two closest competitors this paper reports numbers for (SANA-Streaming, LiveEdit, XMax-X2.0) score very low on LongV2VBench (1.2–1.7 out of 5). The impressive gap is partly a statement that streaming editing was previously very rough, not just that this method is close to offline quality.
•
Real-time throughput assumes a single B200 GPU with FP8 quantization and compiled VAE paths. On commodity hardware the 16B model and 185 ms DiT step won’t hit 30 FPS, and the paper doesn’t report a smaller variant.
•
Paired video-editing training data is synthetic, generated by propagating edits from image-editing supervision through another video model. Failure modes of that generator (motion artifacts, geometric drift on complex scenes) will be inherited, and the paper doesn’t quantify this.