Vidu S2 is a pair of streaming video models, one for interactive avatars and one for live video editing, that keep a segment-by-segment generator stable over long rollouts by re-noising the model’s own past outputs and replaying them with gradients flowing across segments, running at 720p and 25–42 FPS.
Today’s headline video models (Sora, Veo, and similar) run offline: you type a prompt, wait minutes, get a finished clip. That’s fine for pre-made content, but it can’t power a live avatar you talk to, a video call filter that restyles you in real time, or a VR passthrough that swaps your background. Those uses need frames coming out at video rate, responsive to new instructions mid-stream.
Streaming video generators run into a specific failure: they produce video in short chunks, and each chunk conditions on the previous one. Small errors in chunk 3 become bigger errors in chunk 4, and by minute two the character’s face has drifted, the background is smearing, or motion has collapsed. The clean fix, Self-Forcing, trains the model on its own generated history so training matches inference. Vidu S2 argues that fix is incomplete, and this paper’s technical contribution is mostly about finishing it.
The authors are from Tsinghua and Shengshu Technology; Vidu S1 was their previous system, capped at 540p talking-head style avatars with a fixed reference image.
The core new training trick is Self-Replay Forcing (SRF). Existing self-forcing feeds the model’s own past chunks back in during training, but those past chunks are clean (no noise) and gradients don’t flow through them. That means the model never learns to recover from the specific kind of noisy, imperfect history it will actually see when errors accumulate at inference.
SRF fixes both issues in a two-pass structure. First, let the current model generate a long autoregressive rollout on its own, then detach it (throw away the gradient graph and the KV cache). Second, take that self-generated trajectory, add fresh noise to every segment following Diffusion Forcing, and re-run the model over the whole noised trajectory in one gradient-enabled pass. Loss on a later segment can now flow back to earlier segments within this replay, so the model learns cross-segment corrections without having to remember the (expensive) original rollout graph. Supervision comes from Distribution Matching Distillation (DMD), with a perceptual loss added to keep outputs diverse.
# Self-Replay Forcing, one training step
with torch.no_grad():
rollout = student.autoregressive_rollout(ref, conds) # detached
noised = [add_noise(seg, sample_t()) for seg in rollout]
# single causal pass, gradients flow across segments
pred = student.causal_forward(noised, ref, conds)
loss = dmd_loss(pred, teacher) + perceptual_loss(pred)
loss.backward()
On top of SRF, three other pieces matter. A one-step latent-space Refiner upsamples the backbone’s low-resolution output to 720p, using a lower-noise cache than the backbone so spatial detail and long-range motion are handled separately. For editing, frame-aligned attention forces each target frame to attend only to the source frame at the same timestamp (preserving motion and timing exactly), while the reference image is visible to all frames (so the new appearance propagates). And a VLM agent watches generated frames to decide when a requested action is done and rewrites the next prompt accordingly, e.g., keeping “still holding the cup” in the prompt after the pickup completes.
Inference uses SageAttention, SpargeAttention, per-block W8A8 GEMM, CUDA Graphs, and Ulysses context parallelism across GPUs to hit real-time latency.
On StreamAV-Bench, a benchmark for streaming audio-driven avatars with a Progressive and an Interactive track, Vidu S2-Avatar reports the top score on every metric shown, against 13 other systems including Self-Forcing, LongLive, and PixVerse R1. Notably, temporal synchronization error (AVSync) drops to 0.617 (lower is better) versus 0.855 for the next-best streaming baseline, and subject/background consistency scores sit near ceiling (0.998 / 0.993), which the authors read as evidence the identity and scene don’t drift over long rollouts.
For editing, on Sparkle-Bench Vidu S2-Editing hits 4.00 on both global instruction following and foreground motion preservation, top of the table. On the joint OpenVE + RefVIE editing benchmarks it scores 4.26 overall vs 3.92 for the strongest offline baseline (Bernini-R 14B) and beats streaming competitor Decart-Lucy2.5. On the ViViD unpaired virtual try-on test, VFID drops to 9.95, roughly half of the next-best system’s 19.51, though this is a distributional metric on one specific try-on task.
Human preference (GSB) tests on the authors’ internal benchmark, run by 20 trained annotators, show Vidu S2-Avatar preferred over Runway, PixVerse, and HeyGen in the large majority of pairs across overall quality, motion, expression, and consistency, with some categories at 100%. Duration-stratified ratings from 10 to 90 seconds stay high, which is the specific evidence for the “doesn’t drift over long streams” claim.
One caveat on reading these numbers: SRF is presented as part of the whole system, and the paper does not run an ablation isolating SRF from the Refiner, the data pipeline changes, or the preference-optimization stages (Diffusion-DPO and Streaming NFT). So the benchmark gains support “the full system works well,” not “SRF specifically caused N points.”
If you’re building a streaming generative video product (live avatar, real-time filter, VR passthrough editor), the concrete takeaways worth stealing are:
•
The re-noise-and-replay training pattern. If you already do self-forcing style training and see drift over long rollouts, SRF’s recipe (detach the rollout, re-noise it with diffusion forcing, do one gradient-enabled pass over the whole thing) is a self-contained change to try. It targets exactly the mismatch where training sees clean history and inference sees noisy history.
•
Separate a fast low-res backbone from a one-step refiner if you need 720p at video rate. The paper’s design keeps the expensive autoregressive loop at low resolution and pays for high-res only in a single refinement step per chunk, with different cache noise levels for each.
•
Frame-aligned attention for editing. If you want an edit to preserve source motion exactly, restricting each target frame to attend to its time-matched source frame (rather than all source frames) is a simple architectural constraint that gives you that guarantee at training time.
•
Filter training data by measured clarity, not nominal resolution. The paper argues 1080p footage can be visibly blurry after compression, and their multi-signal clarity score is worth replicating if you train on scraped web video.
What this paper does not give you: released weights or code (none is linked in the paper text beyond the product demo at vidu.com/vidu-stream), specifics of training compute, dataset size beyond the 800k filtered clips for editing, or an ablation isolating any single technique. If you want to reproduce SRF, you’ll be implementing from the description.
The evaluation compares against a large slate of systems, but the internal-benchmark GSB comparisons and the duration-stratified curves are all judged by the authors’ own annotators on their own benchmark; product-quality claims like “preferred 100% over HeyGen” reflect this specific test set and rubric, not deployment behavior. Also, several baselines in the tables are themselves streaming systems from other 2026 preprints that a typical reader won’t have independent quality intuitions for.
The spatial-video section is framed by the authors as a feasibility exploration: monocular output is converted to stereo via per-frame depth estimation and disparity warping, with lightweight hole-filling. The paper does not quantitatively evaluate stereo quality or user comfort in a headset. Treat it as “the pipeline can produce stereo output,” not “this is a validated VR experience.”
Finally, the Refiner, backbone, VAE, and multi-GPU scheduling stack are what actually make 720p at 25–42 FPS possible on “regular consumer GPUs” (the paper doesn’t specify which GPUs or how many). Reproducing the latency numbers requires reproducing that whole stack, not just the training method.