Self Gradient Forcing (SGF) fixes a blind spot in Self-Forcing video training: the model learns to read its own generated history but never gets supervised on how it writes that history into the KV cache. SGF adds a bounded second pass that opens gradients through cache-writing, letting a model trained on 5-second clips extrapolate to several-minute videos without identity or scene drift.
You’re shipping a streaming video generation product. The model produces frames one chunk at a time, and each new chunk conditions on a cache of what it already generated. At 5 seconds it looks great. At 60 seconds the main character’s face has morphed, the restaurant is now a parking lot, and the camera has teleported. This is the standard failure mode of autoregressive video diffusion trained with self-rollout.
The dominant recipe here is Self-Forcing: train the student on its own generated history instead of ground-truth clips, so training matches inference. It fixes exposure bias but, as this paper argues, leaves a specific gradient path unused. Everyone building on it (Rolling Forcing, Self-Forcing++, LongLive, and others) inherits the same gap.
Walk through what happens during self-rollout training. The model generates chunk i, denoises it to a clean latent, then runs that clean latent through the network one more time at a special “context timestep” to produce K/V entries. Those entries get appended to the cache. Later chunks read this cache when they denoise. That final “write” step, turning a clean generated frame into cache entries, is what the paper calls the cache-writing computation.
Here’s the problem. Self-forcing training truncates gradients through the cache to keep memory manageable. So when the loss on chunk 20 says “you attended to bad memory,” the gradient can update how chunk 20 reads the cache but never flows back to fix how chunk 5’s frame got written into the cache in the first place. The authors call this the historical context-gradient gap. Making the cache fully differentiable would work in theory but blows up memory, because you have to retain the autograd graph across the entire serial rollout.
SGF’s move: turn a serial-graph problem into a parallel-recomputation problem. Two passes:
# Pass 1: serial rollout, NO gradients (matches inference)
for i in range(num_chunks):
x_clean[i] = denoise(z_noisy[i], cache) # no grad
cache.append(write_kv(x_clean[i], t_ctx=0)) # no grad
record(x_clean, z_noisy_at_sampled_exit_step)
# Pass 2: parallel, WITH gradients on the recorded exit step
x_ctx = stop_gradient(x_clean) # frozen inputs
kv_ctx = write_kv(x_ctx, t_ctx=0) # gradients flow here
pred = denoise_parallel(z_noisy, kv_ctx, causal_mask)
loss = dmd_loss(pred).backward() # updates cache-writer
The recorded context latents themselves are detached, so SGF isn’t backpropagating through the denoising trajectory that produced them. It’s only training the writer function: given this generated frame, produce K/V entries that will be more useful to future chunks. The parallel reconstruction uses a causal mask that mimics the serial cache’s attention pattern, and the authors verify (via bfloat16 numerical analysis) that Pass 2 reproduces Pass 1’s forward computation to within roundoff error.
The prevailing view in self-rollout video training is that the important fix is what history the model sees: use self-generated rollouts instead of ground truth, so training matches inference. This paper argues that’s only half the fix. Exposing the model to its own history isn’t enough; you also have to supervise how that history gets encoded into memory, or the writer silently drifts as shared parameters get updated by other losses. The load-bearing evidence isn’t the headline benchmark, it’s the training-feasibility table showing that the direct alternative (differentiable cache) runs out of memory while SGF adds only ~8 GB peak and ~1.3s per five steps.
The finding that makes the thesis work: a model trained with only a 5-second window can extrapolate to minute-scale videos when the cache-writer is properly supervised. On matched 5-second VBench comparisons, SGF and self-forcing are roughly tied, which is the sanity check. The gap opens at 60 and 240 seconds, exactly where cache-writing errors have time to compound.
•
On aesthetic quality, background consistency, imaging quality, motion smoothness, subject consistency, and flickering, SGF beats matched self-forcing across both frame-wise and chunk-wise settings and across multiple initializations.
•
Blind human preference: all 10 matched pairwise comparisons favor SGF, across more than 1,900 judgments.
•
The one metric where self-forcing sometimes wins is dynamic degree, but qualitative strips show this comes from scene jumps and broken camera geometry, incoherent motion that inflates the score.
•
Training cost is modest: peak memory 79.01 GB \u2192 87.01 GB, wall-clock 10.39s \u2192 11.71s per five steps. The direct differentiable-cache baseline OOMs.
•
Two-pass forward recovery: 1.41% relative L2 error, cosine similarity 0.999886, which is about 1.8\u00d7 the bfloat16 precision floor.
Reach for this if you’re training or fine-tuning an autoregressive video diffusion model, especially one derived from Self-Forcing or its descendants. The recipe is drop-in: keep your existing serial rollout as Pass 1 (no gradients), add a Pass 2 that re-encodes the recorded clean latents at the context timestep with gradients enabled, and route your Distribution Matching Distillation (DMD) loss through the reconstructed K/V path. The paper uses FlexAttention with a compiled block-sparse causal mask to keep Pass 2 cheap. Because SGF only changes the gradient boundary, it composes with retrieval, sparse attention, KV compression, and longer-window tuning.
Code and models are promised at the project page, though at the time of writing the release is listed as forthcoming rather than live. The paper doesn’t specify the exact training data or base model provenance beyond noting that some baselines use released Causal Forcing and Self-Forcing checkpoints and others are reproductions by the authors under matched settings.
In autoregressive generation, teach the model to write memory, not just to read it. Exposure-bias fixes get the model to condition on its own outputs, but if gradients never reach the step that encodes those outputs into cache, the writer slowly drifts under pressure from unrelated losses. Cheap parallel replay of one exit step is enough to close the loop.
•
SGF is a bounded surrogate, not full backprop through time. It doesn’t update the sampled latents themselves or the denoising trajectory that produced them, so it can’t fix errors that originate inside a chunk’s own denoising steps.
•
Pass 2 has to faithfully mirror Pass 1’s attention geometry (sink positions, FIFO window, Rotary Position Embedding (RoPE) handling, chunk alignment). If those drift, SGF trains a writer for the wrong relation, which could be worse than not training it at all.
•
The gains are demonstrated on top of Self-Forcing-family training with Distribution Matching Distillation (DMD) supervision from a bidirectional teacher. Whether the same gap exists, or matters as much, in other long-video regimes (pure teacher-forcing, non-distilled, retrieval-heavy) isn’t shown here.