This survey frames memory in Autoregressive video generation as a persistent state maintained across generation steps under bounded context, distinct from the local window, and organizes the field by carrier form, preservation goal, lifecycle operation, learning setup, and evaluation.
Say you’re building a system that generates a 5-minute video one chunk at a time, each chunk conditioned on what came before. Around minute 2, the character you established in minute 1 walks back on screen. The model has already dropped that character out of its attention window because keeping the full history is quadratically expensive. The face changes. The shirt color drifts. A door someone opened in minute 1 is closed again.
That’s the memory problem this survey tries to systematize. The paper argues that existing solutions (rolling frame windows, KV cache retention, retrieval banks, 3D scene priors, compressed latent summaries) are all attacking the same underlying issue, but the literature calls them different things (“context”, “cache”, “history”, “state”) and has no shared vocabulary. Prior surveys on video diffusion and world models treat memory as a side issue rather than a central design axis of causal rollout. The authors want a unified framework so researchers can compare mechanisms rather than talk past each other.
This is a survey, so the “mechanism” is the taxonomy itself. The authors formalize step-by-step video generation as producing visual units (a token, frame, latent, or multi-frame chunk) one after another, each conditioned on prior units plus external inputs like prompts or actions. In principle the model conditions on all history; in practice it conditions on a bounded window $C_n$ of the last $W$ units. Memory $M_n$ is defined as any persistent state maintained across steps that carries information no longer available in $C_n$ and demonstrably influences later generation. A static text prompt doesn’t count. A KV cache you refresh, a retrieval store you query, or a scene graph you update does.
They then decompose every memory system into a lifecycle with named operators: form a query from current context, read from memory, integrate the read result with local context, generate the next unit, extract a write candidate, update memory, and apply a management operator that enforces a capacity budget through eviction, compression, or revision. This is deliberately analogous to classical differentiable memory work like Differentiable memory architectures.
The taxonomy has five axes. Forms asks what object holds the history: visual memory (RGB frames or VAE latents traceable to specific past observations), implicit memory (model-native neural state like KV cache entries or recurrent hidden states), explicit memory (structured world state such as scene graphs or 3D representations), and parametric memory (weights adapted during rollout). Functions asks what property is being preserved: entity identity, spatial layout, dynamics, semantic commitments, or causal state changes from interventions. Operations covers the lifecycle above. Learning covers training objectives and the distribution mismatch between teacher-forced training and self-rollout inference. Evaluation covers whether a benchmark actually probes memory versus generic video quality.
# One AR step under the paper's memory-conditioned formulation
q_n = form_query(C_n, cond)
r_n = read(q_n, M_n) # retrieve relevant history
h_n = integrate(C_n, r_n, cond) # fuse into generative context
y_n = generate(h_n) # produce next visual unit
w_n = write_candidate(y_n, C_n, cond)
M_tilde = update(M_n, w_n)
M_next = manage(M_tilde, budget) # evict, compress, or revise
Being a survey, findings are structural rather than empirical. The most useful observations:
•
Memory versus long context is not a window-size question. The distinction is operational: if historical information is selectively maintained, indexed, or revised across steps, it’s memory. If it’s just fed in densely every step, it’s active context. The same KV cache can be either depending on how the system manages it.
•
Six recurring failure modes of memoryless rollout, each tied to a preservation goal the survey later maps to a Function: entity forgetting (an object reappears wrong or missing), appearance drift (texture/color/lighting slowly changes), spatial inconsistency (layout doesn’t match when the camera returns), dynamics degradation (motion freezes or violates physics over time), semantic drift (narrative contradicts earlier events), and causal/state inconsistency (an opened door reverts). The authors are careful to note these symptoms can also come from bad dynamics modeling or rendering, not only missing memory.
•
Carrier choice trades fidelity against cost. Pixel-space frame memory is directly inspectable and preserves fine detail but grows storage linearly and can propagate visual artifacts back into the model. VAE latent memory is more compact and matches what latent generators consume natively, but couples the memory to a specific autoencoder and loses information the encoder discarded. Retrieved non-local frames can go stale relative to the current world state.
•
Interactive/action-conditioned generation stresses memory harder than open-ended generation because state-changing interventions (moving an object, opening a door) must persist after the action leaves the window, whereas passive rollout mainly needs appearance and layout to stay stable.
The paper does not run head-to-head experiments comparing methods. Method-level claims are attributed to cited systems, not measured by the survey.
If you’re designing a long-horizon video generator or World Model, the paper gives you a checklist worth walking through explicitly. Which of the six failure modes do you actually see in rollouts? That tells you which Function (identity, spatial, dynamic, semantic, causal) your memory needs to preserve. Then pick a carrier form appropriate to that function: structured explicit state is more natural for causal persistence of interventions, while VAE latent windows or retrieval banks are more natural for appearance and identity anchoring. Then specify the lifecycle operators concretely: what triggers a write, what the read query looks like, and how you enforce the capacity budget. The paper’s argument is that many published systems are underspecified on the management operator in particular, which is where long-rollout behavior actually breaks down.
If you’re evaluating someone else’s long-video system, the survey’s distinction between memory-revealing protocols and generic video-quality metrics is worth borrowing. A high score on temporal smoothness doesn’t tell you whether the model remembers a character from 30 seconds ago. A protocol that removes an entity from the active window and checks whether it reappears correctly does. Worth asking whether benchmarks you cite actually test this.
If you’re doing retrieval-augmented generation in text and curious about the video analog, the KV cache and retrieval-bank sections give a direct mapping: the same read/write/manage decomposition applies, but the write candidates are visual units and the retrieval keys are often visual or geometric rather than semantic.
The Awesome-AR-Video-Memory repository is the living index of methods cited.
This is a taxonomy paper dated September 2026 covering a fast-moving area, and the authors explicitly invite corrections for missed work. It does not benchmark methods against each other, so claims about which carrier or operation “works better” are structural arguments, not measured comparisons. The six failure modes are behavioral descriptions and, as the authors note, can have causes other than missing memory (imperfect dynamics, control execution, rendering), so attributing an observed failure specifically to memory requires separate diagnosis. Coverage of world-action models is deliberately selective, restricted to cases where memory mechanisms inform video generation rather than action prediction, so readers working primarily on policy learning should not treat this as a complete survey of that area. Finally, the excerpt provided covers roughly Sections 1-3.1; the Learning and Evaluation sections are referenced but not fully expanded in the text supplied, so specifics about training objectives and benchmark protocols beyond the naming of categories are not captured here.