Get Started
Home
Topics
Search
Library
Research questionHow can autoregressive language models add recurrent cross-token memory without breaking KV caching or one-forward-per-token decoding?Standard transformers rely on attention and cached keys and values during generation, but they do not naturally carry a separate recurrent state across tokens. Adding one can alter the decoding interface or require additional forward passes.
AI
AI Memory
Inference Optimization
LLM Pretraining & Post-training
Machine Learning
Natural Language Processing
Research Paper
Technology
Latest papersRecent research connected to this question, newest first.Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling BehaviorThe evidence concerns a Latent Recurrent Transformer augmentation in which a high-level source-layer hidden state from the previous token serves as recurrent memory alongside standard attention and KV caching. Training uses interleaved parallel refinement instead of fully sequential unrolling. Reported experiments use 1.3B- and 2.1B-parameter nanochat-style backbones across varied training budgets, measuring BPB, CORE, and decoding latency; the reported latency overhead is 9%, with comparisons to PonderLM-2 and a three-loop Transformer.research paper · Sep 1, 2026
Related questions
How can autoregressive LLM decoding generate multiple tokens in parallel at large batch sizes without sacrificing quality?How can low-bit KV caches save autoregressive decoding memory without losing long-context retrieval quality?How can long-context language models reduce per-token reads from million-token KV caches without losing accuracy?How can sparse autoencoders capture language-model features that persist across token sequences?