Get Started
Research questionHow can long-context language models reduce per-token reads from million-token KV caches without losing accuracy?With million-token conversations, each generated token may require scanning the full KV cache, making decoding expensive. Proxy-based selection can reduce what is read but still adds linear work at every step.
AI
Inference Optimization
Natural Language Processing
Latest papersRecent research connected to this question, newest first.Language Models Can Control Their Own AttentionThe source evaluates its Declarative Attention protocol zero-shot on 15 long-context tasks using off-the-shelf Gemma-4-31B and Qwen-3.6-27B models. It reports 52.0% and 31.1% reductions in total attended tokens, with accuracy drops of 1.27 and 2.75 percentage points, respectively; evidence is limited to those models and tasks.research paper · Sep 2, 2026
Related questions
How can long-context LLM inference reduce KV-cache memory without losing attention-head-specific information?How can long-reasoning KV caches retain useful context while reducing memory and eviction overhead?How can shared-prefix KV caching accelerate diffusion language model serving without corrupting context-dependent KVs?How can low-bit KV caches save autoregressive decoding memory without losing long-context retrieval quality?
Home
Topics
Search
Library