Get Started
Home
Topics
Search
Library
Research questionHow can low-bit KV caches save autoregressive decoding memory without losing long-context retrieval quality?Low-bit KV caches reduce memory use but introduce model- and quantizer-dependent quality loss. Improvements in perplexity do not necessarily preserve long-context retrieval.
AI
AI Memory
Evaluation & Benchmarks
Inference Optimization
Machine Learning
Research Paper
Technology
Latest papersRecent research connected to this question, newest first.Quality Recovery for Quantized KV Caches via Low-Rank Attention AdaptationThe evidence evaluates fixed affine, NF4, KIVI, and KVarN cache formats on TinyLlama, Gemma, and Llama models, using low-rank Q/K/V projection adaptation. Results include held-out perplexity, associative retrieval, and a subset of RULER; the reported 2-bit setting recovered perplexity substantially but restored only 11–12 of 180 retrieval cases.research paper · Sep 2, 2026
Related questions
How can long-reasoning KV caches retain useful context while reducing memory and eviction overhead?How can long-context language models reduce per-token reads from million-token KV caches without losing accuracy?How can autoregressive language models add recurrent cross-token memory without breaking KV caching or one-forward-per-token decoding?How can long-context LLM inference reduce KV-cache memory without losing attention-head-specific information?