Get Started
Home
Topics
Search
Library
Research questionHow can long-context LLM inference reduce KV-cache memory without losing attention-head-specific information?KV caches grow linearly with context length and can become a major inference-memory bottleneck. Because attention heads serve different functional roles, uniform compression may discard information that remains important for long-context processing.
AI
AI Memory
Evaluation & Benchmarks
Inference Optimization
Latest papersRecent research connected to this question, newest first.SGD-KV: Summarization Guided KV Cache CompressionThe evidence concerns Qwen2.5-7B-1M and Qwen3-32B evaluated on diverse long-context benchmarks, with contexts up to 1 million tokens and reported KV-cache memory reductions of up to 75%.research paper · Sep 3, 2026HeadWiseKV: Budgeted Per-Head Cache Residency for Hybrid Long-Context Language ModelsThe source studies training-free compression of residual global KV caches while preserving local, recurrent, and linear paths. It uses static multilevel history windows per physical KV head and evaluates downstream quality across four hybrid long-context models, with memory and serving measurements on Qwen3.6-27B.research paper · Sep 2, 2026
Related questions
How can long-context language models reduce per-token reads from million-token KV caches without losing accuracy?How can LLM serving adapt KV-cache capacity as attention demand changes during long-output reasoning?How can long-reasoning KV caches retain useful context while reducing memory and eviction overhead?How can we robustly compress LLM KV caches across open-domain inputs without input-specific budget thresholds?