Conf-KV sizes the KV cache per decoding step using the model’s own next-token confidence, expanding the budget on uncertain steps and pruning hard on confident ones, cutting peak KV memory by ~2.8× while keeping quality close to the full cache.
You’ve shipped a long-running agent: it browses pages, calls tools, holds a 20-turn conversation. After a while, the KV cache (the stored attention keys/values for every past token) eats most of your GPU memory and slows every new token. The usual fix is a sliding window: keep the last N tokens, drop the rest. That works until the user asks about something from turn 3, which is now gone. Smarter policies like H2O (Heavy-Hitter Oracle) rank old tokens by how much attention they’ve historically received and keep the heavy hitters. But all of these decide what to keep using signals from the past. They never ask the obvious question: is the model struggling on this token, right now?
Every decoding step already produces a probability distribution over the next token. Its shape tells you whether the model is sure (one spike) or hedging (flat). Conf-KV turns that into a scalar confidence score by mixing three cheap signals: normalized entropy, the log-prob gap between the top two candidates, and the top token’s probability. If confidence is above a threshold, the cache manager uses a tight token budget. If it’s below, it uses a looser one. The intuition: when the model is unsure, it probably needs more context to recover, so don’t throw context away yet.
Inside whichever budget is chosen, surviving tokens are ranked by a blend of recency and an Exponential moving average of how much attention mass they’ve accumulated. A fixed protected window of the most recent tokens is never evicted, so local coherence is safe. The system also composes with two orthogonal tricks: older retained tokens are stored in INT8 while the recent window stays in FP16, and deeper layers get smaller budgets (the pyramidal variant, borrowed from PyramidKV). The whole thing is training-free.
for step in range(max_tokens):
logits, attn = model(x_t, cache)
p = softmax(logits)
c = 0.4*(1 - norm_entropy(p)) + 0.3*sigmoid(margin(p)) + 0.3*p.max()
budget = N_high if c >= tau else N_low
for layer in cache:
layer.update_ema(attn)
if len(layer) > budget:
scores = alpha*layer.attn_ema + (1-alpha)*layer.recency
layer.evict_lowest(keep=budget, protect_last=P)
layer.quantize_to_int8(older_than=W)
x_t = sample(p)
The prevailing assumption in cache eviction is that token utility is a property of the past: how often it was attended to, how recent it is, what the prompt looked like. This paper shows the opposite. The most useful signal for when to keep more context is the model’s uncertainty about the very next token, which is sitting in the logits for free on every step. The cleanest evidence is the matched-rate ablation that isolates the schedule from the ranker, not the headline retrieval score.
The load-bearing finding is the matched-rate ablation: if you evict at the same rate and frequency as Conf-KV but pick victims randomly, perplexity collapses to 36.5, worse than a dumb sliding window. Recency-only gets 32.1, attention-only 31.5, full Conf-KV 30.9. So both the confidence-gated schedule and the ranker carry weight. Independently, the authors ablate the last 256 tokens of context and measure how much the next-token distribution shifts in KL divergence: confidence and that shift correlate at r = -0.77 on GPT-2 (weaker but consistent on larger models), which is the mechanistic claim that confidence really does track context demand.
Secondary numbers, in order of how much they should move your priors:
•
Needle-in-a-Haystack up to 32K tokens: 91.4% retrieval vs 53.8% (sliding) and 80.6% (H2O (Heavy-Hitter Oracle)). Trace shows confidence drops right when the retrieval query lands, expanding the budget on demand.
•
VisualWebArena: keeps 95.3% of full-KV task success at 2.8× lower peak memory, vs an 11-point drop for sliding window.
•
Memory at scale: Qwen-32B at 4K generated tokens goes from 15.8 GB to 2.6 GB of KV, which changes what batch size fits on an 80 GB H100.
•
Perplexity: at matched memory with a 512-token sliding window, the pyramidal variant closes 74% of the gap to full KV on GPT-2.
Reach for this when you’re running a long-context agent (web automation, multi-turn assistant, long-document Q&A) where the KV cache is dominating GPU memory and you’re currently using a sliding window or H2O (Heavy-Hitter Oracle)-style policy. The wins are largest on workloads with bursty difficulty: most steps are easy (boilerplate, formatting, common continuations) and a few are hard (retrieving an entity from far back, parsing an unusual page). A static cap either over-provisions for the easy steps or starves the hard ones; confidence-gating lets you spend memory where it matters.
Nothing is released yet. The paper says code will be posted on acceptance. The method is training-free and the algorithm is short enough to reimplement in an afternoon on top of HuggingFace; the harder parts are the contiguous compaction and the fused INT8 dequant in the attention kernel. It composes with PagedAttention, quantization (KIVI, KVQuant), and speculative decoding, since it only decides which tokens stay live, not how they’re physically laid out.
The logits you already compute know whether the model needs more memory; spend the cache budget where uncertainty says it matters. This works because the eviction decision is forward-looking and nearly free: the next-token distribution is on the critical path anyway. It stops working when entropy stops being informative, which is exactly the regime (high-temperature sampling, very short contexts) where you didn’t need adaptive eviction in the first place.
•
The KL-shift correlation that justifies the whole policy is strong on GPT-2 (r = -0.77) but much weaker on the larger models actually used in deployment (r ≈ -0.36 to -0.41). The mechanism is real but not as crisp at scale as the headline suggests.
•
Contiguous compaction is fast in a single-stream prototype but maps awkwardly onto PagedAttention-style paged serving, which is what production stacks like vLLM use. Token-level eviction creates partially dead blocks; integrating cleanly is non-trivial.
•
Failures cluster on a specific pattern: the model is confidently wrong right before needing a rare entity, so the policy tightens the budget at the worst moment. Raising the threshold helps but costs memory, so the tunability is real but not free.