BeaconKV compresses the KV cache of long-reasoning LLMs by keeping a small set of geometrically diverse beacon queries from past decoding steps, so the eviction policy can predict which distant tokens the model will re-attend to later, cutting peak memory by up to 5.8×.
When a reasoning model like DeepSeek-R1 or Qwen3 thinks out loud for 20k-30k tokens, every generated token adds a key and value vector that stays in GPU memory for the rest of decoding. The paper’s concrete example: Qwen3-4B generating 32K tokens at batch size 16 needs about 77 GB just for the cache, saturating an 80 GB A100. You either shrink the batch, cap the reasoning length, or compress the cache.
The standard trick is eviction: score each cached token by how much recent queries attended to it, drop the low-scoring ones. Prior reasoning-focused methods like RPC and R-KV do exactly this, using the last ~32 queries as a stand-in for what the model will want next. The bet is that recent attention predicts future attention.
The authors show that bet breaks in long reasoning traces. The model periodically jumps back to re-read the original problem statement or the plan it wrote thousands of tokens ago. Recent queries never point there, so those distant KV pairs get evicted, and when the model tries to revisit them the context is gone.
The paper’s central observation: queries during long reasoning split into two behaviors. Most are local (attend to nearby tokens, ongoing arithmetic). A minority are global, which the authors name Thought Revisiting Tokens (TRT). These reach back hundreds or thousands of positions to earlier context, like the problem constraints. TRTs are sparse and appear unpredictably across layers and heads.
The useful structural finding: when you look at the query vectors before rotary position encoding is applied (pre-RoPE query), the global queries are not random. They cluster tightly in embedding space, i.e., different TRTs at different decoding steps produce similar query directions. So you can represent the whole family of past global queries with a small set of beacon queries, one or a few per cluster.
BeaconKV’s eviction score at time t uses attention weights from two query groups combined: the last 16 recent queries (for local coherence) plus 16 beacon queries selected from all past queries (to catch future TRT targets). Beacon queries get RoPE applied at the current step t, simulating “if a TRT fired right now, what would it attend to?” Scores are aggregated per KV position using max-pooling across queries and heads, since TRT signals are rare and high-magnitude and would get washed out by averaging.
Picking beacons over a growing query history naively would itself blow up memory, especially with Grouped-Query Attention (GQA) multiplying query count. The authors introduce Continual Farthest Point Sampling: each head keeps a bounded buffer of past pre-RoPE queries. When it fills to 32 entries, run FPS to downsample to the 16 most geometrically distinct ones, then keep collecting. FPS greedily picks the query least similar (by cosine) to what’s already selected, so the buffer keeps evolving to cover new regions of query space.
# per attention head, on each decoding step t
append(q_pre[t], buffer)
if len(buffer) >= 32:
buffer = farthest_point_sampling(buffer, k=16)
if kv_cache_size == B_max:
beacons = [rope(q, t) for q in buffer] # aligned to now
recents = [rope(q, tau) for q, tau in last16] # original positions
W = attention(beacons + recents, kv_cache)
score[j] = max over heads and queries of W[:, :, j]
keep = prefix_tokens + last_recent + topk(score)
kv_cache = kv_cache[keep]
Evaluated on R1-Distill-Qwen-7B, R1-Distill-Llama-8B, Qwen3-4B, and Qwen3-14B across AIME 2024, MATH-500, GPQA-Diamond, and LiveCodeBench, with output length capped at 32K tokens.
•
Accuracy under tight budgets. At matched KV budgets, BeaconKV generally beats RPC, SnapKV, and R-KV. The paper’s headline gap is up to 31.7 percentage points, on Qwen3-14B / AIME24 at a 1024-token KV budget. Gaps are largest in the low-budget regime.
•
Memory and throughput. On Qwen3-4B with 32K generation, a 2K KV budget cuts peak memory from 77.0 GB to 13.3 GB (5.8×), lifts throughput from 82 to 356 tokens/s (4.3×), and loses only ~3 points of LiveCodeBench accuracy vs full KV.
•
Matched-budget efficiency vs RPC. At the same 2K budget and batch 192, BeaconKV has essentially the same throughput and memory as RPC but scores +6.3 points on LiveCodeBench. At a tighter 1K budget the gap widens to +12.3 points.
•
Ablations isolating the mechanism. Replacing beacons with just “initial queries + recent queries” underperforms BeaconKV across all four models. This is the paper’s evidence that keeping the beginning of the trace is not enough; the benefit comes from Continual FPS dynamically tracking evolving global query clusters. A separate ablation confirms max-pooling beats mean-pooling under tight budgets, consistent with the sparse-but-strong TRT signal story. The (recent=16, beacon=16) split is the reported sweet spot; pushing to (1, 31) tanks latency and accuracy.
Note that RPC and R-KV are the reasoning-specific baselines the paper positions against; the comparisons are all under matched KV cache budgets on the same models, not vs closed models or vs uncompressed inference (except in the efficiency table).
•
If you serve open-weight reasoning models and hit memory ceilings on long CoT, BeaconKV is a drop-in inference-time change: no retraining, no architecture edit, no gating module. The mechanism plugs into the same eviction loop as RPC or SnapKV. The paper’s own comparison target is running these models on a single 80 GB GPU, so this is aimed squarely at single-node deployment where full KV won’t fit at your desired batch size.
•
If you’re already using RPC or SnapKV, the concrete question worth testing is whether swapping in beacon queries recovers accuracy at your current budget, or lets you drop the budget further at your current accuracy. The paper shows both regimes on four models; your workload may sit closer to one end.
•
If your workload isn’t long-horizon reasoning, hold off. The whole argument rests on TRTs, which the authors observe in extended CoT traces (~4k-15k output tokens on the tested benchmarks). Standard long-context retrieval, summarization, or short-generation chat aren’t evaluated, and the authors flag this explicitly in their limitations.
•
If you’re picking hyperparameters, the ablation points to n_recent=16, n_beacon=16 with max-pooling as the default. Skewing heavily toward beacons hurts short-range coherence and latency; skewing toward recent queries reverts you to RPC-like behavior.
•
The paper does not mention a public code release in the provided text.
•
Evaluation is exclusively open-source reasoning models on math, science, and coding benchmarks. Transfer to hosted models, non-reasoning workloads, or retrieval-heavy tasks is untested.
•
The largest advertised accuracy gap (31.7 pp) is a single (model, benchmark, budget) point. Typical gains across the reported grid are smaller, and on some (model, benchmark) pairs the methods are close.
•
BeaconKV still evicts. Under aggressive budgets it loses a few points vs uncompressed inference (e.g., 54.4 → 51.1 on LiveCodeBench at 2K budget on Qwen3-4B). It’s a memory-quality trade, not free.
•
The (16, 16) sweet spot and FPS buffer sizes are tuned on the evaluated models; the authors note hyperparameter sensitivity across models and tasks is not systematically studied.
•
The TRT observation and beacon clustering are demonstrated on specific layers/heads and one AIME sample in the figures. The paper argues via aggregate accuracy that the mechanism generalizes, but the mechanistic evidence is illustrative rather than exhaustive.