A training-free swap to Sliding Window Attention with Attention Sinks matches or beats expensive linear-attention conversions of pretrained LLMs, and on long-context retrieval it recovers 2–10× more of the original model’s accuracy.
You’re serving a 70B chat model and your GPU bill is dominated by the KV cache growing linearly with every user token. A popular fix in 2024–2025 is to distill the model into a linear-attention variant (LoLCATs, Liger-GLA, QRWKV, Mamba-in-the-Llama) using tens of millions to billions of tokens, hoping to keep quality while getting O(1) memory per step. This paper argues that most of that engineering is unnecessary: just cap attention to the last w tokens plus the first 4, ship it, and you’ll usually do better than the distilled model, with zero fine-tuning.
The core mechanism is a change to the attention mask at inference time, nothing else. Instead of each token attending to all prior tokens, it attends only to the previous w (window sizes 64 to 512 in the paper) plus the first 4 tokens of the sequence. Those first 4 are the Attention Sinks: positions LLMs empirically dump surplus attention mass into, so dropping them out of the window causes a cliff in quality. Keeping them stabilizes generation for arbitrarily long contexts while the KV memory stays bounded at roughly w + 4 entries per layer.
Why not just use linear attention? Linear attention replaces the softmax with a kernel trick, exp(q·k) ≈ φ(q)·φ(k), so past keys/values collapse into a fixed-size running state. That state has to overwrite itself as the sequence grows, and it needs training (usually via LoRA) to learn a decent kernel like Hedgehog kernel. SWA sidesteps this entirely: no state to compress, no kernel to learn, and existing FlashAttention kernels already implement it.
The swap is essentially this:
# at each decoding step t, for each layer:
keep = list(range(0, 4)) + list(range(max(4, t - w + 1), t + 1))
k_used, v_used = k_cache[keep], v_cache[keep]
attn = softmax(q_t @ k_used.T / sqrt(d))
out = attn @ v_used
# evict everything not in `keep` from the KV cache
No weight changes, no distillation, no post-training. The receptive field still expands across layers (after l layers it covers roughly l·w tokens), similar to how stacked convolutions widen their effective view.
The prevailing move when quadratic attention hurts is to convert the model: distill it into a linear-attention or state-space variant so the KV cache disappears. This paper shows the opposite. The cheapest change, masking attention to a local window plus 4 sink tokens, already recovers nearly all of the base model’s quality and beats the distilled versions on long context. The load-bearing evidence isn’t the short-context averages, where methods look close. It’s the long-context Single Needle-in-a-Haystack and BABILong gaps, where linear-attention conversions collapse and SWA doesn’t.
•
On Single Needle-in-a-Haystack retrieval at 4K context with window 512, SWA reaches 19–23% accuracy across three subtasks while LoLCATs tops out at 8.8% and Liger-GLA at 0%. On BABILong at 4K, SWA recovers 25% of full-attention performance versus 5% for LoLCATs. This is the 2–10× gap that anchors the paper.
•
On short-context knowledge/reasoning (MMLU, ARC, HellaSwag, PIQA, WinoGrande), SWA(64, 4) wins the average in 9 of 11 base models tested, from Phi-1.5-1.3B up to Llama-3.1-70B and Qwen2.5-72B-Instruct. It recovers 99% of the teacher’s average score and 93.2% of MMLU, matching or edging out the best distilled method (QRWKV6) which needed 350M–700M tokens of post-training.
•
LoLCATs, the strongest cheap baseline, needs 40M tokens of LoRA distillation to hit 97.5% average recovery; SWA hits 99.0% with zero tokens.
•
Throughput and memory: SWA runs on FlashAttention and stays flat in speed past 1K context. At window 64 it uses less memory than linear attention; at window 512 it’s slightly higher but still faster than the linear + SWA hybrid kernel from ThunderKittens.
•
The authors also linearized Qwen3-8B, Phi-4-mini-reasoning, and Phi-4-reasoning-plus themselves with Gated DeltaNet, GLA, and QRWKV6 variants on ~100M tokens. All landed well below SWA(64, 4) on the same base models.
Reach for this when you’re running a pretrained chat or reasoning model and your inference costs are dominated by the KV cache, and you were considering paying an ML engineer to distill it into a Mamba-style or RWKV-style variant. Before doing that, just set the attention mask to keep the first 4 tokens plus a sliding window of 64–512 and re-run your evals. If your workloads are short-to-medium context (chat turns, RAG chunks under a few thousand tokens), you’ll likely recover 99% of quality at a fraction of the KV footprint, with no training run and no custom kernel.
The paper does not release a code repository or a new model. The technique is a mask change on top of FlashAttention, which most inference stacks (vLLM, TGI, llama.cpp) already expose as a config flag. The tested base models are all off-the-shelf on Hugging Face (Phi-1.5, Mistral-7B, Llama 2/3/3.1 up to 70B, Qwen2.5 up to 72B, QwQ-32B). Benchmarks used are standard: MMLU, ARC, HellaSwag, PIQA, WinoGrande for short context; Single Needle-in-a-Haystack and BABILong for long.
Before distilling your model into something exotic to shrink the KV cache, try the two-line mask change first. A local window plus 4 sink tokens is a shockingly hard baseline to beat.
•
Absolute long-context accuracy is still poor. SWA recovers more of the base model than linear attention does, but at 4K context on BABILong it’s 15% versus full attention’s 60%. If your product needs true long-range recall, neither SWA nor these linear-attention conversions is the answer; you want full attention or a hybrid with some full-attention layers, which the paper explicitly excludes from comparison.
•
The comparison is training-free SWA versus post-trained linear attention. The authors acknowledge that with matched post-training budgets, linear attention might close the gap. They call scaling-law comparisons future work.
•
Everything here is text LLMs on standard benchmarks. Agentic tool-use, code with long dependencies, and multi-modal or video models aren’t tested, and the receptive-field-grows-with-depth argument may not hold when the useful signal genuinely lives 10K tokens back.