Techniques that make trained models faster and cheaper to serve: quantization, speculative decoding, KV-cache tricks, kernel fusion, batching, and architectural choices for low-latency generation.
Inference optimization: spend memory, bandwidth, and compute where generation needs themInference cost is not one bottleneck. During long generation, stored attention keys and values—the KV cache—consume memory and increase the work per token. Other workloads are limited by repeatedly reading that cache, low-precision execution, running every layer, or generating expensive reinforcement-learning rollouts. Measure capacity, bandwidth, compute, and reproducibility separately before choosing a technique.The readings are organized by the decision each technique changes: which cached tokens survive, which context is read or re-encoded, how quantization interacts with recurrent and cached state, and whether training can prepare a cheaper inference path. Compare quality at a fixed memory or latency target; speedups from different models, stacks, and workloads are not a common ranking.Begin with CONF-KV for a mechanism-first view of adaptive cache budgets. It connects next-token confidence to context demand, separates budget scheduling from victim selection, and explains why paged-serving integration may be harder than the policy itself.
CONF-KV: Confidence-Aware KV Cache Eviction with Mixed-Precision Storage for Long-Horizon LLMStart here for a mechanism-first explanation of confidence-gated KV budgets, attention-and-recency ranking, protected recent tokens, and mixed-precision storage. Its matched-rate ablation helps separate adaptive scheduling from token selection.Results cover four model families, generation to 4K tokens, retrieval to 32K, and 75 VisualWebArena tasks; paged-serving integration remains non-trivial.Read or listen · 6 min audio
What are you trying to solve?
Choose what the KV cache keepsAutoregressive attention stores keys and values for prior tokens, so long outputs can exhaust memory. Retention policies make different bets about future utility: confidence can expand the budget at difficult steps; threshold-free allocation can adapt across mixed inputs; beacon queries can represent distant plans likely to be revisited; or prompt-protected random eviction can exploit redundancy in reasoning traces.
CONF-KV: Confidence-Aware KV Cache Eviction with Mixed-Precision Storage for Long-Horizon LLMStart here for a mechanism-first explanation of confidence-gated KV budgets, attention-and-recency ranking, protected recent tokens, and mixed-precision storage. Its matched-rate ablation helps separate adaptive scheduling from token selection.Results cover four model families, generation to 4K tokens, retrieval to 32K, and 75 VisualWebArena tasks; paged-serving integration remains non-trivial.Read or listen · 6 min audio
ReFreeKV: Towards Threshold-Free KV Cache CompressionUseful when cache policies must handle open-domain variation without a separately selected budget threshold for each input class. It introduces the threshold-free objective and reports evaluation across 13 datasets.The abstract does not expose the allocation algorithm or operational tuning details.
Random Attention: Rethinking KV Cache Eviction for Efficient ReasoningProvides a low-overhead baseline that protects the prompt but randomly evicts reasoning tokens per head, testing whether a learned or computed importance score is necessary.Evidence covers four models, six reasoning tasks, and vLLM comparisons; prompt fragility and trace redundancy may differ elsewhere.
What if one cache threshold fails across domains, lengths, and difficulty levels?ReFreeKV frames input-specific threshold selection as the problem and proposes adaptive, threshold-free budget allocation. Read it when request classes are hard to separate in advance; its abstract reports evaluation across 13 datasets but does not expose the allocation algorithm or deployment settings.ReFreeKV: Towards Threshold-Free KV Cache Compression
Is sophisticated token scoring always worth its serving overhead?Random Attention protects the prompt and randomly evicts generated reasoning tokens independently per head. Across its reasoning tests, it matched the strongest compared evictor while improving vLLM throughput. It is a useful baseline when reasoning text and attention heads contain enough redundancy to survive score-free selection.Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning
Reduce context reads—or change the representationEviction reduces capacity, but decoding can remain bandwidth-bound because every token reads the surviving cache. Declarative Attention lets a model name the region it needs so the engine can skip other entries. LatentPress instead turns documents or conversations into continuous memory tokens for a frozen decoder. One selects access at generation time; the other learns a smaller machine-facing context.
Language Models Can Control Their Own AttentionShows an alternative to scoring every cached token: have the model declare global, focused, or local attention regions so the serving engine can avoid reading irrelevant KV entries.Zero-shot tests used Gemma-4-31B and Qwen-3.6-27B on 15 long-context tasks.
LatentPress: Context Compression Beyond Text and VisionUseful for repeated document or conversation consumption: it replaces readable summaries with compact continuous memory tokens consumed directly by a frozen decoder.Tested on LongMemEval and LongBench-QA; the writer is reader-matched, and 16× compression reduced LongBench-QA quality.
How can a model avoid scanning an entire long-context cache for every token?Declarative Attention has the model declare global, focused, or local attention modes inside its reasoning trace. The engine parses those declarations and skips irrelevant KV reads. Read it when repeated full-context reads dominate decoding and a measured accuracy tradeoff is acceptable.Language Models Can Control Their Own Attention
When is latent context compression preferable to text summarization?LatentPress trains a small, reader-matched writer to encode histories or documents into continuous tokens consumed through a frozen decoder’s embedding interface. It avoids reconstructing readable text and targets repeated machine consumption; in its tests, 4–8× compression was stronger than 16× on LongBench-QA.LatentPress: Context Compression Beyond Text and Vision
Quantize with serving state in viewLow precision changes more than weight storage. Hybrid models carry recurrent state, attention layers use KV scales, fused kernels may combine separately calibrated modules, and prefix caches preserve hidden server state between requests. These readings connect quantization to kernel compatibility and reproducibility rather than treating a low-bit format as an isolated model property.
Same Request, Different Answer: Quantization Amplifies Cache-Induced Divergence in LLM ServingAdds a correctness dimension to optimization: controlled experiments show prefix-cache state changing agent trajectories, with greater divergence under four-bit weights in the tested workload.Measured on 80 serial, batch-one tool-use episodes across two engines and four weight formats.
Can recurrent layers in a hybrid LLM tolerate four-bit weights and activations?Minima quantizes all linear layers of a 27B Qwen3.8 hybrid, including Gated DeltaNet, to NVFP4 W4A4. Its analysis attributes robustness partly to local block scaling, gate nonlinearities that compress error, and a recurrence in which injected noise plateaus rather than compounds. It also identifies a scale mismatch between per-module calibration and fused GEMMs.Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM
Why can the same cached request follow a different agent trajectory?Prefix-cache state is normally absent from the request. In an 80-episode serial tool-use workload, cache-enabled trajectories diverged more often at four-bit than at 16-bit, while recomputation was bit-identical in the tested configurations. Include cache state and reset policy in reproducibility controls for quantized agent serving.Same Request, Different Answer: Quantization Amplifies Cache-Induced Divergence in LLM Serving
Prepare cheaper generation during trainingSome inference paths improve when training explicitly prepares them. Layer dropout teaches one model to tolerate missing depth, supporting early exit, layer skipping, and self-speculation. Online draft co-training instead maintains a separate speculative decoder during reinforcement-learning post-training. These apply at different lifecycle stages and should not be treated as interchangeable serving switches.
Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and InferenceRead when model training can be changed to support inference-time early exit, intermediate-layer skipping, or self-speculative decoding instead of applying sparsity only after training.Models ranged from 271M to 8.2B parameters; pretraining experiments ran on Cerebras CS-3 systems.
Online Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-TrainingUseful for RL post-training systems whose rollout cost justifies co-training a speculative draft while preserving context- and pipeline-parallel execution at large model and context scales.Evidence covers RL post-training models up to 122B and contexts up to 256K; this is not a benchmark of ordinary online serving.
When should layer skipping influence the pretraining recipe?Use layer dropout when you control pretraining and expect to skip intermediate layers, exit early, or let the model draft with part of its own depth. Across models from 271M to 8.2B parameters, tuned schedules reportedly reduced training FLOPs and enabled faster inference with negligible reported accuracy loss.Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference
When is online draft co-training useful for speculative decoding?Read this for large-scale RL post-training where rollout generation dominates cost and the draft must track a changing policy. The system adds branch attention to context-parallel execution and transports target features across pipeline stages, addressing models up to 122B and contexts up to 256K rather than ordinary standalone serving.Online Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Training