Get Started
Topic · 64 recaps
Inference Optimization
Techniques that make trained models faster and cheaper to serve: quantization, speculative decoding, KV-cache tricks, kernel fusion, batching, and architectural choices for low-latency generation.
Play all
...
Posts
Questions
Home
Topics
Search
Library
Questions researchers are working on
Follow a question through Rcap’s explanations and the latest papers addressing it.
Search
Can intermediate LLM activations guide faster jailbreak search without weakening attack effectiveness?
Refusal behavior may be represented in transformer activations before the model produces its output. The difficulty is using that signal to reduce the cost of prompt search without losing the effectiveness of the resulting attacks.
Can post-training ternarization make language models smaller without unacceptable capability loss or slower inference?
Ultra-low-bit weights can shrink model storage, but nominal bit counts may not reflect the stored representation, uneven task degradation, or actual inference speed. Compression may therefore improve footprint without improving end-to-end deployment performance.
How can 70B language models fit on one GPU while preserving long-context speed and accuracy?
A 70B model must fit its weights and growing KV cache within one GPU’s limited memory. Long prompts make compression choices affect both decoding speed and model accuracy.
How can a single search agent improve multi-hop web research without sub-agents or test-time verification?
Multi-hop web research requires an agent to connect evidence across several pages while retaining the information needed for later steps. Long search trajectories can overwhelm the available context, making both training and reliable performance difficult without additional agents or verification passes.
How can agent runtimes avoid context poisoning and latency from growing histories during long-horizon skill execution?
Long-running agents often append observations, actions, and intermediate reasoning to their conversation history. As that history grows, execution becomes slower and earlier context can interfere with later decisions.
How can AI agents adapt execution routes as runtime evidence invalidates their planned continuation?
Agents often commit to a multi-step execution route before intermediate outcomes are known. When those outcomes change what should happen next, stale actions or broad replanning can waste computation and discard useful progress.
How can analog compute-in-memory attention perform softmax without costly analog-to-digital conversion?
Analog compute-in-memory attention produces scores as voltages, but softmax requires exponentiation and normalization. Converting those scores to digital form adds overhead before the attention weights can be computed.
How can attention heads be pruned in text-to-image diffusion transformers without losing prompt-specific object identity?
During denoising, semantic information may be maintained by structural template tokens and image-to-text interactions rather than by the prompt tokens that initially encode it. This makes it difficult to identify redundant attention computation without disrupting object identity.
How can autoregressive language models add recurrent cross-token memory without breaking KV caching or one-forward-per-token decoding?
Standard transformers rely on attention and cached keys and values during generation, but they do not naturally carry a separate recurrent state across tokens. Adding one can alter the decoding interface or require additional forward passes.
How can autoregressive LLM decoding generate multiple tokens in parallel at large batch sizes without sacrificing quality?
Autoregressive LLMs repeatedly predict one next token at a time, limiting throughput and increasing serving costs as batch sizes grow. Parallelizing generation is difficult because it must retain the quality and distribution of the original autoregressive model.
How can batched entropic optimal transport stop updating converged instances without changing its stopping rule?
In a static batch of independent entropic optimal transport problems, every update continues until the slowest instance meets tolerance. Computation is therefore spent updating instances that have already converged.
How can batched LLM serving handle request-specific read-write adaptation state without corrupting streams or losing throughput?
Standard batched LLM serving assumes shared static weights, but test-time training changes request-owned state during generation. Mixing incompatible reads and writes can transfer state across streams, while serial execution limits throughput.
How can black-box language models reliably follow procedural instructions at inference time for downstream trajectory repair?
Instruction-following failures can leave downstream components without the procedural steps needed to inspect or repair a generated trajectory. Better procedural compliance may not improve final-answer accuracy and can change how early the model commits to an answer.
How can causal streaming video editing remain real-time while preserving backgrounds and unedited regions over long sequences?
Each frame must be edited without access to future frames, while backgrounds and untouched regions remain stable over time. Meeting that consistency requirement without sacrificing interactive responsiveness is difficult.
How can clustering scale per-user LLM recommendations while guaranteeing relevant, attribute-consistent outputs?
Running an LLM separately for millions of recommendation inputs can be prohibitively costly and slow. Reusing an output from a cluster representative can instead produce irrelevant or unsafe recommendations when individual users are poorly matched to that representative.
How can CNN inference fit on bare-metal microcontrollers with severe memory, binary-size, and runtime limits?
Bare-metal microcontrollers may provide only a few kilobytes of SRAM and limited firmware space, while CNN inference also imposes execution-time demands. These constraints can make standard CNN deployment impractical even when the model's accuracy is acceptable.
How can complete transformer layers be removed from LLMs without sacrificing quality or standard serving compatibility?
Removing whole transformer layers can lower decoding cost, but quality losses vary across models and can become severe at aggressive compression levels. The challenge is to reduce model depth without requiring an altered serving architecture.
How can complex neural-network graphs be mapped across heterogeneous SoCs to balance inference latency and throughput?
Complex neural-network graphs expose parallel operators, but executing them across unlike on-chip processors introduces dependencies and communication costs. Choices that improve pipeline throughput can worsen single-inference latency or energy efficiency.
How can continuous diffusion language models reduce denoising steps without sacrificing text-generation quality?
Continuous diffusion language models generate text through repeated denoising steps, which can make inference too slow for practical use. Reducing the number of steps may also degrade the quality of generated language.
How can decode requests be routed in PD-disaggregated MoE serving when expert sets cause unequal latency?
Different decode batches can activate different sets of experts, changing the amount of expert weight loading even when workers have similar request loads. This makes conventional load balancing insufficient for keeping decode latency consistent.
Previous
1 / 9
Next