Get Started
Home
Topics
Search
Library
Research questionHow can on-device LLM inference overlap weight I/O and computation under tight DRAM without stale sparsity decisions?Activation-sparsity decisions become more accurate with later context, while effective I/O-computation overlap requires decisions early. This tension can serialize execution or cause redundant weight transfers and excessive cache use.
AI
Inference Optimization
Small / On-device Models
Technology
Latest papersRecent research connected to this question, newest first.LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM InferenceThe source describes systems that offload weights to SSD or flash storage and progressively refine computation, loading, and cache-retention priorities using partial GPU results. It reports implementations on mobile and embedded platforms, with improvements in memory usage and token-generation throughput relative to prior on-device inference systems.research paper · Sep 2, 2026
Related questions
How can long-context LLM inference reduce KV-cache memory without losing attention-head-specific information?How can we robustly compress LLM KV caches across open-domain inputs without input-specific budget thresholds?How should dependent LLM inference tasks be coordinated across edge servers when deadlines allow only limited extensions?How can large language models cut training and inference costs without materially harming accuracy?