Get Started
Home
Topics
Search
Library
Research questionHow can residual sparsification reduce MoE LLM memory use without accuracy loss from coupled expert projections?Residual sparsification represents each expert projection using a shared base and a per-expert residual. Compressing residuals independently can overlook interactions among projections and hidden representations, allowing small local errors to become larger expert-output errors that reduce model accuracy.
AI
Inference Optimization
Machine Learning
Research Paper
Technology
Latest papersRecent research connected to this question, newest first.Residual Sparsification via Output Importance for Compressing Mixture-of-Experts LLMsApplies to residual sparsification for MoE LLMs. The supplied evidence covers experiments on Qwen and DeepSeek at the same peak memory reduction; it does not establish results for other architectures or deployment settings.research paper · Sep 2, 2026
Related questions
How can inference prune activated MoE experts without confounding compute savings with output rescaling?When can factual recall in sparse MoE language models be attributed to one expert rather than an expert set?How should experts be pruned in over-dispersed MoE routing when router importance and perplexity mislead?How does sparse mixture-of-experts routing trade off approximation, learning error, and computation under misspecification and evolving experts?