Get Started
Home
Topics
Search
Library
Research questionHow should experts be pruned in over-dispersed MoE routing when router importance and perplexity mislead?Aggressive load balancing can distribute tokens nearly uniformly across experts, causing router-based importance scores to collapse. In this regime, perplexity may disagree with downstream capability, and different pruning scores can preserve one domain while damaging another.
AI
Evaluation & Benchmarks
Inference Optimization
Machine Learning
Natural Language Processing
Reasoning
Latest papersRecent research connected to this question, newest first.When Load-Balancing Goes Too Far: Expert Pruning in Over-Dispersed Mixture-of-Experts ModelsThe evidence covers over-dispersed and standard routing, with detailed results on gpt-oss-20B and additional results on gpt-oss-120B, Gemma-4-26B-A4B, and OLMoE-1B-7B. It evaluates 25% expert pruning across 11 benchmarks, including mathematical reasoning and knowledge-intensive science, and compares activation-aware, frequency-based, and domain-aware scoring.research paper · Sep 3, 2026
Related questions
How does sparse mixture-of-experts routing trade off approximation, learning error, and computation under misspecification and evolving experts?How can inference prune activated MoE experts without confounding compute savings with output rescaling?When can factual recall in sparse MoE language models be attributed to one expert rather than an expert set?How can we tell whether cross-layer routing predictability reflects shared routing dynamics or generic hidden-state smoothness?