Get Started
Research questionHow can decode requests be routed in PD-disaggregated MoE serving when expert sets cause unequal latency?Different decode batches can activate different sets of experts, changing the amount of expert weight loading even when workers have similar request loads. This makes conventional load balancing insufficient for keeping decode latency consistent.
AI
Inference Optimization
Machine Learning
Latest papersRecent research connected to this question, newest first.ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE ServingThe source concerns PD-disaggregated LLM serving and request assignment after prefill, including expert-activation signatures and prefix-cached KV blocks. Evidence comes from a vLLM implementation evaluated on up to 40 GPUs across three MoE models and two workloads; it reports unchanged outputs and median time-per-output-token reductions against four load-balancing baselines.research paper · Jul 1, 2026
Related questions
How should experts be pruned in over-dispersed MoE routing when router importance and perplexity mislead?How can inference prune activated MoE experts without confounding compute savings with output rescaling?How does sparse mixture-of-experts routing trade off approximation, learning error, and computation under misspecification and evolving experts?How should dependent LLM inference tasks be coordinated across edge servers when deadlines allow only limited extensions?
Home
Topics
Search
Library