ELDR routes decode requests in a disaggregated Mixture of Experts serving system to workers whose other in-flight requests activate overlapping experts, cutting median per-token latency by 5.9–13.9% without changing model outputs.
You’re running an inference service for a large Mixture of Experts model. To keep latency down, you’ve split prompt processing (Prefill phase) and token generation (Decode phase) onto separate GPU pools, then use a router to send each request to a decode worker. Your router picks the worker with the shortest queue, because that’s what everyone does. That’s fine for dense models: equal queue depth means equal work. For Mixture of Experts models it isn’t. Each decode step has to load, from GPU memory, every distinct expert weight that any request in the batch touches. So a worker holding four requests that all lean on the same code experts finishes a step much faster than a worker holding four requests spanning code, medical, math, and legal, even though both queues look identical to a load balancer. The dominant baselines here (Join-Shortest-Queue (JSQ), Power-of-Two-Choices (P2C), round-robin) are all blind to this. ELDR is the first decode router that isn’t.
The key observation: the Mixture of Experts gating network runs during Prefill phase too, and which experts a request activates during Prefill phase correlates strongly (0.70 to 0.92 across three models) with which experts it will activate during Decode phase. So Prefill phase hands you a preview of the request’s expert footprint right at the moment you have to pick a decode worker.
ELDR converts that preview into a compact vector called an expert signature. Naive counts of which experts fired don’t work well, because a few generalist experts fire on almost every request and drown out the specialists that actually distinguish workloads. So ELDR reweights per-expert counts by Inverse Document Frequency (IDF), keeps only the layers that best predict decode-time overlap, and L2-normalizes. Signature quality is judged by Spearman rank correlation between signature distance and true decode-time expert-usage distance, so the representation is picked independently of the router built on top.
Routing then has two objectives that pull apart: pack expert-similar requests together (locality) versus keep worker queues even (load). ELDR splits these across offline and online stages. Offline, Hungarian-balanced K-means partitions signature space into one region per decode worker, forcing equal-size clusters so no worker becomes a hot spot. Online, for each incoming request:
sig = sum(signature_cache[b] for b in kv_blocks(request))
sims = [cosine(sig, c) for c in centroids] # one per worker
s_star = max(sims)
band = [w for w, s in enumerate(sims) if s >= s_star - tau]
return min(band, key=lambda w: in_flight[w]) # least-loaded in band
The band width tau (0.1 in the paper) lets load balancing kick in only when several workers are near-tied on locality. One last wrinkle: Prefix caching skips Prefill phase for shared prompt prefixes, so the gate never runs on those tokens and the signature would be incomplete. ELDR sidesteps this by storing signatures at KV cache block granularity, co-indexed with the KV cache itself. Summing over a request’s blocks reconstructs the exact signature whether the prefix was freshly computed or cached.
The prevailing assumption in decode routing is that workers are interchangeable, so balancing queue length is enough. This paper shows the opposite for Mixture of Experts models. Which requests share a worker is a first-order latency knob, because decode cost is set by the union of experts the batch activates, not by the token count. The load-view that dense-model routers rely on is literally invisible to this bottleneck. The load-only baselines the paper compares against reduce median per-token latency by essentially nothing versus each other; adding locality is what moves the number.
The load-bearing finding is that same-domain batches activate 17–21% fewer distinct experts per decode step than mixed batches on task workloads, and that Prefill phase activations correlate with Decode phase activations at 0.70–0.92 across three models. Together these say the mechanism is real (locality shrinks the expert union) and exploitable at the right moment (the Prefill phase-to-Decode phase handoff).
On top of that mechanism, on an 8-prefiller / 16-decoder deployment across Qwen3-30B-A3B, GPT-OSS-120B, and Gemma-4-26B-A4B:
•
Median per-token latency drops 7.0–13.9% on task workloads and 5.9–10.0% on language workloads versus the best of four load-balancing baselines.
•
A naive “route by oracle domain label” baseline helps on tasks (four clean domains) but collapses on language, where several signature clusters live inside a single language. ELDR’s 16 signature clusters see substructure the labels miss.
•
Ablations confirm the design: discrete top-k counts beat continuous gate probabilities, Inverse Document Frequency (IDF) reweighting adds up to +14 percentage points on the worst cell, and Hungarian-balanced K-means is essential (vanilla K-means regresses tail latency by up to 17.4% because clusters go lopsided).
•
Runtime overhead is 0.86 ms per request (1.2% of median time-to-first-token); the signature cache is under 1% of KV cache size. Outputs are bit-identical to standard top-k gating, since ELDR only changes which worker serves a request.
Reach for this when you’re operating a Mixture of Experts serving stack with Prefill phase-Decode phase disaggregation and your decode-side latency is memory-bandwidth-bound. If your traffic has any domain structure (a code assistant plus a general chat endpoint sharing hardware, or multilingual traffic with a heavy head), a locality-aware router should give you single-digit to low-double-digit percent latency wins with no accuracy cost and no model changes. If your traffic is uniformly mixed with no domain structure, the mechanism has nothing to exploit and you’ll get load-balancer-equivalent behavior.
The paper says ELDR is implemented as roughly 2,000 lines of Python on top of vLLM, with the design portable to SGLang. The paper does not link a public code repository. Recalibration requires a 1,000-prompt trace through Prefill phase (4–15 minutes) plus a sub-10-second offline fit, and needs to rerun only when the model, workload mix, or Prefill phase-Decode phase topology changes.
In Mixture of Experts serving, who a request sits next to matters more than how long its queue is. Load balancing was the right default for dense models because every token paid the same weight-loading cost. Sparse experts break that symmetry, and the fix isn’t a smarter queue policy, it’s routing on a signal (expert activation) that the queue can’t see.
•
The wins depend on workload structure. On the language workload for two of three models, tail latency was flat or slightly worse versus the best load balancer. If your traffic has no domain clustering, there’s nothing to exploit.
•
Everything is evaluated on AMD MI300X GPUs with the largest deployment at 40 GPUs. Behavior at hyperscaler-sized deployments, on different interconnect topologies, or with different expert-parallelism layouts isn’t shown. The 235B run does confirm the mechanism holds under expert parallelism, but with smaller gains (2.7–4.3% median).
•
The offline calibration assumes workload composition is stable enough that a fit stays valid. Drift handling is mentioned as “just re-fit,” but the paper uses a single offline fit throughout and doesn’t measure how fast quality degrades as traffic mix shifts.