IntBMoE decouples three costs in Mixture-of-Experts that prior designs entangle: how many experts influence a token (participation), how many run (execution), and how many parameter sets get built (materialization). It does this by having a small learned codebook of blocks, each composed from the entire expert pool via a Hypernetwork, while each token only routes to a few blocks.
If you serve a large MoE model, you’re stuck picking a bad trade. Standard sparse routing (Top-k over experts, like Switch Transformer or DeepSeekMoE) is cheap to run, but each token only ever sees knowledge from the 1-2 experts picked for it. The other experts are dead weight for that token, and get no gradient from it either. Dense output-mixing (run every expert, weight-combine outputs, as in MMoE) gives every expert a say, but compute scales linearly with pool size. Parameter-merging methods like SMEAR and Lory first blend all expert weights into one composed expert per token (or per segment), then run that once. Participation is full and execution is cheap, but you now have to construct a fresh expert-sized parameter set for every routing decision, which is a memory and compute tax the paper calls materialization.
The authors’ framing: participation, execution, and materialization are logically independent, but every existing MoE variant couples at least two of them. If you want every expert to contribute knowledge to every token while keeping both compute and parameter-construction bounded, none of the standard designs give you that.
The key move: separate building experts from running them on tokens.
Each IntBMoE module keeps a codebook of K learned embeddings (default K=8). Each embedding names one reusable multi-layer block. A shared Hypernetwork reads a codebook embedding and emits two vectors of E coefficients (one for a “value” path, one for a “gate” path). These coefficients linearly combine all E expert-basis matrices in that layer’s pool into a single composed expert. Because the coefficients aren’t softmax-normalized, they can be negative and don’t sum to one, so the composition spans the full linear space of the basis rather than being stuck in their convex hull. A 1/sqrt(E) scale keeps magnitudes stable as E grows.
Since the hypernetwork only sees codebook embeddings, not tokens, the K composed blocks are token-independent. Build them once, cache them, reuse. At inference this removes the composition cost entirely from the request path.
At token time, a router scores the K blocks for token x_t and picks the top k (default k=2). Before entering a block, the token is passed through a sigmoid mask conditioned on that block’s codebook embedding, so different blocks see different views of the same token.
Inside each block, the value and gate paths (each independently composed from the same expert pool) are combined with a residual multiplicative rule: output = value * (1 + lambda * SiLU(gate)). The authors call this Dual-Path Residual Gating (Dual-Path Residual Gating). Because the two paths multiply, the block is nonlinear in the expert bases even though each path alone is a linear combination. lambda is learned.
# once per model (cacheable):
for b in range(K):
alpha_v, alpha_g = hypernet(codebook[b]) # shape (E,), (E,)
for l in range(L):
W_v[b,l] = sum(alpha_v[e] * W_base[l,e] for e in range(E)) / sqrt(E)
W_g[b,l] = sum(alpha_g[e] * W_base[l,e] for e in range(E)) / sqrt(E)
# per token:
scores = block_router(x_t)
for b in topk(scores, k):
z = x_t * sigmoid(W_f @ concat(x_t, codebook[b])) # feature filter
for l in range(L):
v = W_v[b,l] @ z; g = rmsnorm(W_g[b,l] @ z)
z = v * (1 + lambda_ * silu(g))
out += softmax(scores)[b] * z
return out + shared_swiglu(x_t)
An always-on shared SwiGLU expert runs in parallel to absorb the generic transformation, letting the routed blocks specialize.
The primary benchmark is ImageNet-1K with an 8-layer DeiT-Tiny-style backbone trained from scratch. All MoE baselines are matched to roughly 24M total parameters.
•
IntBMoE reaches 73.76% top-1 vs 66.40% for the dense backbone and 71.78% for the strongest MoE baseline (SMEAR), a +1.98 pp margin over that baseline.
•
Compute is 4.063 GFLOPs per image uncached, 3.457 GFLOPs with block-caching. That’s higher than the cheapest sparse baselines (~1.5 GFLOPs) but far below the dense-participation family that has to touch every expert per token in a different way.
•
Ablations: removing the composed gate path (leaving only the value path, i.e. a linear composition) is the single largest hit (top-1 drops from 73.76 to 68.04). Collapsing each 2-layer block to a parameter-matched 1-layer block is the next biggest (to 68.78). Softmax-normalizing composition coefficients also hurts (to 72.63), supporting the design choice to allow negative, unnormalized coefficients.
•
Full-pool participation is not just architectural. When they zero out individual expert bases post-hoc, every removal in layers 0/2/4/6 reduces accuracy (minimum drop 0.26 pp), and layer 0 has a few bases whose removal costs up to 9.15 pp. So the pool is actually being used, though unevenly in early layers.
•
Caching behavior: as pool size E grows from 1 to 128, uncached memory grows from 55 MB to 628 MB, while cached memory stays flat at 104 MB. Caching pays off from E >= 16.
•
Generalization: on MiniPile language modeling, IntBMoE hits PPL 14.59 vs 15.03 for the best baseline (muMoE CP), a 2.9% reduction. On IntTravel sequential POI recommendation it’s top on HR@1/HR@5/NDCG@5, though margins over the strongest baselines are narrow (HR@1 0.6852 vs ~0.6837).
•
Production: deployed in AMap’s generative POI recommender, cached IntBMoE runs at 19 ms average / 38 ms P99 on Alibaba T-Head PPUs under a 60 ms budget, and a one-week A/B test at ~5000 QPS showed a 2.4% relative UVCTR lift.
One thing to keep straight: the FLOPs table is measured at batch size 1. The compute advantage of caching gets larger as batch size grows because composition is amortized across the batch, but the paper only quantifies the single-image case.
•
If you’re building an MoE where you want the flexibility of a large expert pool but can’t tolerate either the compute of dense mixing or the per-token parameter synthesis of SMEAR-style merging, the IntBMoE recipe (fixed codebook of blocks, hypernetwork composition, sparse block routing, cache the composed blocks at serving time) is worth trying. The cache is the load-bearing efficiency trick, not the architecture per se.
•
If you already run sparse Top-k MoE and are considering scaling the expert count, the ablation showing every basis contributes something suggests IntBMoE’s composition style may use added experts more thoroughly than pure Top-k routing does. Worth testing whether you actually need E to be large: their sensitivity study shows going from E=16 to E=64 gains only 0.15 pp on ImageNet, so the default may already be near saturation for that task.
•
Dual-Path Residual Gating is essentially free (a second linear composition of the same pool) and gave the largest ablation delta. If you’re building any composed-expert architecture, having two independently composed paths that multiply is a cheap expressiveness gain worth borrowing on its own.
•
The GitHub repo is released, which matters because the deployment story (cached blocks, PPU serving path) is a nontrivial engineering artifact that the paper describes only at a high level.
•
The visual gains are on an 8-layer DeiT-Tiny trained from scratch, not on a large pretrained model. Whether the same composition mechanism helps when dropped into an already-trained large backbone is not tested.
•
On the language and recommendation tasks, IntBMoE wins but by small margins (PPL 14.59 vs 15.03 best baseline; HR@1 deltas in the third decimal). The strong ImageNet result is the outlier, not the norm across the three settings.
•
The 2.4% UVCTR lift is against “the existing production model without an MoE module,” not against an alternative MoE. So the A/B result tells you MoE-with-IntBMoE beats no-MoE in that stack, not that IntBMoE would beat, say, a cached Top-k sparse MoE in the same slot.
•
Materialization is bounded by K, but you still pay O(K * E * D) to build the block cache once, and cached memory grows with K and block size. This is fine at K=8; the paper doesn’t explore what happens if you want hundreds of blocks.
•
The load-balancing question (do all K blocks get used, or do a few dominate) is addressed via the class-conditioned routing visualization but not with a hard utilization metric or an auxiliary balance loss discussion. If you scale K up, expect to have to think about this yourself.