LLMRouter recasts every existing LLM router (binary, cascade, graph-based, agentic, personalized) as one sequential decision process with five swappable parts, then benchmarks 16+ of them on one protocol. Learned routers beat always-pick-the-biggest-model by 14.6% relative because the largest model gets many queries wrong that cheaper ones handle.
You’re running a chat product that fans out to five or six LLM backends. Some queries need DeepSeek-V3.1-class reasoning, most don’t, and your bill reflects the ones that don’t. So you added a router. The problem: every published router (RouteLLM, FrugalGPT, GraphRouter, Router-R1, and so on) ships as its own codebase with its own training data, its own candidate pool, and its own evaluation. You cannot tell whether one method actually beats another or whether the numbers reflect a nicer benchmark setup. Prior routing benchmarks like RouterBench precompute one fixed pool for single-turn text, which leaves multi-turn, personalized, vision, and long-context routing without a shared yardstick.
The authors’ claim is that every router, no matter how it looks on the surface, is doing the same thing: at each step, look at the state (query, optional user context, history so far), pick a model from the pool or stop, and pay for the call. That reframing gives them five slots any router must fill:
•
a context encoder that turns the current state into a vector or text,
•
a model encoder that describes each candidate LLM (metadata, past performance, learned embedding, or just its name in a prompt),
•
a scoring function that says how well a candidate matches the state,
•
a decision rule that picks one (argmax, threshold, cascade escalation, sample, or stop),
•
a learning signal (pointwise labels, pairwise preferences, contrastive, or reinforcement learning reward).
From that, families fall out naturally. Single-turn routers see only the query. Multi-turn and agentic routers also see accumulated responses and can decide to keep dispatching. Personalized routers add a user identifier and past interactions. Their objective is one weighted expression: expected quality minus λ times cost of the whole trajectory, where λ tunes how cost-sensitive you are.
The library operationalizes this. To add a new router, you subclass one interface:
class MyRouter(MetaRouter):
def route_single(self, query):
s = self.encode_state(query) # E_q
scores = self.score(s, self.models) # g over E_m
query["model_name"] = self.decide(scores) # d
return query
class MyRouterTrainer(BaseTrainer):
def loss_func(self, outputs, batch):
return my_objective(outputs, batch) # L
Data construction, training loop, evaluation sweep, and OpenAI-compatible serving are shared infrastructure. A companion data engine takes a list of tasks and 18 candidate models, runs everyone on everything, scores with task-native metrics, and prices by tokens, producing a dense query-by-model matrix that doubles as training supervision and test bed.
The prevailing story in routing papers is that each family (cascades, contrastive scorers, graph routers, RL agents) has its own inductive bias that determines when it wins. This paper shows the opposite. Once you write every router as (state, encoders, score, decision, loss), the differences are configuration, not architecture, and no single configuration wins across tasks or cost budgets. The evidence for this is not the headline average, but the rank-reversal plot: routers that top one track drop near the bottom when λ shifts or the task changes.
The load-bearing finding is that router rankings scramble as you sweep the cost weight. On generic tasks, RouterDC leads when only quality counts and falls to tenth of eleven under the most cost-sensitive setting. EloRouter leads on vision and time-series at zero cost weight and loses that lead once cost enters. MLPRouter sits near the bottom of vision on quality-first, then becomes the best vision router for every cost weight ≥ 0.4. So “which router is best” is not answerable without naming an operating point.
Secondary findings that hang off that:
•
Learned routers beat the always-largest baseline by 14.6% relative on average, because the largest model is expensive yet still wrong on many queries that a smaller one handles.
•
Multi-turn routing did not consistently beat single-turn. Router-R1 and the multi-round baselines averaged 22–23 against roughly 40+ for good single-turn routers on xRouteBench. Extra rounds add cost and depend on the small base model (Qwen2.5-3B) that decomposes and aggregates.
•
Personalization helps, but the sim-to-real transfer is fragile. GMTRouter wins under the persona-conditioned LLM judge at 68.78% accuracy. On 234 real Slack preference records from 15 users, PersonalizedRouter wins at 83.05% and GMTRouter drops to sixth. The simulated judge and real users prefer different personalized designs.
•
Routing per node inside a multi-agent system pays off. Six of seven learned routers beat always-use-largest across five coordination topologies, with the best averaging 76.48 against 71.48.
Reach for this when you already run more than two LLM backends and want to A/B a routing policy without rebuilding your training pipeline for each candidate method. The workflow is: point the data engine at your task list and your model endpoints with prices; it produces a query-by-model matrix of quality and cost; every router in the library trains and evaluates on that matrix under the same weighted objective; you sweep the cost weight to trace a frontier and pick the operating point. Adding your own router is subclassing MetaRouter and writing a loss function.
The artifacts: code on GitHub, the xRouteBench dataset covering generic text, long-context memory, image and video, time-series, and personalized dialogue (4,767 test queries), and a project page. The library exposes any trained router as an OpenAI-compatible server and ships a Node-based workflow tools (e.g., ComfyUI) canvas for wiring the pipeline visually. Eighteen candidate models are pre-priced, from Gemma-2-9B at $0.10/1M input tokens up to Cogito-v2-671B at $1.25.
Pick your router after you pick your cost budget, not before. The paper’s main empirical result is that rankings flip when the cost weight moves, so the honest deployment question is not “which router is best” but “which router is best at the λ my finance team actually cares about,” and this library exists so you can answer that with a config sweep instead of a rewrite.
•
The 14.6% headline is a relative lift over always-picking-the-largest model, which the paper itself shows is a weak baseline. Against a smart fixed-model policy (say, always-Llama-3.3-70B for text and always-DeepSeek for math), the gap would likely be smaller. The paper does not report that comparison.
•
The persona-judge personalized track uses DeepSeek-V3.1 conditioned on a PersonaHub persona to score answers. The real-user Slack study, with only 15 users and 234 pairs, contradicts the judge’s ranking. Personalization gains at scale are not yet demonstrated.
•
Vision and time-series tracks convert images and series to text via a fixed captioner before routing, so the router never sees pixels. Results say which LLM handles a description best, not which multimodal model handles the raw input best.