From Query to Module — A History of LLM Cascade Routing
Strong models are good and expensive; cheap models are affordable and inconsistent. That trade-off isn't new. What keeps changing is who decides which model handles which request, when that decision gets made, and on what basis. Cascade routing is the umbrella term for the technology that automates this decision.
This post walks through three papers that represent the field's leading edge — RouteLLM, the starting point for query-level routing; SWE-Router, built for multi-turn agentic software engineering; and LLMSelector, which frames the problem from the standpoint of compound AI systems and routes at the module level. These three aren't competing with each other so much as they're a lineage, each one pushing the point where the routing decision gets made into a different layer of the stack.
The premise: what cascade routing is actually solving
The simplest cascade tries a cheap model first and only escalates to an expensive one when the result looks shaky. The real question is what counts as "shaky." You can decide that with a rule fixed in advance, with a trained classifier, or by watching signals that only surface once the work is already underway. Where you plant that decision point is the whole design space of routing technology, and each of these three papers sits at a different coordinate in it.
Where query-level routing started: RouteLLM
RouteLLM (2024), a joint effort from UC Berkeley and Anyscale, was one of the first papers to turn cascade routing into a proper learning problem. The premise is straightforward: instead of a human writing a rule for which model handles which request, train something on data to make that call.
The method's core move is using human preference data. The authors trained routers on Chatbot Arena-style preference data, found that this data alone left performance near random, and closed the gap with data augmentation — adding gold labels and LLM-judge labels on top. The resulting routers were validated across several architectures, from similarity-weighted ranking to matrix-factorization classifiers to a fine-tuned BERT classifier.
The numbers are concrete. On MT-Bench, routing only 13.4% of queries to GPT-4 preserved 95% of GPT-4's standalone performance while cutting cost by 3.66x. On MMLU, the router held 92% of GPT-4's performance at a 1.41x cost reduction; on GSM8K, 87% of performance at 1.49x. The router also showed some ability to transfer to new strong/weak model pairs it hadn't seen during training, without retraining.
The paper is upfront about its own limits, though. First, the query distribution in the benchmarks may not match what a production system actually sees, which argues for domain-specific data collection. Second, the design is confined to binary routing between exactly two models — extending it to three or more is left as future work. Third, routers trained on identical data can still show meaningful performance variance on the same benchmark, and the paper doesn't fully explain why.
Extending routing into the execution trajectory: SWE-Router
RouteLLM-style routers share one trait: they look at the prompt alone and make a single decision before anything runs. SWE-Router (2026) argues that this premise breaks down for multi-turn agentic software engineering. Two superficially similar issue descriptions can be a one-line typo fix or a refactor spanning several modules, and prompt text alone can't tell them apart in principle. The paper frames this as an information-theoretic Bayes-error bound — a ceiling on how good a prompt-only router can ever get, no matter how sophisticated it becomes.
SWE-Router's answer is a value-based, temporal routing framework. A cheap weak model runs first for K turns, generating a partial trajectory — failing test output, file contents, stack traces. A value function, a LoRA fine-tune of Qwen2.5-Coder-7B, reads that partial trajectory and predicts whether the weak model will eventually solve the task. If the predicted value clears a cost-adjusted threshold, the weak model keeps going; otherwise the task escalates to the strong model. The paper backs this with a theoretical result: conditioning on the partial trajectory is never worse than prompt-only routing, and strictly better whenever the trajectory carries real information.
On SWE-bench Verified, using deepseek-v3.2 as the weak model, SWE-Router posted a Route-AUC (a normalized cost-to-resolution-rate metric) of 0.780 — 15.3 percentage points above a non-temporal baseline. What's notable is that the weak and strong models end up solving different subsets of problems, producing a synergy effect where SWE-Router's curve beats even the strong-model-only baseline at certain cost points.
The paper is candid about what's left unresolved. When escalation happens, the strong model doesn't inherit anything from the weak model's exploration — it restarts cold from the original prompt. That's a deliberate choice, made to keep the weak model's reasoning from biasing the strong model's judgment, but it also means real work gets thrown away. A heavier concern is safety: the paper states plainly that leveraging partial trajectories this way could be exploited to route around safety-focused frontier models, and flags safety-aware routing as important future work. On top of that, the evaluation covered only four models, leaving generalization an open question, and the K turns of weak-model inference are a cost paid upfront — one that can eat into the benefit when the weak model is genuinely weak.
Moving inside the pipeline: module-level routing and LLMSelector
The third paper, LLMSelector (2025), comes at the problem from a completely different angle. Rather than attaching one model to one request, it asks which model should run each module in a compound AI system — think self-refine's generator-critic-refiner chain, or multi-agent-debate — where the number of viable model-to-module assignments explodes combinatorially as modules and candidate models multiply, making brute-force search a non-starter.
LLMSelector rests on two empirical observations. The first is monotonicity: hold every other module fixed, and improving one module's own performance tends to improve the system's overall performance too. The second is that a model is fairly good at estimating its own performance at the module level. Put those together and you don't need to search the full combinatorial space — you can pick one module at a time, assign it whichever model tests best for that module, and repeat until nothing improves. That greedy optimization gets you to a good-enough assignment.
The work, a collaboration between Microsoft Research and researchers at Stanford, Princeton, and UC Berkeley, appeared at ICML 2025. Tested across six benchmarks — LiveCodeBench, SimpleQA, and FEVER among them — and three compound-system architectures (self-refine, multi-agent-debate, locate-solve) with GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 as candidate models, it delivered accuracy gains of 5% to 70% over using a single model across every module.
The approach's vulnerability sits in its founding assumption. Monotonicity is an empirically observed tendency, not a guaranteed property of every system. Where modules interact in complex ways or feed back into each other, optimizing one module at a time, greedily, can land on an assignment well short of the global optimum. And because the method leans heavily on a model's ability to self-estimate its own module-level performance, that self-diagnosis can get shaky in new domains where there isn't enough validation data to check it against.
The thread running through all three: what, when, and where to route
Line these three papers up and the direction cascade routing has been expanding in becomes clear. RouteLLM treats a single, independent query as the unit of judgment and makes one decision before execution. SWE-Router widens the unit of judgment to the time a task spends running, updating its decision on signals that only appear mid-execution. LLMSelector moves the unit of judgment again, this time onto the system's own structure, assigning a different model to each of several modules inside one request. In other words, routing's evolution reads as progress along two axes: when the decision gets made — splitting into before execution and during execution — and what it's made about, comparing at the query level, the trajectory level, and the module level.
What's still open
A few problems show up, in one form or another, across all three papers.
One: routing beyond the binary case. RouteLLM is confined to routing between exactly two models; SWE-Router's evaluation covered only four; and even LLMSelector's greedy optimization stops guaranteeing quality once the number of modules and candidate models grows large enough. Nobody has a general method yet for avoiding combinatorial blowup while still guaranteeing quality once the candidate pool gets realistically large.
Two: generalizing across distributions. Whether a router trained or validated on a benchmark performs the same way against real production traffic is a separate question the benchmark doesn't answer. As RouteLLM itself points out, a domain shift likely means retraining or revalidating from scratch.
Three: exploration cost colliding with safety. Exploiting mid-execution signals, the way SWE-Router does, means paying for weak-model exploration upfront — and that same exploration carries the risk of being exploited to route around a model that was deliberately built to be safety-conscious. Cost efficiency and safety end up pulling against each other at exactly the same design point.
Four: context lost at the moment of escalation. When a decision flips and the task moves to a stronger model, there's no standard way to decide how much of the prior exploration and progress should carry over. SWE-Router chose to restart from scratch to avoid bias — which also means throwing away everything computed up to that point.
In summary
Cascade routing started as a simple binary choice between a strong model and a cheap one, but where that choice actually gets made has kept expanding. RouteLLM framed the problem as a learned decision made before execution. SWE-Router pulled in trajectory signals that only surface during execution. LLMSelector moved the unit of decision itself, from a single query to the several modules that make up a system. Read in sequence, the three papers form a lineage that has, one step at a time, widened when a routing decision can be made, what evidence it can draw on, and at what granularity it operates.
At the same time, a set of problems keeps reappearing across all three, in slightly different shapes: extending beyond two-way routing, the gap between benchmark and production distributions, balancing exploration cost against safety, and carrying context across an escalation. Bringing cascade routing into a real system, in the end, isn't a matter of bolting on whichever routing algorithm a paper validated — it's a design problem of figuring out, for your own system's structure, which unit and which moment of decision actually fits.