Model routing: sending easy questions to cheap models
Routing is the highest-leverage cost control in an LLM system, and it is almost always implemented last, after eighteen months of paying premium prices for questions a smaller model answers identically.
What happened
A team runs everything on the strongest available model because it was the safe choice at the start and nobody revisited it. Eighteen months later the bill is the largest line item in the project, and an analysis shows that a majority of queries are lookups, rewrites and classifications that a model an order of magnitude smaller handles identically.
Why it matters for deployment teams
Uniform model selection is a tax on caution. The cost difference between tiers is often ten to thirty times, while the quality difference on simple tasks is frequently indistinguishable. Routing recovers most of that spread without touching the hard cases, which is why it beats every other cost lever on return per hour of work.
How to build it
Do not start with a router model — that adds latency and a failure mode. Start with rules, because you already know the categories: short factual lookups, classification and extraction go to the small model; anything multi-step, anything requiring synthesis across documents, and anything where the small model's confidence is low goes to the large one. Then measure. Route a sample of live traffic to both tiers, score both with your eval set, and promote a category to the small model only where the score holds.
Guardrails
Keep a floor: never route a request the small model refuses twice. Log which tier served every request, so a quality complaint can be traced to a routing decision rather than guessed at. And re-run the comparison whenever either model is upgraded — a routing table built against last quarter's models is a routing table describing models you no longer use.