> ../patterns/routing_table.md
§ 01 · Why a routing table exists
The cheapest week in agentic history landed three frontier models in seven days, and the price of running high volume agentic work fell sharply. The temptation is to swap the whole agent to the cheapest model that clears a benchmark and take the savings. That temptation is a trap, and we have written the arithmetic behind it in our July postmortem on the cheaper model: a model that is seventy percent cheaper per token can raise the bill, because price per token is the smallest term in the cost of an agent. Turns, retries, and human overturns dominate.
The routing table is how you capture the price collapse without shipping a worse agent. Instead of one model behind the whole system, you keep a small ordered list of models, each with a defined job, and route every task to the cheapest model that can do it. The strong model is not deprecated. It is reserved.
── The table is the decision ──
§ 02 · Classify the work before you route it
Routing only pays if the tasks are not uniform, and in a real agent they never are. Before you build the table, split your traffic into tiers. A useful cut has three: the easy majority, where the task is short, the schema is tight, the failure cost is low, and the answer is either right or obviously wrong. The judgment middle, where the task requires weighing context, the schema is loose, and the failure is not obvious to a reviewer skimming the queue. The hard tail, where the task is long, the context is large, an error is expensive to undo, and a human is going to look at the output either way.
The point of this split is not elegance. It is that the same model is overqualified for the first tier and underqualified for the third, and pricing them the same way is what makes cheap swaps expensive.
── Four fields ──
§ 03 · The routing table, four fields
A routing table has four fields per row: the tier, the primary model, the fallback model, and the decision rule that selects the row.
◆ Tier. The class of work, named the same way the team names it in conversation. Not a model capability. A job.
◆ Primary model. The cheapest model that passes the eval set for that tier, with a pinned version, never a floating alias. A floating alias is how a silent model regression enters production without a deploy.
◆ Fallback model. A stronger, more expensive model that runs when the primary fails a defined check. The fallback is not a hope. It is a route.
◆ Decision rule. The predicate that puts a task in this row. Deterministic where possible: a category, a size, a customer tier, a schema. A classifier where necessary, with its own eval and its own confidence threshold.
Keep the table short. Three rows covers most agents. Ten rows is a signal that the classifier is doing work that belongs in the architecture.
── Routing is a decision ──
§ 04 · A route is a decision, so log it
Every routed task must record which row picked it, which model ran it, and which model the fallback would have used. Without this record you cannot answer the only question routing eventually raises: is the cheap tier still doing the job. The instrumentation pattern is where this record lives. The route is part of the decision trace, not a debug print, and the share of traffic per row is a behavioral metric with a band around it.
The moment a row's overturn rate rises, or its fallback rate spikes, the table is telling you something. Usually it is telling you that the tier was drawn wrong, or that the primary model drifted underneath you, or that the input distribution has moved. All three are recoverable. None of them are visible without the log.
── When a route fails ──
§ 05 · The fallback path
The fallback is the reason routing is safe. A primary model runs, and the output is checked against a fast, deterministic gate: schema valid, required fields present, confidence above the threshold, no hard refusal, no obvious hallucination pattern. If the gate fails, the same input is sent to the fallback model, once, and the fallback's output is used. If the fallback also fails, the task escalates to a human queue.
This is not a retry loop. It is a single, bounded promotion, and it borrows its discipline from the retry and backoff pattern: one attempt at the primary, one attempt at the fallback, then a person. Anything more is a way to spend real money to arrive at the same answer more slowly.
The fallback rate is the health check for the whole table. A stable low rate means the tiers are drawn right and the primary is doing its job. A rising fallback rate means one of three things is true, and you find out which by reading the traces of the promoted tasks.
── Four ways this goes wrong ──
§ 06 · Four failure modes
◆ The tiers are drawn wrong. The easy majority contains a class of task that looks easy and is not. The overturn rate on that row is quietly bad, and the aggregate looks fine because the row is small. The fix is not a bigger model. It is a new row.
◆ The classifier is the bottleneck. A weak classifier sends judgment middle work to the easy tier. Every downstream saving is undone by the misroute. Evaluate the classifier separately, with its own set, and treat its accuracy as a first class metric on the routing dashboard.
◆ Route drift. The primary model, or the input distribution, moves. The row is still cheap. It is no longer correct. This is the failure the eval set note was written for: run the eval on a schedule against the pinned primary of every row, and treat a score move as an alarm.
◆ Fallback as a crutch. The fallback rate creeps up, and nobody looks. The table is now a slow, expensive way to run the strong model on a large share of traffic. Set a hard band on the fallback rate per row, and treat a breach the same way you treat an error budget: the row is out of policy until the primary is fixed or the tier is redrawn.
── Checklist ──
§ 07 · Before you ship a routing table
[ ] Traffic split into named tiers, drawn from how the team already talks about the work, not from model capability sheets.
[ ] One row per tier, four fields: tier, primary model with a pinned version, fallback model, decision rule.
[ ] Every routed task logs the row, the primary, the fallback, and whether the fallback ran.
[ ] A fast deterministic gate defines what counts as primary failure, and the fallback runs exactly once before a human.
[ ] Overturn rate, fallback rate, and share of traffic per row are on the same dashboard, with bands.
[ ] Eval set runs on a schedule against every pinned primary, and a score move is an alarm.
[ ] The ROUTING.md records the tiers, the rules, the models, the pinned versions, the gate, the bands, and the owner.
── End of pattern ──
◆ Price per token is the smallest term. Cost per successful task, which includes turns, retries, and human overturns, is what the routing table optimizes for.
◆ A row per tier, a pinned primary, a defined fallback, a deterministic decision rule. Short tables are healthier than clever ones.
◆ The route is a decision. Log it, band the rates, and let the fallback rate be the health check for the whole table.
ORBIRESEARCH