> ../failures/cheaper_model_higher_bill.md
§ 01 · What happened
A document processing agent read inbound supplier invoices, extracted the line items, matched them against purchase orders, and either posted the match or escalated a discrepancy to a human. It ran on a strong frontier model. It worked. It was also, per the finance review, the single most expensive line in the team's model spend.
A cheaper model landed. On the published price sheet it was around seventy percent below what the team was paying. The swap was one line of config. The eval suite passed. The team shipped it on a Monday and told finance the model bill would drop by roughly two thirds.
Three weeks later the monthly bill was higher than before the swap. Not lower by less than expected. Higher. And the rate at which humans overturned the agent's decisions in the approval queue had roughly doubled.
── Where the money went ──
§ 02 · Where the money actually went
Nobody was overcharged. The provider billed exactly the advertised rate. The tokens were cheap. There were simply far more of them, and they came from four places nobody modeled.
More turns per task. The frontier model typically read the invoice, called the purchase order lookup once, and produced the match. The cheaper model was less reliable at pulling the right identifier on the first pass, so it looked it up, got an unhelpful result, reasoned about it, and looked again. A task that took three tool calls now took six or seven. Every turn re-sends the accumulated context, so turn count multiplies token spend rather than adding to it.
More retries. Malformed output that failed schema validation triggered a retry. On the old model this was rare. On the new one it was common enough to matter, and each retry paid for the whole run again.
Longer outputs. The cheaper model was more verbose in its reasoning. More reasoning tokens per turn, multiplied by more turns.
More escalations, and this is the expensive one. Escalations doubled. Each escalation is a human reading an approval item, which costs a salaried minute, and every overturn also meant the agent's work for that task was thrown away and paid for anyway. The model bill was the visible cost. The human bill was the larger one, and it does not appear on the provider's invoice at all.
── The arithmetic ──
§ 03 · The arithmetic nobody did
The swap was justified on price per million tokens. That is the wrong unit. The unit that matters is cost per successfully completed task, and it has four terms, not one.
Cost per successful task equals: the model cost of one attempt, multiplied by the average number of attempts a task takes, divided by the share of tasks that succeed without human intervention, plus the loaded human cost of every task that escalates.
Run the swap through that. The per attempt model cost dropped by about seventy percent. The average number of attempts rose. The share of tasks completing without human help fell. And the escalation term, the one denominated in human minutes rather than tokens, roughly doubled. A large discount on the smallest term in the equation, paid for with increases in every other term.
This is why a model that is sixty percent cheaper and materially less reliable is not sixty percent cheaper. It is cheaper failure, and failure is the expensive part.
── Why the eval suite said yes ──
§ 04 · Why the eval suite let it through
The eval suite passed, and this is the part that should worry anyone about to make the same swap this month.
The suite was scored on final answer accuracy against a set of cases. The new model got the answers right often enough to clear the pass threshold. What the suite did not measure was the cost of getting there: how many turns, how many retries, how many escalations along the way. A model that arrives at the right answer after six tool calls and one retry looks identical, on a pass or fail eval, to a model that arrives in two calls cleanly. Those two models have very different bills and very different operational load.
The eval suite was measuring capability. Production pays for efficiency and reliability. That gap is the same one the pass@1 note argued about accuracy, appearing again on the cost axis: a benchmark measures whether the model can, and the invoice measures what it took.
── The fix ──
§ 05 · The fix, in three parts
Part one: bill against cost per successful task, not price per token. Instrument it. For every task the agent completes, record model cost, turn count, retry count, and whether a human had to intervene. Cost per successful task is then a number on a dashboard, not an argument in a meeting. Any model change is judged on that number.
Part two: make the eval suite score the path, not just the destination. Every eval case now records turns to completion, retries triggered, and whether the run would have escalated, alongside the correctness score. A candidate model that reaches the same accuracy in twice the turns fails the eval, because it will cost more in production even at a lower headline rate.
Part three: route by task class instead of standardizing on one model. The swap was framed as a binary, expensive model or cheap model, for the whole agent. That was the wrong frame. The invoice matching work has an easy majority and a hard tail. The cheap model handles the clean, unambiguous invoices well and cheaply. The hard tail, the ones with partial matches and ambiguous identifiers, is where it burns turns and escalates. Route the easy cases to the cheap model, keep the hard tail on the strong one, and the blended cost per successful task lands below both single model configurations. The right question was never which model. It was which model for which case.
── The control that runs now ──
§ 06 · The control that runs now
Cost per successful task is on the same dashboard as the behavioral metrics from the instrumentation pattern, with a baseline and an alert band. Any model change, including a provider rolling a version underneath us, moves that number, and moving it is an event. The team no longer approves a model swap on a price sheet. They approve it on a dashboard, three weeks of data, and a routing table.
── End of report ──
◆ Price per token is the smallest term in the cost of an agent. Cost per successful task is the number that pays the bill: attempts, turns, retries, and the human minutes spent cleaning up.
◆ An eval suite that scores only the final answer cannot see a model that gets there in twice the turns. Score the path, not just the destination.
◆ The question is never which model. It is which model for which class of case. Route the easy majority cheap, keep the hard tail strong, and the blended cost beats either model alone.
ORBIRESEARCH