> ../research/eval_set_as_spec.md
§ 01 · Why this note, this week
Three frontier models shipped in seven days. One of them is cheap enough that someone on your team has already opened a pull request to switch. Meanwhile a provider can roll a floating alias underneath a running agent with no deploy and no notice, as one team discovered when their triage agent quietly got worse over a weekend.
Both of those are the same event: the most important dependency in your system changed. The only thing that stands between that change and your production behavior is an eval set. Not a benchmark. A benchmark tells you how a model performs on someone else's tasks. An eval set tells you whether it still does yours.
This is why we say the eval set is the spec. Your prompt is an instruction. Your architecture is a set of boundaries. But the answer to "what does correct behavior mean for this agent" exists nowhere in your codebase except in the cases you decided to test. If those cases are thin, your definition of correct is thin, and every model change is a coin flip you are calling heads.
§ 02 · What an eval set is not
Three things get mistaken for an eval set, and none of them gate anything.
Not the public benchmark. A frontier model that scores at the top of the index can still be worse than a weaker one for your specific instruction. The triage agent's replacement model was better on every public measure and read urgency differently on the one instruction that mattered. The leaderboard is the provider's evidence. You need yours.
Not the demo set. The dozen inputs the team pasted in while building are all cases that already worked. An eval set built from them tests that the model can do the thing you already saw it do. It contains no information about the boundary of failure, which is the only place a regression shows up.
Not the vibe check. Someone runs a few prompts against the new model, likes what they see, and approves. This is not a control. It is a mood, and it will be a different mood on a different afternoon.
── The four case classes ──
§ 03 · The four case classes
An eval set that gates a production agent needs four kinds of cases. Most teams build the first, sometimes the second, and skip the two that catch the failures that actually happen.
◆ Class I, the golden path. Ordinary, representative tasks with a known correct outcome. These prove the model still does the job. They are the easiest to build, they are necessary, and they are the only class most teams have. Weight them by frequency in your real traffic, so the eval mirrors the mix production actually sees.
◆ Class II, the known failures. Every case that ever broke in production goes in here, permanently, with the correct outcome recorded. The lost confirmation on a payment. The invoice with the ambiguous identifier. The outage report written in calm language. This class is the accumulated memory of your system, and it is the reason the same bug does not ship twice. Every failure report on this site is a case in someone's Class II.
◆ Class III, the adversarial cases. Inputs that carry hidden instructions, malformed tool outputs, content designed to hijack the agent. Given the rates at which models follow embedded instructions in documents they read, this class is not paranoia, it is a measurement. A model change is exactly the moment your injection resistance can silently regress, and nothing in Class I will notice.
◆ Class IV, the drift sentinels. Cases at the edge of the agent's judgment, where the correct answer is defensible but not obvious. Their purpose is not pass or fail. Their purpose is to be sensitive. When a model changes character, subtly weighting tone over impact, being more or less willing to escalate, these are the cases whose answers move first. They are the canary, and they are what a golden path set can never give you, because on the golden path both models are right.
── Scoring the path ──
§ 04 · Score the path, not just the destination
A pass or fail on the final answer is not enough, and the invoice matching agent is the proof. A candidate model reached the right answer often enough to clear the threshold, and cost more in production because of how it got there: more turns, more retries, more escalations.
So every eval case records four things beyond correctness. Turns to completion, because turn count multiplies token cost rather than adding to it. Retries triggered, because a retry pays for the whole run again. Whether the run would have escalated, because an escalation is denominated in human minutes and is the most expensive term in the equation. And cost per successful task, the composite that finance actually pays.
A model that matches your current accuracy in twice the turns is a regression. On a correctness only eval it is a pass. This single change to how you score is the difference between an eval set that protects you and one that waves through the thing that doubles your bill.
── The gate ──
§ 05 · The gate: how the eval set is actually used
The eval set is worth nothing as an artifact. It is worth something as a gate, wired into three places.
On every model change, before production. A pinned version is replaced only after the candidate runs the full set and is compared against the incumbent, case by case, not against an abstract threshold. The question is never "did it pass." The question is "which cases did it lose that the current model wins." A model with a higher aggregate score that regresses on Class II is not an upgrade. It is a promise to repeat a bug you already paid for.
On a schedule, against the live model. Run the set against production daily. If a provider rolls a version underneath you, the score moves, and the score moving is an alarm. This is the canary that turns a four day silent regression into a Saturday morning alert.
On every prompt and architecture change. The eval set is the spec, so it gates changes to the spec's implementation too, not only to the model. A prompt edit that improves one case and quietly breaks three is exactly the change an eval set exists to catch.
── How eval sets decay ──
§ 06 · Eval sets decay, and the decay is invisible
An eval set is not a fixed asset. It rots, in three ways, and each is silent.
Distribution drift. The mix of real traffic moves. A case set built in March mirrors March's inputs, and by July the agent is seeing a class of task that is well represented in production and absent from the eval. The set still passes. It is measuring the wrong thing. Re-weight Class I against real traffic quarterly.
Overfitting to the set. The team tunes prompts until the eval is green. The eval is now a test the agent has memorized rather than a measurement of whether it can do the job. Keep a held out slice of cases that no prompt work is ever tuned against, and treat the gap between the tuned score and the held out score as a measure of how much you are fooling yourself.
Staleness of the correct answer. Business rules change. The escalation threshold moves, a category is renamed, a policy is updated, and the eval set is still grading against last quarter's definition of correct. Now the eval is actively defending the wrong behavior. Review the expected outcomes when the business rules change, not when the eval breaks.
── The file ──
§ 07 · The EVALS.md
Create an EVALS.md alongside the case set. It records the four classes and the count of cases in each, how Class I is weighted against real traffic, the scoring definition including the path metrics and the cost per successful task calculation, the gates the set is wired into, the held out slice and its most recent gap, the last re-weighting date, and the owner.
It also records the incumbent model's current scores, case by case. That table is the thing you compare a candidate against, and without it "the new model is better" is a claim, not a finding.
── The economics ──
§ 08 · Why this is the highest leverage artifact you own
The orchestration layer is commoditizing. The model is a dependency you rent and can be swapped underneath you. The framework you chose this year is likely replaceable next year. Very little of the stack you are building is durable.
The eval set is. It is a written, executable definition of what your agent is supposed to do in a specific business, accumulated from real failures, and no platform vendor can ship it to you. When the model changes, it is what tells you whether to move. When a client asks how you know the agent works, it is the answer. It is the closest thing an agent engineering practice has to an asset that appreciates, and it is built the same way every time: one production failure at a time, written down and never removed.
── End of note ──
◆ A benchmark is the provider's evidence about someone else's tasks. An eval set is your evidence about yours. Only one of them can gate a model change.
◆ Four classes: the golden path, the known failures, the adversarial cases, and the drift sentinels. Most teams build only the first, and the first is the one that never catches a regression.
◆ Score the path, not just the answer. A model that is right in twice the turns is a regression on the invoice, whatever the accuracy column says.
◆ Eval sets decay silently through distribution drift, overfitting, and stale expected answers. Re-weight against real traffic, hold a slice out, and update the answers when the business rules change.
ORBIRESEARCH