> ../patterns/verified_agent.md
§ 01 · The assumption that just expired
Every outbound agent built in the last two years rests on an assumption nobody wrote down: that the open web will serve a request from an automated client the same way it serves a request from a browser, provided the request is polite.
That assumption is expiring on a schedule. The edge layer now classifies automated traffic by behaviour rather than by whether it is nominally AI: crawlers that index in order to answer questions later, crawlers that collect training data, and agents acting in real time on behalf of a person. Those categories are managed independently, and from the middle of September 2026, newly onboarded domains receive defaults in which agent-class and training-class traffic is blocked on pages carrying advertising while search-class traffic stays allowed. Free-tier sites that have not changed their settings move to those defaults as well.
Your agent is in the second-worst category on that list, and the reason is not hostility toward agents. It is that a page carrying an advertisement was monetised on the assumption a human would see it, and an agent reading that page consumes the cost of serving it without producing the revenue that justified serving it.
The failure mode this produces is not an outage. We wrote up the version of it that reaches production last week: challenge pages returned with a 200 status, empty extractions, and confident summaries of nothing. The pattern below is what an outbound agent has to look like on the other side of that change.
── Declare yourself ──
§ 02 · Layer one: declared identity
The first decision is whether your agent tells the truth about what it is. There is a persistent temptation to solve blocking by spoofing a browser user agent and rotating residential addresses. Do not. Aside from the terms-of-service problem and the fact that fingerprinting defeats it anyway, it makes your traffic indistinguishable from abuse, which means the only available response from the other side is to block it.
Declared identity has four components.
A stable user agent string that names the operator, states the purpose, and carries a URL to a page describing the agent: what it fetches, how often, what it does with the content, and how to contact you or ask you to stop.
A dedicated address range or a documented set of egress addresses that only this agent uses. Shared cloud address space is a category signal, and the category it signals is not good.
A single declared purpose per crawler. If one system indexes, one acts in real time on behalf of a user, and one collects data for training, those are three crawlers with three identities, not one crawler doing three jobs. Multi-purpose crawlers are now evaluated under the most restrictive rule that applies to any of their purposes, which means the most benign of your three behaviours inherits the blocking posture of the least benign. Separating them is not politeness. It is the difference between being allowed and being blocked.
Machine-readable respect for stated preferences. Read robots.txt, honour crawl delay, and parse the content signal fields that express how a site permits its content to be used. Whether those signals are legally enforceable is not the question you should be optimising. Ignoring them is now grounds for losing verified status, and verified status is what keeps the door open.
── Prove it ──
§ 03 · Layer two: cryptographic verification
A user agent string is a claim, and anyone can make it. The infrastructure layer has moved past accepting claims, which is why the same declaration is now expected to be signable.
The mechanism is HTTP message signatures. The agent publishes a key directory, registers the directory URL and its user agent with the verification programme, and then signs every outbound request, attaching the signature agent, signature input and signature headers. The receiving side can now distinguish your agent from anything impersonating it.
Two consequences worth internalising. First, verification no longer implies access. Under the current model, being a verified bot confirms who you are; whether you are allowed through depends on your behavioural classification and the site owner's policy for that class. Verification is necessary and not sufficient. Second, verification is revocable, and it is revoked for exactly the behaviours you would expect: ignoring stated crawl preferences, reproducing content in full, and traffic that does not match the declared purpose. The identity is an asset with an integrity requirement attached.
There is a further wrinkle for anyone building agents that act through a platform rather than directly. The distinction now tracked is between an operator running its own agent and an intermediary operating on behalf of many end users. If you are the intermediary, you are asserting something about parties you do not control, and you should expect that assertion to be scrutinised. This is the same accountability chain the agent identity note describes: identity has to answer who acted, on whose behalf, and under what authority. The web is now asking those three questions at the edge.
── Pay, sometimes ──
§ 04 · Layer three: paid access, with a budget and a gate
The blocking regime arrived alongside a payment regime. HTTP status 402, dormant in the specification since 1997 because no human wants to approve a micropayment per page, becomes usable precisely when the requesting party is not a human. An agent can receive a 402, read the stated price, settle it, and retry with proof, in a fraction of a second and with no interaction.
Which is exactly why this needs a gate.
An agent that can pay for access is an agent with a spending capability, and every rule that applies to a write-operation agent applies here. Three controls are mandatory before any agent is allowed to settle a 402.
A hard budget per task and per period, enforced by the runtime rather than by the prompt. This is the same lesson as the agent that burned four thousand dollars over a weekend: a limit that lives in the system prompt is not a limit.
A price ceiling per resource, with anything above it routed to the approval queue rather than settled automatically. A resource priced ten times above the norm is either a mistake or a trap, and neither is something an agent should resolve alone.
A full record of every settlement: which task, which resource, what price, under which budget, and what the retrieved content was used for. Spending without an audit trail is the accountability gap the guidance from the national security agencies has been naming since the spring.
The strategic point behind the tactical one: paid access converts an availability problem into a cost-per-successful-task problem, and cost per successful task is a number you already know how to reason about. Sources you must have and cannot get otherwise become a line item. Sources you merely like become a routing decision.
── Fail loudly ──
§ 05 · Layer four: degradation that stays honest
This is the layer teams skip, and it is the one that produces the incidents.
A retrieval is successful when the response satisfies the shape the extractor expects. Not when the transport succeeded, and not when the body is non-empty. Every source carries an assertion describing what a real response from it looks like: a headline element, a price table, a minimum count of extractable blocks. A challenge page fails that assertion instantly, because it is structurally nothing like the page it replaced.
The retrieval layer then returns explicit distinct states, never one collapsed value. Retrieved and changed. Retrieved and unchanged. Blocked at the edge. Payment required and declined by policy. Failed for transport reasons. Blocked and unchanged must never be the same value, because one is a normal outcome and one is a hole in the evidence.
Downstream, the synthesis step is contractually forbidden from producing an artifact when coverage falls below a threshold. It escalates instead. And every artifact the agent produces states its own evidence base: sources attempted, retrieved, blocked, paid for. A reader who can see that coverage was sixty percent reads the conclusions differently, which is the whole purpose of putting the number there.
── Watch the boundary ──
§ 06 · Layer five: treat access posture as an operational metric
Blocking is now a moving target, and a source list assembled in March is a source list that decays.
Retrieval coverage gets a baseline and an alert band, checked per source and in aggregate, before any artifact is produced rather than after someone notices a gap. Per-source access posture is reviewed on a schedule: does this source publish a feed, does it state terms about automated access, does it charge, and have we been blocked by it in the last thirty days. Sources that consistently block are either replaced with a licensed or feed-based route, or removed from the list and named in the output as a known blind spot.
Two dates belong in an operational calendar rather than a blog post. The middle of September 2026, when the new defaults take effect, is a foreseeable step change in coverage for any agent touching ad-supported pages. And the opt-out window before that date matters in the other direction: if you also operate a site, the same defaults will govern which agents can read you, and multi-purpose search crawlers are caught by the training rule under the most restrictive interpretation.
── Checklist ──
§ 07 · Before you ship an outbound agent
[ ] Stable, honest user agent string naming operator, purpose and a contact page. No browser spoofing anywhere in the stack.
[ ] Dedicated, documented egress addresses.
[ ] One declared purpose per crawler identity. Indexing, acting, and training data collection are never the same crawler.
[ ] robots.txt, crawl delay and content signals parsed and honoured in code, not in policy documents.
[ ] HTTP message signatures on every outbound request, with a published key directory and registered identity.
[ ] Runtime-enforced spending budget per task and per period, a price ceiling above which settlement routes to human approval, and a full settlement audit trail.
[ ] Per-source content assertions. A fetch is successful when the response has the expected shape, never because it returned 200.
[ ] Five distinct retrieval states. Blocked and unchanged are never collapsed.
[ ] Coverage threshold below which the agent escalates instead of producing an artifact.
[ ] Every artifact states its own evidence base.
[ ] Retrieval coverage dashboarded with an alert band, and access posture reviewed monthly.
── End of pattern ──
◆ The open web is no longer neutral toward agents. Verification confirms who you are, classification decides whether you get in, and those are now two separate questions.
◆ An agent that can settle a 402 is an agent with a spending capability. Runtime budget, price ceiling, approval gate above it, and a settlement audit trail. Same rules as any write operation.
◆ Being blocked is a legitimate outcome. Being blocked and not knowing it is the defect. Assert on content shape, keep blocked separate from unchanged, and put coverage in the artifact.
ORBIRESEARCH