The agent that filed a confident weekly report on an empty web

> ../failures/empty_web_blocked_fetches.md

§ 01 · What happened

A competitive monitoring agent ran on a schedule. Every weekday it pulled a fixed list of around ninety sources: vendor blogs, pricing pages, trade press, a handful of regulatory feeds. It extracted anything that had changed since the previous run, clustered the changes by theme, and delivered a short brief into a channel where three people read it before their Monday call.

It had run without incident for months. Then, over about eleven days, the briefs got shorter. Not empty. Shorter. Fewer items, more hedged language, more sentences of the form "no material announcements were identified this period from the pricing and packaging cohort."

Nobody escalated. Quiet weeks happen. The briefs were still well written, still correctly formatted, still delivered on time. The people reading them adjusted their expectations rather than their trust.

The problem surfaced when one of the three readers saw a competitor's pricing change on a public page, went back through the last two weeks of briefs, and found no mention of it at all. Reconstructing the run logs showed that on the day that page changed, the agent had fetched it, received a response, and recorded the fetch as successful.

§ 02 · What the retrieval layer actually returned

The agent's fetch step treated any 2xx response as a successful retrieval. That was the whole success criterion.

What a growing share of those ninety sources were returning was a 200 response whose body was not the page. It was an interstitial: a bot challenge, a "verify you are human" shell, or in a few cases a short access notice served in place of the article. Status code 200. Content type text or html. Non-zero body length. Every check the retrieval layer performed, it passed.

The extraction step then ran over that body, found no article structure, and returned an empty change set. Empty change set is a legitimate state. A page that genuinely has not changed also returns an empty change set. The agent had no way to distinguish "I read this page and nothing was new" from "I did not read this page."

The synthesis step received a set of empty results, did exactly what it was asked to do, and wrote an accurate summary of the input it was given. The input was wrong. The output was faithful to it.

This is the same structural shape as the false consensus failure we traced in May. No component behaved incorrectly. The fetcher fetched, the extractor extracted, the summariser summarised. The system produced a confident falsehood because the contract between two of its steps was underspecified.

── What changed underneath ──

§ 03 · The change underneath the failure

The agent did not break. The web moved.

Through the first half of 2026 the infrastructure layer stopped treating automated traffic as one category. The dominant classification now separates crawlers that index content, crawlers that collect training data, and agents acting in real time on behalf of a person. Those three are managed independently, and the third one, the class that most production agents fall into, is increasingly blocked by default on commercially monetised pages.

The direction is not ambiguous. From the middle of September 2026, newly onboarded domains on the largest edge network receive defaults that block agent-class and training-class traffic on pages that display advertising, while search-class traffic remains allowed. Daily agent requests across that network grew by more than seventeen hundred percent over the preceding year, which is the pressure that produced the policy.

For anyone operating an outbound agent, the practical consequence is that the share of the open web that will serve you a real response is shrinking, and it is shrinking silently. You are not being told no. You are being handed something that looks like a yes.

Any agent whose retrieval success criterion is a status code was already broken. It just had not encountered enough blocked sources to notice.

── Why nothing fired ──

§ 04 · Why no alarm fired

Three separate gaps had to line up, and they did.

The success criterion was transport-level, not content-level. The agent asked "did the request complete" when the question that mattered was "did I receive the document I asked for."

The empty state was overloaded. One value, an empty change set, meant two very different things: nothing changed, and nothing was read. Overloading a state like that removes the system's ability to distinguish a normal outcome from a failure.

And the output carried no provenance. The brief said what had happened. It did not say what had been read. A brief that had opened with "eighty one of ninety sources retrieved" would have been caught by a human on day one, because the number would have been fifty something.

The instrumentation existed for the failures that announce themselves, which is the gap the instrumentation pattern describes. Nothing was watching for the failure that arrives dressed as a normal quiet week.

── The fix ──

§ 05 · The fix, in three parts

Part one: assert on the content, not the status code. A fetch is successful when the response satisfies the shape the extractor expects, not when the transport succeeded. In practice that means a per-source assertion recorded alongside the source itself: this page must contain a headline element, or a price table, or at least a minimum count of extractable text blocks. A response that fails the assertion is a retrieval failure, regardless of status. Bot challenge shells fail these assertions immediately, because they are structurally nothing like the page they replaced.

Part two: separate the empty states. The retrieval layer now returns one of four distinct values per source: retrieved and changed, retrieved and unchanged, blocked, and failed. Blocked and unchanged are never collapsed. The synthesis step is contractually forbidden from summarising a run in which the blocked count exceeds a threshold. It escalates instead.

Part three: put coverage in the artifact. Every brief now opens with a retrieval line: sources attempted, sources retrieved, sources blocked, and the names of anything in the blocked list that the reader has previously flagged as important. The output states its own evidence base. A reader who sees a coverage number of sixty percent reads the rest of the brief differently, which is the entire point.

── The control that runs now ──

§ 06 · The control that runs now

Retrieval coverage is a first-class metric with a baseline and an alert band, on the same dashboard as cost per successful task. It is the number that would have caught this on day one, and it is checked before the brief is written, not after someone notices a gap two weeks later.

Separately, the source list now carries a per-source access posture, reviewed monthly: whether the source has a public feed, whether it publishes terms about automated access, and whether we have a declared identity that source can verify. Sources that repeatedly return challenges are either replaced with a licensed or feed-based route, or removed from the list and marked as a known blind spot in the brief.

Being blocked is a legitimate outcome. Being blocked and not knowing it is the defect.

── End of report ──

ORBIRESEARCH