Agent Loop: Reliability & Verification · Part 5 of 7

When More Evidence Makes an AI Agent Less Reliable

Abstract illustration of evidence overload and retrieval redundancy

More evidence sounds like a safe strategy for an AI agent. If the answer is uncertain, search again. If a migration detail is missing, retrieve another page. If a citation feels weak, collect one more source.

That instinct is useful until retrieval becomes a substitute for interpretation.

I examined two Agent Loop sessions answering an Axum 0.6-to-0.7 migration question: a baseline session and a session given stronger evidence-discipline instructions. The investigation reconstructed tool calls, source identities, repeated evidence, and the path from retrieved text to final claims.

The result is not a claim that “more retrieval always hurts.” It is narrower: additional retrieval did not automatically correct interpretation errors, and some searches repeated or failed to improve the evidence bundle. The final answer could still misstate what the primary source said.

Retrieval A/B measurements from the preserved Axum sessions

Two different failure classes

The sessions exposed two problems that are easy to mix together.

The first was retrieval inefficiency: repeated searches, non-indexed source labels, and operations that added no new relevant evidence. That is a tool-use problem.

The second was evidence interpretation. The final answer described a list of Axum types and traits as removed, while the release evidence was about the removal of a request-body generic parameter. It also claimed that no code changes were required and supplied an exact TypedHeader import path that the evidence bundle did not establish.

Those are not the same defect. A perfect retrieval plan cannot make a model faithfully interpret evidence it has already received. Conversely, a careful reviewer cannot recover evidence that the retrieval phase never found.

What the A/B comparison was for

The comparison was designed to keep the question falsifiable. Both sessions were preserved as JSON records. The analysis reconstructed the chronological tool-call sequence rather than judging only the final prose.

For each search, the investigator asked whether it introduced a new source, new content from a known source, repeated content, no relevant result, or an error. A source URL appearing twice was not treated as proof of redundancy; the returned content had to be compared.

That distinction matters. A second search on the same release page can retrieve a genuinely different section. Conversely, a different query can return the same excerpt.

The analysis also separated “the agent searched too much” from “the answer reasoned incorrectly.” That separation is the main value of the experiment.

The offline analyzer makes the difference measurable for these two preserved sessions. The baseline used 7 retrieval calls, produced 48 recognized source/chunk references, and repeated 24 reference instances across 24 unique pairs. Three calls contained embedded prior output, so their references were flagged rather than counted as proven fresh evidence.

The evidence-discipline session used 32 retrieval calls and produced 181 recognized references, but only 46 unique source/chunk pairs: 135 reference instances were repeats. It also recorded 5 unindexed-source calls, 4 no-hit calls, and 2 pagination-exhaustion signals. The analyzer found no exact repeated query groups, which is a useful caveat: redundancy was not simply copy-pasted arguments. It arose from overlapping source retrieval, pagination, and source-selection behavior.

These are measurements of the saved traces and parser signals, not a universal retrieval-efficiency score. The analyzer explicitly does not decide whether a call was semantically necessary.

The Axum evidence boundary

The canonical evidence bundle used later for independent review contained four bounded items:

  • release-note text about removed or changed APIs;
  • release-note text about the hyper::Body re-export;
  • release-note text saying that TypedHeader moved to axum-extra;
  • a docs.rs example using axum::serve(listener, app).await.unwrap().

The evidence directly supported the axum::serve example, the move of TypedHeader to the axum-extra crate, and the axum::body::Body replacement described in the release material. It did not establish the exact import path inside axum-extra. It also did not support the broader assurance that no code changes were required.

Most importantly, the release evidence did not support rewriting “the request-body generic parameter B was removed” into “all these types and traits were removed.” That is a semantic difference, not a missing citation.

Why additional evidence can reduce reliability

Every retrieval result adds text that the model must keep distinct. If the runtime does not preserve source boundaries, version labels, and claim-to-evidence links, more material increases the number of ways a statement can be assembled incorrectly.

The risk is not that a longer context window magically makes the model less intelligent. The risk is operational:

  • repeated excerpts look like independent confirmation;
  • nearby release-note bullets are merged into one claim;
  • a source name is remembered while its exact wording is lost;
  • a missing API detail is filled with familiar prior knowledge;
  • the final answer cites a source that is relevant but does not establish the exact sentence.

Retrieval therefore needs a stopping rule based on claim coverage, not on a desire to make the evidence pile larger.

A better stopping rule

For each intended claim, record one of four states:

SUPPORTED            the evidence states the claim
CONTRADICTED         the evidence states something incompatible
INSUFFICIENT         the evidence is relevant but incomplete
UNVERIFIED           the required source was not obtained

Stop searching when the claims that matter are classified—or when the next search has a specific hypothesis about which missing state it will resolve. Do not search merely because the answer feels uncomfortable.

That rule would have treated the exact TypedHeader path as insufficient rather than inventing one. It would have treated the generic-parameter claim as contradicted rather than paraphrasing it as a removal. And it would have kept retrieval inefficiency visible instead of hiding it inside a large context.

The limited conclusion

The Axum A/B investigation does not establish a universal law about evidence volume, nor does it provide a general benchmark for Agent Loop. It establishes a practical engineering warning: retrieval coverage and claim correctness are separate dimensions. In this trace, the evidence-discipline session collected substantially more retrieval output while still requiring a separate review to catch the final-answer interpretation errors.

An agent can search too little, search redundantly, or search effectively and still synthesize an unsupported claim. Reliability improves when the system records not just what was retrieved, but what each source actually licenses the agent to say.

In other words, the goal is not maximum evidence. It is the smallest evidence bundle that makes every important claim auditable.

Related: Evidence-to-Claim Verification: Why LLM Reviewers Still Fail and Testing AI Agents With Independent Agents.

Tags: AI, AI Agents, Retrieval, Evaluation, Axum

← Back to all posts