AI agents fail for reasons we never dealt with in traditional software.
In standard software, correctness lives in the code. You write it, you verify it, you trust it. Review the code and you've reviewed every result it will ever produce.
Agents don't work like that. The correctness isn't in the code anymore; it's in every individual output. Even a capable agent will turn in bad work for reasons you couldn't predict from the prompt. You can't just trust the code. You have to inspect the work.
This is the story of what I found when I inspected mine.
The setup
I run an AI customer support agent for a Shopify store. It answers questions about orders, shipping, returns, and products, pulling from the store's policies and live catalog.
To evaluate it, I built what most teams build: an automated eval. A set of golden questions with known correct answers, run against the agent, graded by an LLM judge. I deliberately used a different vendor's model for the judge than the agent, to avoid a model grading its own family.
The judge isn't a single "rate this 1 to 5" prompt. It decomposes each answer into factual claims, verifies each claim against the context the agent retrieved, checks whether the answer addressed the customer's need, and combines those into a score. Temperature zero, structured outputs. A reasonably serious setup.
Across 17 runs and 1,234 graded responses, the agent scored 96%.
That's a number you'd ship on. Instead, I decided to check the checker.
The blind check
I pulled 58 answers from runs that shared one frozen copy of the knowledge base, so there was no ambiguity about what the agent should have known. 53 of them were answers the judge had passed, 5 it had failed. Mostly passes on purpose: you can only catch a false pass by re-grading something the judge approved.
I graded each one by hand, against the knowledge base, before looking at the judge's verdict. Three things fell out.
1. A stable score is not a stable system
I ran the identical eval three times, same questions, same knowledge base, hours apart. The aggregate scores: 95%, 96%, 96%. Rock solid.
Underneath, nothing held still. All 62 answers came back worded differently on every run. Four changed verdicts. Two flipped hard between pass and fail.
One example carries the whole finding. The question was "how much does shipping cost?" The rate was in the knowledge base: $9.95 flat, free over $100. Across the three runs the agent answered:
- Run 1: "our knowledge base doesn't include shipping rates" — wrong
- Run 2: "$9.95 under $100, free over $100" — correct
- Run 3: "I don't have access to shipping rates" — wrong
Right once in three tries, with the answer sitting there the whole time. The judge passed all three.
The aggregate held steady because the errors moved around, not because the system behaved the same twice. If your eval score is reproducible, that tells you less than you think. Rerun the eval and diff the answers, not the score.
2. The LLM judge shares the agent's blind spots
I agreed with my judge on 76% of the 58 answers. That sounds fine until you look at where we disagreed, because the misses ran one direction: the judge passed bad answers about three times as often as it failed good ones.
8 of the 53 answers the judge passed were real failures. Roughly 1 in 7. Every one of them was fluent, polite, and formatted like a right answer. No error, no alert, nothing in the logs. In production monitoring, all 8 count as wins.
And they clustered exactly where the agent was most confident: hard product facts. Twice, the agent told a customer we didn't sell an entire product category. The catalog was full of them.
Here's the mechanism. The judge grades an answer on faithfulness to the context the agent retrieved. When the agent's product search came up empty, the agent asserted the products didn't exist, and the judge, looking at that same empty context, found nothing to contradict. My judge's code literally treats "no evidence" as "no factual errors detected." A reviewer with the same inputs as the worker is not an objective reviewer.
The obvious objection: just connect the judge to the live catalog. Partly fair. But it would catch these failures only by comparing specific claims against known facts, and once you're doing that, you don't need an LLM to do it. Which brings us to the third finding.
3. Every critical failure was checkable without an LLM
All 8 of the judge's misses involved a hard fact: a product that exists, a price, a shipping rate, a restock date. Facts you can check with code.
So I wrote those checks, from the knowledge base and the golden answers only, the way you'd write them in CI before ever seeing an output. Never reverse-engineered from the failures. Then I re-ran all 58 answers through ciagent, the eval tool I built, which runs deterministic checks first and only falls back to an LLM judge when there's nothing concrete to check.
The deterministic layer caught all 8 failures my judge had passed. Zero LLM calls. A confident "I don't have that information" fails a presence check the same way a wrong price does.
Full honesty about the limits, because they're the point:
- Only 36 of the 58 answers contained a fact that code can check. The other 22 genuinely need judgment, from an LLM judge or a human. Code shrinks the blind spot. Nothing closes it.
- The checks flagged 2 answers I had passed as a human. Both were the check being stricter than me, not wrong, but that's the cost side of the ledger.
- This works because I had golden answers with hard facts in them. No goldens, no deterministic checks.
That run is why ciagent works the way it does: deterministic checks first, an LLM judge as the last resort, and golden tasks graded on correctness, tool path, and cost, so an agent that drifts fails the build instead of surfacing three days later in a customer complaint.
Was it ever really 96%?
One more piece of arithmetic the blind check makes possible. If roughly 1 in 7 of the judge's passes were failures, and a few of its fails were actually fine, the human-verified score on this slice lands somewhere in the low-to-mid 80s. Not 96.
Small sample, wide error bars. But the direction is certain, and it reframes the shipping question entirely. "Is 96% good enough to ship?" assumes the 96% is real. Mine wasn't. Anyone shipping on an unaudited LLM judge doesn't know their number either.
The shape of the wrongness
Humans make mistakes in front of customers every day, and business survives. That's because human error has a shape we know how to manage: it clusters in predictable places, and people usually know when they're unsure. They hedge, they escalate, they ask.
My agent's failures had none of that. They were silent, confident, and different on every run.
That's the real gap, and it's why "review the work" isn't a slogan. Agents are already in front of customers, shipped on scores like my 96%. The error rate isn't what worries me. It's that we're still learning how to see the errors.
The eval data: 1,234 graded responses across 17 runs; blind sample of 58 from a fixed knowledge-base slice; all checks written from golden answers before seeing agent output. Questions about the method, ask me.