How to tell if your AI agent actually got better

People keep asking me how you actually know an agent got better. I collected my answers to the most common questions in one place.

Isn't it obvious from the answers?

No. The agent says "You're booked for Thursday," and that sentence tells you nothing.

Anthropic's guide on agent evals shows it with a flight-booking agent: it may say "Your flight has been booked," but the outcome is whether a reservation actually exists in the database. What counts is the state the agent leaves behind, not what it says.

So the questions that matter are boring. Did the row appear in the database? Did the tool call fire with the right arguments?

A demo watches the reply. An eval watches the system. That is why demos always look great.

Doesn't an LLM judge solve this?

Partly. The judge brings its own bias.

In June a study of 21 LLM judges came out, Claude Opus 4.6 and GPT-5.4 among them. One of the tests: show the judge two answers, then swap their order and ask again. A good judge should not care which one came first.

On the easier benchmark, the median judge flipped its verdict 9% of the time. On the harder one it was 17%, and some judges went as high as 38%. The harder the task, the closer the judge gets to a coin toss.

The unpleasant part is that rerunning does not help. On a rerun, the same judge gives the same verdict roughly nine times out of ten or more, bias included. Stable is not the same as correct, so the only way to catch the bias is to calibrate the judge against human labels.

Should I score from 1 to 10?

Binary. Pass or fail.

The gap between a 3 and a 4 is a matter of taste, and everyone hides their uncertainty in the middle of the scale. A binary verdict makes you name the reason something failed instead of quietly giving it a 7.

How many cases do I need?

Dozens, not thousands. Start with 20 to 50 tasks pulled from real failures.

Thirty ugly cases from production beat a thousand synthetic ones.

So evals come before prompts?

It sounds logical, like TDD. For capabilities it can work: writing the eval first pins down what done means, and the Anthropic guide recommends exactly that.

For failures, no. An LLM can break in practically endless ways, so whatever you write upfront covers the failures you imagined, not the ones that will happen.

For failures the order is different:

ship → read the traces → write an eval for a failure that really happened.

While you are looking at replies, you are not measuring the agent, you are measuring its politeness.