Every agent demo we are shown works. That is not a compliment. A demo is one run, on one input, chosen by the person who built it. It tells you the system can succeed. It tells you nothing about how often, on what, or what it does when it fails.
In ordinary software this distinction barely matters, because ordinary software is deterministic: if it worked on the demo input it will work on the same input tomorrow. Agents are not like that. They fail probabilistically, they fail mid-plan, and they fail silently — producing something that looks like an answer. A single successful run is an anecdote.
What an eval actually is
An evaluation set is a collection of real inputs, each paired with what a correct outcome looks like, that the agent is run against before it ships and every time it changes. The score is the contract. If it drops, the change does not merge.
The important word is real. Public benchmarks measure whether a model can do tasks in general; they say nothing about your documents, your edge cases, or the specific way your process fails. The eval has to be built from the client's own history — the cases that went wrong, the cases that were ambiguous, the cases where a human had to step in.
How we build one when there are no labels
Clients rarely have a clean labelled dataset. What they have is history: tickets that were escalated, decisions that were reversed, outputs a reviewer corrected. That history is the eval set in disguise.
We start with the reversals. Every case where a human overrode the previous process is a case with a known correct answer and a known wrong one. A few dozen of those, plus a sample of routine cases, is enough to tell whether a first version is acceptable — and, more importantly, enough to catch a regression when the prompt or the model changes.
What we refuse to do
We decline engagements where the client will not agree on how correctness is measured. It sounds severe. It is the only honest position: a system nobody can prove is working cannot be defended after it ships, and it is the client who will be asked to defend it.
Tagged: Agentic AI · Evaluation
More writing
What a permission model actually needs
'The agent asks before it acts' is easy to say. Building Ella taught us the four properties a permission model has to have before that sentence is true.
Replay is the feature that makes everything else auditable
Traces are table stakes. Being able to reproduce a run bit-for-bit, weeks later, is the difference between a system you can operate and one you can only restart.