An agent without an eval is an anecdote
Most agent demos are true. They are also useless as evidence. Here is what we insist on before an agent is allowed anywhere near production.
14 Aug 2026 · 6 min read
Writing
Notes from the studio
Working notes on evaluation, agent permissions and the parts of applied AI that do not fit in a case study.
01Why we write these
The things every discovery call asks
The same questions come up in nearly every first conversation: how do you build an evaluation set when you have no labels, what does an agent permission model actually look like, why does replay matter more than logging. These are our answers, written down once so we can point at them.
No newsletter, no gate. Everything here is server-rendered and readable by anyone — including the assistants people now ask about us.
023 pieces
Recent
Dealt newest first. Open any card to read.
An agent without an eval is an anecdote
Most agent demos are true. They are also useless as evidence. Here is what we insist on before an agent is allowed anywhere near production.
What a permission model actually needs
'The agent asks before it acts' is easy to say. Building Ella taught us the four properties a permission model has to have before that sentence is true.
Replay is the feature that makes everything else auditable
Traces are table stakes. Being able to reproduce a run bit-for-bit, weeks later, is the difference between a system you can operate and one you can only restart.
03
All writing
What a permission model actually needs
'The agent asks before it acts' is easy to say. Building Ella taught us the four properties a permission model has to have before that sentence is true.
22 Jul 2026 · 7 min read
Replay is the feature that makes everything else auditable
Traces are table stakes. Being able to reproduce a run bit-for-bit, weeks later, is the difference between a system you can operate and one you can only restart.
30 Jun 2026 · 5 min read
04Start here
Let's build
the impossible.
Tell us the system you can't get built. We come back with a short, paid discovery — a clear plan and a fixed first milestone — usually within two working days.
01
Discovery
A working session to map the problem and define what "good" is measured against.
02
The plan
Architecture, milestones and a fixed first deliverable — yours to keep, either way.
03
We build
Embedded with your team or as a dedicated pod, shipping with traces, evals and docs.