When a market-data pipeline misbehaves at 03:14, the question is not 'what is it doing now' — it is 'what did it see at 03:14'. Logs answer the first. Only replay answers the second.
Deterministic replay means every input the system consumed is captured in order, so that any window — a minute, a day, ninety days — can be re-run and produce the same outputs, bit for bit. We built it first to debug our own agent prototypes. It belongs in every pipeline we deliver.
Incidents stop being guesswork
A post-incident review that can replay the exact inputs is a review that ends with a fix. One that cannot ends with a theory. The difference is concrete: 'we think it was a burst' becomes a reproduced fault with a line number.
Evals become continuous
If you can replay history, you can run the new version of the system against last week's traffic before deploying it. The eval set stops being a fixed file and becomes 'everything that actually happened'. Regressions are caught against real load, not a sample.
Agents become explainable
For an agent, replay is what makes a decision defensible. When a reviewer asks why a claim was adjudicated a certain way, the answer is not a prompt — it is the exact sequence of retrievals, tool calls and intermediate outputs that led there, re-run in front of them. That is the standard we hold ourselves to, because it is the standard a client will be held to.
Tagged: Infrastructure · Evaluation
More writing
An agent without an eval is an anecdote
Most agent demos are true. They are also useless as evidence. Here is what we insist on before an agent is allowed anywhere near production.
What a permission model actually needs
'The agent asks before it acts' is easy to say. Building Ella taught us the four properties a permission model has to have before that sentence is true.