Topic
9 articles on Evaluation, written from production systems rather than documentation.
Normal testing asks whether the system works for people trying to use it. Adversarial testing asks what happens when somebody is trying to make it fail.
Moderating what users send is the familiar half. Moderating what your own product says, in your name, is the half that carries the liability.
A surprising share of what matters in an AI feature is mechanically checkable. Those checks are fast, free and boring, which is why teams skip past them.
Agents rarely crash. They loop, they stop early, and they confidently do the wrong thing, which is why ordinary error handling never catches any of it.
People are the only judges who catch what nobody thought to check for. Spending that attention on random samples wastes most of it.
Grading output with another model scales in a way humans cannot. It also has biases of its own, and a judge nobody validated is just a number.
A prompt is a load-bearing configuration string with no type checker behind it. Changing one line can break behaviour nobody remembers depending on.
Request logs tell you a call succeeded. For a model-driven system the useful record is the whole chain, because the mistake is rarely in the last step.
Without evals you cannot tell an improvement from a regression, so every change becomes an argument. That is a team problem before it is a quality one.
Whether it needs designing from scratch or rescuing from its own success, tell me what you’re building and I’ll tell you how I’d architect it.
Open to remote and hybrid work worldwide