Evaluation is the product

Demos are cheap. An agent that refunds the wrong customer is expensive.

Most teams still treat evaluation as a report they generate after a launch. A handful of prompts, a green average, a screenshot in Slack. That is not a quality system. That is folklore with extra steps.

If you cannot fail a pull request when an agent regresses, you do not have quality. You have anecdotes.

Unit tests prove the code still runs. They do not prove the agent still behaves. A model swap can preserve the final string and change the tools. A prompt tweak can preserve fluency and invent a fact. A retry can turn one payment into two.

The job of evaluation is not to produce a number that looks good in a dashboard. The job is to produce evidence that a specific behavior still holds, and a gate that stops the change when it does not.

That is why AgentEval is a harness, not a notebook:

• Golden cases live in git.
• Baselines are versioned.
• Infrastructure failures are not agent failures.
• Production incidents become minimized tests only after a human approves them.

A useful eval system will show you a real failure. If it never does, it has learned to hide.