01 · Evaluation infrastructure
AgentEval
CI for AI agents that turns production failures into minimized regression tests.
- Deterministic tests
- 1115+
- PyPI release
- v0.3.0
- Framework adapters
- 4
- Failure Memory tests
- 59
The story
I built AgentEval because I kept watching agent systems fail the same way: a prompt tweak, a model swap, a tool rename — then a silent regression in production. Observability tells you something happened. Evaluation tells you whether it is still allowed to happen. The product is not a dashboard. It is a regression gate with evidence.
Problem
An agent can return a confident answer, call the wrong tool, invent a fact, or get slower after a one-line prompt change. Unit tests prove the code runs. They do not prove the agent still behaves.
Why it matters
Teams ship agents that issue refunds, write SQL, and talk to customers. Without an evaluation gate, quality is a handful of plausible demos. AgentEval makes agent behavior reviewable, comparable, and blockable in pull requests.
Architecture
Adapters normalize CrewAI, AutoGen, LangGraph, OpenAI Agents SDK, and custom agents into one contract. The runner executes YAML goldens, scores correctness, hallucination, tool-call accuracy, trajectory, latency, and cost, then compares against a versioned baseline. Failure Memory captures production traces, redacts, clusters, replays, minimizes, and promotes human-approved cases back into CI.
System
A harness, not a notebook.
Agent / adapter
CrewAI · AutoGen · LangGraph · OpenAI SDK
Golden suites
YAML expectations, tools, trajectories
Production recorder
Redacted traces, opt-in capture
Runner + evaluators
Correctness · tools · RAG · SQL safety
Failure Memory
Cluster · replay · minimize · approve
Local SQLite
Artifacts, provenance, SHA
JSON / HTML reports
Case-level evidence
CI gates
Baseline + production coverage
Dashboard / API
Streamlit + local read-only API
Production failure lands as a redacted trace.
- pass
1. deterministic
pytest · no network · taxonomy integrity
- pass
2. evaluate
agenteval run · YAML goldens · adapters
- may fail PR
3. compare
baseline delta · correctness / tools / cost
- pass
4. memory
coverage of approved production cases
Correctness
95.2%
Hallucination
4.8%
Tool F1
100%
Suite
1115 pass
| Case | Correct | Tools | Hallu | Traj | Flake |
|---|---|---|---|---|---|
| refund_order_4821 | PASS | 1.00 | 0.00 | 0.94 | stable |
| cancel_without_lookup | FAIL | 0.50 | 0.18 | 0.41 | — |
| rag_citation_bound | PASS | 1.00 | 0.00 | 0.88 | stable |
| sql_drop_blocked | PASS | 1.00 | 0.00 | 1.00 | stable |
| indic_script_drift | PASS | 1.00 | 0.00 | 0.91 | flaky |
Technical decisions
Local-first, git-native
Golden cases live in YAML. Baselines are versioned artifacts. CI exit codes are the product. The harness does not require a hosted control plane to be useful.
Metric integrity over green scores
Provider timeouts and evaluator failures are not agent failures. Taxonomy keeps infrastructure noise from corrupting quality numbers. A useful eval system exposes real failures instead of hiding them.
Human approval before CI promotion
Failure Memory never auto-enters blocking suites. Capture is off by default. Secrets are redacted before disk. Minimized cases become goldens only after a human signs them.
Framework-agnostic adapters
Reliability is a contract, not a framework feature. CrewAI, AutoGen, LangGraph, and the OpenAI Agents SDK all run through the same evaluation surface.
Challenges
Probabilistic behavior in deterministic CI
Agents are not pure functions. Flakiness detection, repeat runs, and consistency labels separate unstable behavior from one-off luck.
Trajectory, not just the final string
The right answer with the wrong tools is still a failure. LCS-based trajectory scoring compares expected step sequences without forcing a single legal path.
Production incidents as tests
Raw traces are noisy, secret-bearing, and oversized. The minimizer shrinks payloads while preserving reproduction so CI stays cheap and reviewable.
Outcomes
- 01
Published as nishanttyagi-agenteval on PyPI with a direct agenteval CLI.
- 02
v0.3.0 release verification: 1115 passed, 1 skipped; Failure Memory suite 59 passed.
- 03
Composite GitHub Action for consumer repositories, plus a Streamlit dashboard and HTML reports.
- 04
RAG evaluation, SQL agent safety scanning, Indic-language pack, and zero-network flagship demo.