How to evaluate an agent before it touches production

The acceptance test I keep finding in agent projects is a sentence: it worked in the demo. The demo is real, the workflow completed, the room was impressed, and none of that answers the production question. A demo exercises the happy path, and the happy path was never the risk. The risk is what the agent does with a tool it chose wrong, a permission it should not use, a task it half-completed, a retry that acted twice, and an instruction that arrived hidden inside the data it was processing.

The security field has already made this turn. In December 2025, OWASP published its Top 10 for Agentic Applications, built by hundreds of contributors with a Distinguished Expert Review Board drawing on NIST, Microsoft, AWS, and others, on a blunt premise: “once AI began taking actions, the nature of security changed forever.” Agent goal hijack (ASI01), tool misuse (ASI02), memory and context poisoning (ASI06), and rogue agents (ASI10) are named, numbered risk classes now. Your evaluation should test for them by name.

What changes when the AI acts

A chatbot's failure is a wrong paragraph. An agent's failure is conduct: it is stateful, meaning it carries work-in-progress from step to step, and it chooses among tools, operates under permissions, spends real money, and can stop halfway through a job that has already changed your systems. That is a business process, and mature organizations already know how to hold processes accountable: define the work, bound the authority, record what happened, and audit the record. Agent evaluation is that discipline applied to software that improvises. The task contract still anchors it: inputs, outputs, quality gates, and legitimate non-success states, written before any test runs, because you cannot grade conduct without a definition of correct.

The four case classes

Normal casesprove the agent does the job: the real task distribution at real volume. The vendor guidance here is solid and worth following: Anthropic's evaluation documentation pushes task-specific test sets, automated grading, and volume over hand-graded perfection, with a different model grading than generating. Hundreds of cases beat a dozen beautiful ones.

Edge cases prove the agent survives the messy middle: malformed inputs, ambiguous requests, missing records, tools that time out, data that contradicts itself. Every mature test discipline has this class; agents add the twist that an edge case can strand a half-completed job.

Adversarial cases prove the agent resists redirection. This is where the OWASP list earns its place in your test plan: prompts that attempt goal hijack, inputs that try to smuggle instructions through processed content, tool responses crafted to mislead the next step, and poisoned context. A red team helps, and you can start without one: write test cases from the named risk classes and run them on every release.

Forbidden cases are the class agent evaluation adds to the world, and the one I weight heaviest. Write down the actions this agent must never take: delete records, contact a customer without approval, exceed a spending threshold, act outside its permission scope. Then hand it realistic opportunities to cross each line, and require a refusal, an escalation, or a hard stop, every time. The demo never shows you this class. The evaluation must.

Record the whole trace, score two numbers

Modern tooling makes the recording straightforward: OpenAI's agent-eval guidance centers trace grading, an end-to-end record of model calls, tool calls, guardrails, and handoffs (the moments work passes to another agent or a person) for each run, scored against structured criteria (did the agent pick the right tool, did the handoff happen when it should). Use it, or build the equivalent. The trace is the evidence; without it you are grading final answers and guessing about conduct.

Then keep two scores separate: task completion and policy adherence. The separation is the point. An agent that completes 95% of tasks while quietly violating policy twice is worse than one that completes 85% and stops cleanly at every boundary. The second fails safely. The first fails in ways you find out about later. One number hides that difference. Two numbers make it a release decision, and the gate pair reads plainly: completion at or above whatever threshold the business set, adherence at one hundred percent on the forbidden rows, and no trading one for the other.

The Production Agent Evaluation Scorecard

Interruption deserves its row: kill the agent mid-task and restart it. A production agent lives in a world of timeouts and deploys, and replay safety (recovering without acting twice) is the difference between an incident and a non-event.

Release gates, rollback, and the part that never ends

Pre-deployment testing is a standing governance expectation: NIST's generative AI profile organizes its suggested actions around four considerations, governance, content provenance, pre-deployment testing, and incident disclosure, and notes that monitoring for confabulation “may be especially important” where decisions are consequential. Translate that into gates: the scorecard passed at volume, forbidden actions at one hundred percent, a rollback that has actually been rehearsed, and named humans owning escalation. Then keep monitoring, because the threat classes keep growing and no benchmark guarantees safety in an open environment. Evaluation bounds the risk. It does not retire it.

This is how I run my own systems: operations built replay-safe, errors and refusals and incomplete results recorded as explicit outcomes instead of retried into silence, cost recorded per run, and human gates where consequence warrants them. The cited-or-declined contract is the same philosophy applied to answers; this scorecard applies it to actions. Scale the matrix to consequence: a low-stakes internal agent needs a lighter version, and an agent touching customers or money needs every row.

If you are about to accept an agent on the strength of a demo, ask the vendor for one artifact first: the trace of a run where the agent refused something. A system that cannot show you its stopping behavior has only been rehearsed.

Questions worth asking next

Sources

1. OWASP GenAI Security Project, "OWASP Top 10 for Agentic Applications," December 9, 2025. https://genai.owasp.org/2025/12/09/owasp-top-10-for-agentic-applications-the-benchmark-for-agentic-security-in-the-age-of-autonomous-ai/
2. OpenAI, "Agent evals" guide. https://developers.openai.com/api/docs/guides/agent-evals
3. Anthropic, "Create strong empirical evaluations" documentation. https://platform.claude.com/docs/en/test-and-evaluate/develop-tests
4. NIST, "Generative Artificial Intelligence Profile" (NIST AI 600-1), July 2024. https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf