AI Agent Evals: How Do You Know an Agent Actually Works?

A demo proves a path is possible. An eval measures whether it holds.

A demo is useful: it can reveal a compelling capability and help a team decide whether a problem is worth solving. But a successful live run does not establish how reliably the system works, what happens at the edges, how much it costs, or whether it succeeded for the right reason. Those are evaluation questions.^[1][2]

Anthropic describes an agent eval in terms of a task, trials, and a grader. The task states the job. A trial is one attempt by the agent. The grader evaluates the result against criteria that matter. For agents, the evidence should include not just the final output but the trace of actions and the resulting environment state.^[1]

Start with the outcome, not the agent’s prose

The most important design move is to define what must be true when the task is complete. “Agent produces a helpful answer” is vague. “The requested record exists with these validated fields” is testable. “The failing test passes and no unrelated tests regress” is testable. “The answer cites the retrieved primary sources and states uncertainty where evidence is incomplete” can be reviewed against a rubric.

1. Research taskCheck the cited sources, whether claims match them, whether dates are correct, and whether the answer distinguishes fact from inference.
2. Coding taskCheck the resulting diff, targeted tests, broader regression tests where appropriate, and whether the requested behavior actually changes.
3. Browser taskCheck the state of the site or application after the run—not merely the text the agent returned after clicking buttons.
4. Support taskCheck that the answer is correct, follows policy, cites the approved knowledge source, and routes high-risk requests to a person.

Use the strongest practical grader

Different outcomes call for different graders. Deterministic checks are the clearest when they exist: a test suite, schema validator, expected file, database query, policy assertion, or exact state transition. Some tasks require judgment, such as whether a research synthesis is complete or whether a response is appropriately nuanced. In those cases, a rubric can support model-assisted or human review.^[1][2]

• Deterministic grader: Did the test pass? Does the record match the schema? Did the required state change occur?
• Model grader: Does the response cover the rubric, remain grounded in supplied evidence, and avoid unsupported claims?
• Human grader: Is the result useful, safe, and acceptable in a high-consequence or ambiguous setting?
• Trace review: Did the agent use the permitted tools, make unnecessary attempts, or succeed through a loophole that will not generalize?

A model grader can be useful, but it should not conceal an objective check that is available. For example, a model can help judge whether an incident summary is clear, while a deterministic check verifies that the incident ID, owner, and timeline are present. Combining graders makes the evaluation more honest about what is measured and what remains judgment.^[1]

Run more than once—and read the traces

Agent runs can vary because model sampling, tool timing, retrieved context, and environment state can vary. One pass is a useful data point, not a reliability claim. Run a task multiple times, record the outcome, and inspect failures rather than reducing everything to a single success percentage.^[1]

task_id: update-the-customer-record
trial: 04
outcome: pass | fail | blocked
external_check: record matches schema and expected fields
trace: tools used, observations, retries, final state
metrics: completion time, tool calls, cost, human intervention
notes: unexpected behavior and suspected cause

Trace review matters because an agent can pass for a fragile reason. It may retrieve the answer from an accidental hint, take an unsafe shortcut, retry excessively, or nearly violate a boundary. The trace gives engineers the material to improve context, tool design, permissions, or the agent loop—not merely to celebrate a pass.^[1]

Build a first suite before you scale the agent

You do not need a benchmark with thousands of cases to begin. Create a small set of tasks that represent the job the agent is supposed to do, include a few easy cases and a few failure-prone cases, and define the grader before you tune the system around the examples. Keep the suite separate from ad hoc demonstrations so it can catch regressions when the prompt, tools, model, or harness changes.^[1][2]

1. Choose representative workUse tasks drawn from the real workflow, with sensitive data removed or safely simulated. Do not evaluate only on the smoothest examples.
2. Write the verifier firstState how a pass is determined before optimizing prompts. This protects against building a system that merely looks good in a demo.
3. Keep a regression setWhen you fix a meaningful failure, preserve the task and its grader. Future changes should demonstrate that the fix held.
4. Report the limitsSay what tasks were tested, how many trials were run, and what the eval does not measure. Honest scope is part of useful evidence.

The result is not certainty. It is a repeatable way to learn whether an agent is improving, regressing, or only appearing capable. That is the shift from a demo culture to an engineering practice: success has a definition, evidence, and a record of how it was checked.^[1]