Guide Details Step-by-Step Best Practices For Evaluating AI Agents

Do you need to learn how to properly evaluate your agent? Here’s a step-by-step guide for how to do this, informed by best practices in recent research… (1) Define success. We need to first think about what it means for the agent to succeed. We should write clear and detailed criteria such as: - Outcome goals that verify aspects of the outcome (e.g., whether the expected database entries for the task were created). - Process goals that verify components of the transcript (e.g., whether certain tools were called). Recent agent benchmarks are heavily outcome-oriented, as outcome goals provide a reliable and objective mechanism for assessing the success of an agent. (2) Collect a small task set. Instead of curating a lot of data up front, we can start with a small number of tasks that we manually curate for evaluating the agent. As we use the agent and find new failure cases, we should record these issues and use them to add new tasks to our evaluation suite. Over time, we should continue collecting new—usually more difficult—tasks that challenge the agent. Legacy tasks can be maintained in a regression set. (3) Create useful tasks. We should create high-quality tasks that test important aspects of agent behavior in a reliable manner. Tasks should be clear enough that repeated evaluations yield consistent results. Ambiguous or noisy tasks complicate the evaluation process with unstable and misleading results that can obfuscate the actual performance of an agent. (4) Configure graders. We should begin with simple graders like deterministic checks (e.g., check if tools were called or if a final answer matches ground truth) because they are simple and easy to debug. For subjective criteria (e.g., code style) we need model-based graders (LLM-as-a-Judge) or human review. The human evaluation process should be calibrated, and we should monitor the level of agreement between LLM judges and human experts. (5) Build the evaluation harness. We must be able to execute the evaluation efficiently and repeatably. To do this, we can create an evaluation harness that: - Runs the agent in a realistic (but controlled) setup. - Collects the transcript, including tool calls and intermediate outputs. - Captures the final outcome. The agent should ideally use the same scaffold, tools, and environment that are used in production during the evaluation process. Each trial should start from a fresh environment to avoid any failures caused by shared state or evaluation infrastructure issues. (6) Inspect, iterate, and maintain the benchmark. Agent evaluations can become saturated quickly, so we should treat evaluation suites as living artifacts that continually improve in difficulty, diversity, and reliability. The best agent evaluations evolve continuously through new failure cases and ongoing maintenance.