AI Evals Handbook

A Practical Guide to evaluating AI Agents

Transcript

1. A Practical Guide to Evaluating AI Agents
2. Contents Understanding Agentic Systems and their Quality 1 Metrics for Evaluating Agentic Workflows 6 Building Robust Evaluation workflows for AI Agents 14 Conclusion 18
3. Understanding Agentic Systems and their Quality 1 To define agents, we can turn to Anthropic’s definition of building effective agents:  “Systems that can autonomously perform tasks by perceiving their environment, processing the information, and acting upon it to achieve specific objectives.” This definition underscores the ability of agents to adapt and make decisions based on the context, setting them apart from simpler systems that only follow pre- determined instructions. To understand the architecture of effective agents, it’s essential to consider key components such as tool use, planning, memory, and reasoning: Tool use: Agents can interact with external tools or systems to extend their capabilities. For instance, an AI agent might use a web browser to retrieve information or access a database to fetch relevant data. This interaction allows agents to perform tasks beyond their inherent capabilities. Planning: Effective agents can formulate plans to achieve specific objectives. This involves setting goals, determining the necessary steps, and executing actions in a sequence that leads to the desired outcome. Planning enables agents to handle complex tasks that require multiple steps and decision points. Memory: Agents with memory can retain information from past interactions (long- term memory) or across multiple steps of the interaction (short-term memory). This capability allows them to provide contextually relevant responses, learn from previous encounters, and improve over time. Reflection: Reflection enables agents to evaluate past actions and outcomes, allowing them to draw inferences and make informed decisions based on available data. This cognitive ability helps agents handle ambiguity, solve problems, and adapt to new situations by learning from previous experiences and adjusting their strategies accordingly. What are Agents? In today’s rapidly advancing world of artificial intelligence (AI), agentic systems are becoming an integral part of numerous industries, powering everything from customer support to robotics. But what exactly are these systems, and why is measuring their quality so critical for businesses and users alike? We will explore the nature of AI agents, their various types, real-world applications, and the importance of evaluating their quality for widespread adoption.
4. 2 Some important architectural differences between simple workflows and agentic systems are: Aspect Workflow Agents Definition
 A variety of definitions exist, but we find Anthropic's to be the most accurate. Workflows are systems where LLMs and tools are orchestrated through predefined code paths. Agents, on the other hand, are systems where LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks. Example Airline Ticket Booking Workflow:
 User selects flight Enters passenger details Makes payment Receives confirmation email Virtual Assistant for Flight Booking:
 Suggests alternative routes if a flight is unavailable. Processes and understands natural language queries (e.g., “Find me the cheapest flight to New Delhi for next weekend.”) Learn from past bookings to refine recommendations. Nature Predictable, operates based on predefined logic without real-time decision-making beyond set rules. Adaptable, responds to unexpected inputs, and improves over time. Decision making Follows a rigid structure, executing tasks exactly as defined. C an make flexible, intelligent decisions based on real-time data and learning. I nteraction w ith users Minimal or predefined interaction based on structured inputs. Engages dynamically with users, understanding and processing queries in natural language.
5. 3 Operates under direct human supervision, executing predefined commands without autonomous decision-making capabilities. Organized in a tree-like hierarchy with varying levels of autonomy. Higher-level agents oversee and coordinate the activities of subordinate agents, ensuring that tasks are completed efficiently and in alignment with overarching goals. Agents operate on an equal footing, collaborating and negotiating with each other without a central authority. This structure promotes flexibility and adaptability, as agents can dynamically form alliances and adjust their roles based on the situation. Performs tasks autonomously within a limited scope, handling simple decision- making processes and adapting to minor environmental changes. Possesses sophisticated decision-making abilities, allowing it to adapt to dynamic environments, learn from experiences, and perform complex tasks without human intervention. This level of independence is still a subject of ongoing research and development. Basic autonomous Hierarchical structure Heterarchical structure Intermediate autonomous Advanced autonomous A single-agent system consists of a dynamic entity responsible for perceiving its environment, making decisions, and executing actions to achieve specific goals. This dynamic behavior of these agents can be classified into three tiers: Multi-agent systems (MAS) consist of multiple dynamic agents that interact and collaborate to achieve collective objectives. These systems can be structured in two primary ways: Single-Agent Architectures Multi-Agent Architectures Agent architectures can be categorized into single-agent and multi-agent systems, each with distinct structures and levels of autonomy. Typ es of Agents
6. 4 AI Applications AI agents are rapidly evolving and still in their early stages, yet they are already beginning to transform industries by streamlining operations, enhancing user experiences, and driving better outcomes. Some of the key areas where AI agents are making an impact include: Coding agents: AI-powered coding agents, like Cursor and Copilot, assist with code generation, debugging, and optimization. They provide real-time suggestions, automate repetitive tasks, and enhance developer productivity by reducing errors and speeding up development. Personal assistants: Voice-activated AI agents, such as Google Assistant and Alexa, are widely used for daily tasks and smart home controls. Customer support: AI-powered chatbots and virtual assistants are revolutionizing customer service, providing 24/7 assistance, handling routine queries, and resolving issues swiftly, thus enhancing customer satisfaction. Travel agents: AI-powered virtual assistants that enhance travel by providing personalized recommendations, itinerary planning, reservations, and real-time updates. Key Reasons for Measuring Quality Evaluating the quality of AI agents is not just about ensuring they function—it's about maximizing their effectiveness in delivering value to users and organizations alike. Here are some key reasons why measuring agent quality is a priority: The primary goal is to ensure the AI agent effectively helps users complete their intended tasks, prioritizing real-world success over isolated accuracy metrics. High-quality agents provide smooth, fast, and accurate interactions, boosting satisfaction and retention, while poor agents frustrate users and drive them away. Business impact: Efficient AI agents improve key metrics like response times, resolution rates, and cost savings, directly benefiting business performance. Scalability: Well-designed agents can handle growing user demand without compromising service quality, enabling businesses to scale efficiently. Long-term viability: Regular evaluation ensures AI agents remain effective, especially in high-stakes sectors like healthcare and BFSI, where errors can be costly. Task completion: User experience:
7. 5 Despite the obvious benefits of agent evaluation, there are several challenges that organizations face in ensuring the consistent quality of their agents: Different users have different interaction styles, requiring the agent to adapt dynamically to meet varied needs. User-specific variations: Real-world complexity: AI agents must function in unpredictable environments, handling diverse user queries, expectations, and contexts. For example, in customer support, an agent may need to handle queries from users with different backgrounds, expectations, and contexts. Evaluating an agent’s performance across such varied scenarios can be complex. Long-term adaptability: Performance evolves as agents interact with users and collect data, making it difficult to assess sustained effectiveness. Non-deterministic, dynamic systems: AI agents exhibit non-deterministic behavior due to their reliance on large language models (LLMs). This means that even with identical inputs, an agent’s decision-making process may produce different results each time. Evaluating performance in such probabilistic systems is difficult because the agent may perform well in some cases and fail in others, depending on the specific conditions it encounters. ⚠️ Unpredictable failure modes: AI agents can fail in unexpected ways, often only discovered in real-world deployment, necessitating ongoing monitoring and improvements. These challenges make it clear that evaluating the quality of agentic systems is far from straightforward. Ensuring that an agent can handle the variety, unpredictability, and complexity of real-world interactions requires rigorous, ongoing testing and refinement.

 The real-world impact of low-quality agentic systems is undeniable. Poorly designed or underperforming agents can erode customer trust, escalate operational costs, and significantly damage a brand’s reputation. The stakes are even higher in industries like healthcare, finance, and law, where the risks of error can be catastrophic. Therefore, businesses must prioritize the evaluation, testing, and ongoing refinement of AI agents to ensure they consistently meet both user expectations and business goals.

 As we move forward, measuring the quality of agents at every stage—from development to post-release—will be key to maintaining high standards and driving long-term success. In the upcoming sections of this document, we will explore the metrics necessary to evaluate agentic workflows and ensure that AI systems deliver the best outcomes in real-world scenarios. Common Challenges in Evaluating Agent Quality
8. 6 As AI agents start to gain traction across industries, driving innovation in tasks ranging from customer support to automation of tasks like booking requires their real-world performance evaluation to go beyond static benchmarks. Evaluating agents helps assess their decision-making process, adaptability, and goal-directed behavior in dynamic environments. This requires moving from single-turn responses to multi-turn evaluations, which necessitate different metrics to understand effectiveness. Why Traditional Metrics aren't Enough Modern AI agents handle complex, multi-step tasks that require planning, tool use, reflection, and adaptation. We need metrics that capture this complexity. Consider a restaurant booking agent - while it might show a 95% success rate in making reservations, this metric fails to capture whether it can adapt when complications arise. When a requested time slot is unavailable, a reliable agent should explore alternatives like nearby time slots rather than simply reporting failure. This demonstrates how traditional success metrics can mask the agent's ability to navigate real-world complexity. Thus, agent evaluation metrics need to be more dynamic in nature, capturing the challenges of complex environments.  Let’s discuss some system-specific metrics that one should consider to better understand their agent. Figure 1: Components of agentic evaluations Metrics for Evaluating Agentic Workflows
9. 7 Metrics for Agent Evaluation System Efficiency Metrics Figure 2: System efficiency metrics The first set of metrics helps us understand if our AI agent is operating efficiently. These metrics help assess resource utilization in terms of tokens and tool usage.

 Total completion time helps to better understand how long each part of the process takes. When an agent spends three minutes on a task, we need to know if it was stuck in a loop for two of those minutes or making steady progress.

 Every API call and token processed quickly accumulates in an agentic setup, impacting efficiency and cost. Effective agents minimize costs while maximizing value. Hence, metrics like Task token usage, Number of tool calls helps track the task efficiency. This will help quickly identify whether the agent is solving tasks in a cost- optimal manner. Such metrics also help to pinpoint the pitfalls and iterate on them quickly. Agent Quality Metrics These metrics evaluate the effectiveness of the agent in solving tasks, the methods used, and the encountered failures. These metrics can be broadly classified into overall agent evaluation and their component evaluation: Session Level Evaluation Figure 3: Session-level evaluation Figure 2: Session-level evaluation
10. 8 Step completion: This is where we get granular but in an interesting way. If the user has a predefined approach to solving a task, this metric evaluates if the agent also conforms to the expected steps without deviating much to reach the goal. This evaluation metric can help assess if all the expected steps were executed correctly. Agent trajectory: There may be multiple trajectories to reach a goal. This metric assesses whether the agent follows a reasonable and effective path to solve the user query. What we can be looking for: 
 Smart choices: Does the agent pick the right tool for the job? Adaptability: Can it handle unexpected situations? Think of a GPS recalculating when you take a wrong turn.
 This metric can also be thought of as another way of evaluating the plan being followed by an agent inspired by this research paper.

 Self-aware failure rate: This metric helps to measure failures where the agent is aware of its limitations in solving a task which could be ascertained with messages such as “I am unable to do this task due to <xyz> reason” or “I have hit the rate limit errors”. These failures could be due to a lack of capability of the agent or due to the agent looping on the same step again failing at it. In addition to the above agent-specific metrics, we can extend the capabilities of existing single-turn metrics using LLM as a judge to measure the bias or toxicity in the agent outputs.  Node Level Evaluation Figure 4: Node level evaluation Task success: This metric determines whether the agent successfully achieves the user’s goal based on its session output. It helps to measure if the agent, over its multiple steps, despite all its adaptations, can successfully reach the intended target to solve the task it set out to achieve.
11. 9 Tool selection: It is imperative to check if the agent is calling the correct tool to solve the task in that step, along with passing the adequate input parameters for the tool call to be implemented. This metric helps to ensure that there was no part of the agent in the tool call failure.  Tool call error rate: To ensure that future steps taken by the agent are not affected, one can check if the tool being called by the agent is giving an output. This evaluation helps recognize some steps, in the tool call pipeline being the problem to rectify. Tool call accuracy: Finally, the quality of the tool call needs to be verified, for which the output obtained by the agent’s tool call, given the input query is evaluated. Given it is not possible to know the accurate output from the lens of evaluation, we can compare the output with an expected output to score the accuracy. Plan evaluation: Planning what steps to take is hard and can result in failures. Hence, it's important to evaluate an agent for planning failures. The key questions we aim to address are- will the plan help solve the task given constraints, or if there are errors in reflection due to which the planning is failing, or if there are tool failures that can happen in the plan generated. One can use LLM models to verify plans, as highlighted in this research paper.  Step utility: This metric helps to evaluate the number of contributing steps. The question to answer via the metric is whether the step is helpful, harmful, or neutral in the context of the overall objective? Did it move the task forward, did it create obstacles that hindered progress, or was it inconsequential? Now, let’s get into the nitty-gritty of the agentic evaluation. These metrics help to dive deeper to assess the performance of the agent's planning, tool use, and steps:  Tool use metrics: While dealing with agents, tool use is an essential part of helping improve the performance of agents. However, it is essential to evaluate the tool used to ensure that the agent calls the right tools and gets the right outputs from the tools to better understand their functioning. Examples of Real-World Agents To better understand what we discussed above, let’s look at a few examples of real- world agent evaluations: Travel Agent  If we gave the following scenario: "Book a round trip from London to San Francisco on the cheapest dates in March for CEO of a company"
12. 10 Turn User Response 1 Please find and book a round trip from London to San Francisco on the cheapest available dates in March for the CEO of a company. I have validated the details for the airports in London and San Francisco. Please confirm 

 - London as the departure city.
 - San Francisco as the destination city.
 - Round trip.
 - Cheapest dates in March.
 - Passenger type: CEO (adults: 1).
 - Cabin Preference: Economy (default assumed).

 Shall I proceed with the flight search? 2 Yes, please proceed with the flight search based on the confirmed details. I've searched for the cheapest round-trip flight options between London (LON) and San Francisco (SFO) in March 2025. Here are a few options for your review.

 Please select a flight, and I will proceed to retrieve detailed information for confirmation before booking. 3 Please provide me with the details of the cheapest available flight option on the selected dates in March. The cheapest round-trip flight details have been retrieved for London (LON) to San Francisco (SFO) in March.

 Review the details carefully and confirm if you'd like me to proceed with the booking or need further adjustments. 4 Yes, please proceed with the booking of the cheapest round-trip flight option for March. To proceed with the booking of the cheapest round-trip flight, I will require the following passenger information for the CEO:

 1. Full Name of the passenger.
 2. Email Address for the booking confirmation.
 3. Contact Phone Number.

 Please provide these details to finalize the booking.
13. 11 Turn User Response 5 Here is the passenger information needed for booking:

 1. Full Name: John Doe

 2. Email Address: 3. Contact Phone Number: +44 1234 567890

 Please proceed with finalizing the booking. [email protected]

 The round-trip flight from London to San Francisco for John Doe in March has been successfully booked. A confirmation email has been sent to . If you need further assistance or have additional requests, feel free to let me know! Safe travels. [email protected] As we can see above, we can use Maxim AI’s simulation agent to query a target travel agent to book a flight for a customer. To evaluate this agent, let's use a few metrics that we discussed till now: This metric helps us understand that the agent did the right steps to reach its target of booking the flight seeing only the trajectory of the agent. This metric helps us understand that the agent did the expected steps defined by the user while booking the flight. Figure 5: Agent trajectory evaluator result for Travel Agent Figure 6: Step completion evaluator result for Travel Agent
14. 12 This metric helps us understand that the agent was successful in completing the task, fulfilling all the user requirements in booking it. Customer Service Agent If we gave the following scenario: "Output the current address of the person with email id [email protected] and then change their address to BHIVE, Indiranagar in the database" Figure 7: Task success evaluator result for Travel Agent Turn User Response 1 Get the current address of the person with the email id [email protected] The current address of the person with the email id [email protected] is: 7375 Grayhawk Avenue, Waipawa, New Zealand, 5782. 2 Change the address of the person with email id [email protected] to 'BHIVE, Indiranagar' in the database The address for the person with the email id [email protected] has been successfully updated to \ "BHIVE, Indiranagar\ " in the database. Figure 8: Agent traj ectory evaluator result for customer service agent
15. 13 Figure 9: Step completion and task success evaluator result for Customer Service Agent By combining system-specific metrics and metrics to evaluate agent performance, this framework provides a structured approach to evaluating efficiency, accuracy, and goal achievement, leading to more reliable AI agents. In the next chapter we will learn how to build a structured AI evaluation process encompassing both pre-release and post-release phases for developing robust and reliable agents. For a successful customer service agent as well, we can refer to the above metrics via Maxim to holistically evaluate our agent and ensure it's working in an optimal manner.  Agentic evaluation shifts AI assessment from static benchmarks to dynamic, multi- turn interactions, ensuring a more accurate measure of decision-making and task completion.
16. 14 Building Robust Evaluation Workflows for AI Agents In the previous chapters, we explored AI agents and the key performance metrics for evaluating them. Now, we focus on building end-to-end evaluation workflows. A structured AI evaluation process encompassing both pre-release and post-release phases is crucial for developing robust and reliable agents. This chapter discusses best practices for systematically evaluating agents, including simulation-based pre- release testing, real-world post-release monitoring, and continuous refinement strategies for building effective agentic systems. Before an AI agent is deployed, comprehensive pre-release testing is required to validate its functionality, adaptability, and performance across various scenarios. This phase of pre-release evaluation mitigates risks, minimizes failures, and enhances user trust in the system. Pre-Release Evaluations (aka Offline Evaluations) Using Simulation for Pre-Release AI Agent Evaluation Simulation-based testing allows developers to assess agents in controlled environments before real-world deployment. This simulation testing approach is essential for: Evaluating agent behavior and performance across diverse real-world scenarios. Identifying edge cases and potential failure modes. Testing adaptability across different user personas. For example, interactions with a customer support AI agent can be simulated varying in complexity, sentiment, and urgency and then tested using different evaluators. These evaluations help ensure the agent’s responses are both relevant and aligned with simulated user preferences.
17. 15 Example of interaction with a customer support agent Figure 10: Example of interaction with a customer support agent
18. 16 Evaluating Individual Nodes in the Workflow Using Test Sets Agents often navigate a series of sub-tasks, each requiring individual decisions along the way. Evaluating each node independently is crucial for understanding where breakdowns may occur. For this evaluation, it is essential to build datasets that reflect possible user scenarios and attach the respective node evaluators in a test run to better understand how the agent navigates the scenario. System-specific evaluation metrics ensure a more precise assessment of agent performance. Metrics such as task success, step completion, and agent trajectory help verify that each workflow component contributes meaningfully to the overall objective. Self-aware failure rate can highlight areas where the AI recognizes and responds appropriately to limitations, ensuring that breakdowns are handled gracefully. Plan evaluation and step utility further ensure that the AI’s decision- making process is both logically sound and heading in the right direction to reach the eventual goal. For instance, in a travel booking agent, individual nodes such as flight search, hotel selection, and itinerary generation should be tested independently using these evaluation metrics to ensure reliability and accuracy before deployment. Ensuring that each step contributes effectively to overall goal achievement prevents inefficiencies and improves system robustness. Figure 11: Example showing pre-release evaluation metrics applied to a session of an AI agent simulation.
19. 17 Incorporating Human Feedback for AI Agent Refinement Human evaluation remains one of the most effective ways to fine-tune agentic behavior. Gathering expert and user feedback during pre-release testing helps: Validate output correctness and coherence of the AI agent. Identify biases and inconsistencies in decision-making. Improve user experience through iterative refinements based on human feedback. Feedback can be collected via: Human-in-the-loop testing: Domain experts review agent decisions. Crowdsourced evaluation: A diverse set of users interact with the system. Direct annotation: Users provide corrections to agent responses. Post-Release Evaluations (aka Online evaluations) Once an AI agent is deployed, continuous post-release monitoring and iterative improvements are crucial for maintaining high agent performance and adaptability to evolving user requirements. Post-Release Monitoring with Logs (Sessions, Traces, and Spans) Logging real-world agent interactions provides insights into performance and areas for improvement. Key monitoring logs elements include: Sessions: A top-level entity capturing multi-turn AI agent interactions. Traces: Complete processing of a request through a distributed system, covering all actions between request and response. Spans: A logical unit of work within a trace, representing tagged time intervals. B y analyz ing these logs, teams can identify bottlenecks, failure points, and areas for optimiz ation in the agentic performance. T he advantage of logging agents constantly is that pre-release metrics can be applied to continue analyz ing agent performance on real customer data.
20. Session-level metrics: Task completion rates, agent trajectory success, resolution times, and user satisfaction scores.
 Node-level metrics: Tool use metrics (e.g., tool call error rate), programmatic evaluators (e.g., isValidEmail()), and other quality metrics (e.g., bias, toxicity). Many node-level metrics can also be applied to the overall session as needed. These metrics align with those introduced in chapter 2, ensuring a structured evaluation process. Evaluating Session and Node Level Performance Post-release evaluations should assess both overall session performance and individual decision-making nodes using relevant performance metrics: Data Annotation and Curation for Continuous Improvement AI Real-world interactions provide valuable data for refining AI models and agent performance. Systematic data annotation helps: Identify failure patterns to improve agent output quality. Improve tool usage by the agents. Reduce biases, toxicity, and personal information exposure while improving clarity and other quality metrics in decision-making. A structured annotation pipeline ensures continuous learning and refinement for the AI agent. Integrating Pre- and Post-Release Feedback Loops A robust evaluation workflow integrates insights from both pre-release evaluation and post-release monitoring phases. This ensures that: Pre-release testing incorporates real-world learnings and user simulations. Post-release improvements leverage real-world user usage data. The agent evolves effectively to meet changing user needs and business goals through this feedback loop. Best Practices for Integrating Evaluation Feedback Loops AI agent evaluation plays a crucial role in creating an effective feedback loop. By leveraging evaluation, teams can systematically analyze agent decision-making, response accuracy, and adaptability over time. This evaluation-driven approach enables: 18
21. 19 Automated benchmarking: Continuous measurement of agent performance metrics across different scenarios, ensuring the agent remains effective. Adversarial testing: Identifying weak points in the agent’s reasoning by exposing it to challenging edge cases. Scalable AI evaluation: Automating feedback collection and performance assessment, reducing reliance on manual human evaluation while maintaining quality. By incorporating evaluation-driven feedback loops, AI agents can evolve more efficiently, ensuring alignment with user expectations.
22. 20 Conclusion As AI agents grow in complexity and capability, the challenge of evaluating their performance, reliability, and real-world utility becomes more critical and nuanced. Through this three-chapter series, we explored the foundations of AI agent quality evaluation, defined essential evaluation metrics, and walked through practical workflows that development teams can use to evaluate and improve their agents across diverse real-world scenarios. The key takeaway is clear: traditional testing methods fall short when it comes to the dynamic, open-ended behaviors of modern agents. What’s needed is a systematic, continuous, and scenario-aware approach to evaluation, one that not only identifies gaps and failures but also drives iterative improvement. Key takeaways:
 Evaluation is multidimensional: A single metric is rarely enough. Useful evaluations consider multiple factors. These factors include human and auto- evals, node-level to overall session-level evals, and system-efficiency to output quality. Context matters: Agent behavior must be assessed in realistic, task-specific, and user-representative scenarios, not just in sandbox tests. Evaluation is ongoing: It's not a one-time checklist. Continuous monitoring and iterative testing are essential to building agents that win user love. Evals are becoming the backbone of high-quality AI products: Building great agents in today’s world requires effectively balancing product quality and shipping speed. A well-structured eval process, both offline and online, helps teams move much faster while keeping taste at the heart of their products. That’s exactly what we’re building at Maxim. Maxim is the agent simulation, evaluation, and observability platform empowering development teams -- from start-ups to enterprises -- to ship AI applications with the quality and speed required for real-world use. Whether you're building a simple GenAI chatbot or orchestrating a multi-agent system, Maxim helps you test before release, monitor post-deployment, and improve over time, so your agents deliver differentiated value and earn user trust. Need help with evals for your AI products? Talk to us or try out for yourself at evals.new.