Evaluating_MCP_Based_AI_systems.pdf

Transcript

1. Welcome
2. It worked perfectly… until it didn’t. SYSTEM WE BUILT User Question MCP Agent (multi-step reasoning) Answer SIMPLE PROMPT TEST Prompt → Model → Output Works perfectly REAL MCP SYSTEM User → MCP Agent → Answer ✘ Fails in production
3. W O M E N W H O C O D E • N E W Y O R K Prompt to Production: Evaluating MCP-Based AI Systems Priti Pathak• Senior Software Engineer • Microsoft
4. Quick Pulse Check Familiar with MCP? Raise your hand! Used MCP in production? Keep that hand up! This shapes how deep we go on the basics today
5. Life Before MCP: A Fragmented Mess • Manual, custom integrations • Platform silos & duplicated effort • High maintenance & poor scalability • Limited capability (read > act)
6. MCP Architecture ✔ One standard protocol ✔ Host manages tool discovery ✔ Servers expose tools + resources ✔ Context flows clearly at every step H O S T A P P L I C A T I O N AI Model + MCP Client M C P P R O T O C O L ( J S O N - R P C o v e r s t d i o / H T T P + S S E ) MCP SERVER Database Tools + Resources MCP SERVER External API Tools + Resources MCP SERVER Local Files Tools + Resources Each MCP Server exposes a consistent interface — one protocol, many integrations
7. MCP: Strengths & Gaps • Solves: • Standard interfaces • Clear context exchange • Does NOT solve (Still your responsibility) : • Decision accuracy • System reliability • Evaluation complexity
8. How MCP Systems Work User Context Model Tool Updated Context Output
9. Where is “Correctness”? Answer: ALL OF THESE User Context Model Tool Updated Context Output C H E C K A T E A C H S T E P Context Relevant? Complete? Decision Right intent mapped? Tool Execution Correct params? Final Output Answers the intent? → Correctness lives at ALL of these checkpoints — not just the final answer
10. Our Project Data MCP Agent User Question MCP Agent (multi-step reasoning) Answer EXAMPLE QUERY “How many bugs were fixed in the last sprint?” TEST RESULTS In testing… everything passed ✓ Intent correctly identified ✓ Correct data source queried ✓ Valid response returned
11. Then we asked this… “How many issues were resolved last sprint?” Outcome: Agent interpreted “issues resolved” as all closed tickets — not just bugs — and returned the wrong count Query executed Data valid Intent misunderstood — “issues” ≠ “bugs”
12. What We Thought We Were Testing ✓ Did the agent pick the right data source? ✓ Did the query execute without errors? ✓ Is the response formatted correctly? ✓ Does the number look plausible? All of these passed.
13. We Weren’t Evaluating the Right Thing “Are we answering the user’s intent correctly?”
14. Evaluate MCP Systems in Layers Component (LLM/ Tool/ Retrieval) Workflow (Multi-step tasks) System (End-to-end experience) “Each layer fails differently — so we must evaluate them differently.”
15. Component-Level Evaluation MODEL / INTENT MAPPING Paraphrase robustness Can the system handle different phrasings of the same question? • Build synonym coverage • Add normalisation step before intent classification • Use LLM-as-judge for intent alignment “This is where ‘issues resolved’ vs ‘bugs fixed’ should’ve been caught.”
16. Workflow-Level Evaluation AGENT BEHAVIOR Multi-step correctness Does each step in the chain produce the right output for the next? Ambiguity handling Does the system ask for clarification when the query is vague? “Should the system have asked a clarification question?”
17. Single-Turn vs Multi-Turn MCP Evaluation Single-Turn Evaluation BEHAVIOR • Direct retrieval-style response • Minimal interpretation EVALUATION • Compare to ground truth • Binary: correct vs incorrect Multi-Turn / Agentic Evaluation BEHAVIOR • Interprets intent (“bugs fixed” vs “issues resolved”) • May expand scope (Scope 1, 2, 3) • Multi-step reasoning / retrieval EVALUATION • Was the interpretation correct? • Were intermediate steps valid? • Is the final answer justified?
18. A Better Framework: Hybrid Evaluation Hybrid evaluation combining automated testing, human review, and continuous monitoring provides comprehensive agent assessment Automated Scoring LLM-as-judge provides repeatability and scale for common patterns Trace analysis tracks tool calls, state changes, decision points Load testing validates performance under production concurrency Human Judgment Domain experts assess tone, trust, contextual appropriateness Rubrics calibrate automated judges on sample traces Captures what automation misses in nuanced interactions Continuous Feedback Real-time monitoring for customer-facing agents Batch assessment for internal tools Performance trend tracking and degradation detection
19. Core Evaluation Metrics L L M M E T R I C 1. Tool Selection F1 “Did the LLM pick the right MCP tool(s)?” L L M M E T R I C 2. Slot F1 “Did the LLM extract the right parameters?” S O F T W A R E M E T R I C 3. API Call Accuracy “Did the MCP server call the right APIs?” E N D - T O - E N D K P I 4. Execution Accuracy “Did we get the right data back?” Four core metrics capture functional correctness here; consider supplementing them with domain-specific measures. Security evaluation operates through a dedicated instrumentation layer.
20. End-to-End Evaluation Pipeline Automated workflow from versioned test data through scoring to regression-gated reporting 1 Eval Dataset versioned 2 Test Runner batch / CI trigger 3 MCP System Execution 4 Scoring Engine auto + LLM judges 5 Metrics Store history, trends 6 Regression Checks pass/fail gates 7 Report / Dashboard
21. Real-Time Evaluation Dashboard Code Review MCP Agent Run #47 · Sprint 24 · 5 queries 1. Tool Selection F1 LLM metric · P/R/F1 0.80 P: 0.83 | R: 0.78 | F1: 0.80 2. Slot F1 LLM metric · (key, value) pairs 0.72 P: 0.78 | R: 0.67 | F1: 0.72 3. API Call Accuracy Software metric · pass / fail 3 / 5 Pass: 3 | Fail: 2 | 60% rate 4. Execution Accuracy End-to-end KPI · vs baseline 0.60 3 correct | Below SLO 0.85 USER QUERY TOOL SELECTED TOOL SEL F1 SLOT F1 API CALL EXEC. ACC. “How many bugs were fixed last sprint?” bug_query 1.00 0.94 / 0.90 Pass 1.00 “How many issues were resolved last sprint?” bug_query 1.00 0.00 (wrong params) ✘ Fail 0.00 “What’s the open PR count for the backend team?” pr_query 1.00 0.95 / 0.92 Pass 1.00 “Show me deployment stats for last week” bug_query (wrong!) 0.00 n/a n/a 0.00 “How many PRs are awaiting review right now?” pr_query 1.00 1.00 / 1.00 Pass 1.00 Trend: Execution Accuracy stable at 0.85–0.90 for 7 runs, then dropped sharply to 0.60 on Run #47. Regression gate triggered. Two root causes: (1) Slot F1 failure on Row 2 — right tool, wrong parameter mapping; (2) Tool Selection failure on Row 4 — wrong tool chosen entirely. Fix: retrain intent classifier + add synonym coverage to slot extraction. DASHBOARD SIGNAL ROOT CAUSE FIX HERE Exec. Acc. ↓ + Tool Sel. F1 ↓ Wrong tool selected — all downstream n/a Refine tool definitions Exec. Acc. ↓ + Tool Sel. F1 ✓ + Slot F1 ↓ Right tool, wrong params Fix slot extraction + synonyms Exec. Acc. ↓ + Slot F1 ✓ + API Acc. ↓ Right params, wrong API called in server Fix MCP server business logic Exec. Acc. ↓ + All other metrics ✓ Data mismatch vs baseline Update ground truth
22. Security & Safety: The Elephant in the Room ATTACK VECTORS Tool-based prompt injection Malicious instructions hidden in tool outputs hijack agent behavior Server impersonation Rogue servers masquerading as trusted MCP endpoints Parameter poisoning Crafted inputs manipulate tool parameters for unintended actions Agent overtrust Agents blindly executing tool responses without validation K E Y I N S I G H T “Security in MCP systems is not a feature you add — it’s an architecture you commit to.” Each attack vector maps to an eval test case. Build them in from day one.
23. Safety & Security Metrics Harmful Content Rate Attack / Jailbreak Success Rate Sensitive Data Leakage Tool Misuse / Indirect Attacks
24. What Works in Practice ENGINEERING PATTERNS Log full MCP execution traces Validate tool calls structurally Treat tool descriptions as untrusted input Enforce explicit tool contracts Fail closed, not open “Good MCP evaluation is architectural, not ad hoc.”
25. The Future of MCP Evaluation What’s Coming METRICS • Behavior-based metrics over output-only scoring • Replayable agent traces for regression INFRASTRUCTURE • Cross-model robustness checks • Security benchmarks as default CI • Standardized “agent SLOs” THE SHIFT “In 2026, evaluating AI systems looks a lot like evaluating distributed systems.”
26. Key Takeaways MCP ≠ just another integration Behavior > output — Judging the process matters more than judging the final answer Tool Selection F1 is your canary – everything is conditional on it Safety is a design-time concern — Reliability and security must be architected in, not patched on
27. Q & A Thank you for joining — let’s keep the conversation going! Priti Pathak Microsoft KEEP LEARNING MCP Specification modelcontextprotocol.io https://github.com/modelcontextprotocol/mode lcontextprotocol Slides & Resources your-link-here.com Women Who Code womenwhocode.com