Building AI Agents with Azure AI Foundry and AI Toolkit

The GitHub Copilot & AI Toolkit Pet Planner Workshop — creating a smart, friendly agent that plans pet playdates using data and AI reasoning. Learning how to design, configure, generate, trace, and evaluate an AI-powered workflow.

> Codebase: https://github.com/BethanyJep/AI_Toolkit_Samples

Transcript

1. Workshop: Building AI Agents Bethany Jepchumba | @bethanyjep
2. • Cloud Advocate (DevRel) at Microsoft • Works with NGOs & Communities, currently ICT Director at TOFA • I am a hobbist, I enjoy trying out new hobbies Today’s session: https://bethany-jep.com/events/agentcon-accra-2025/ Bethany Jepchumba Microsoft confidential 2
3. Question: Have you built an agent before?
4. The concept of LLMs…. 4
5. How language models work Natural language input Model Tokens Probability distribution Natural language output Decoding + Post-processing Get results Pre-processing
6. RAG, Agents and MCP Giving your LLMs some powers, sort of…. 6
7. BYOD -> Retrieval-Augmented Generation (RAG)
8. What is an AI agent? LLM Instructions Tools Agent + + An AI agent is a micro-service that takes unstructured messages, optionally invokes other APIs and returns messages/action 1 2 3 Input System events User messages Agent messages 1 Tool calls Knowledge Actions Memory 2 Output Agent messages Tool results 3
9. Unlike previous tools, AI Agent can… reason, act and learn
10. What is Model Context Protocol (MCP)? Ship once – reach every MCP-enabled client in your organization A secure, open standard that lets AI agents access tools, APIs, and data through a consistent interface — without custom integrations. In Azure AI Foundry Agent Service, MCP connects agents to your enterprise and external systems with a single, reusable protocol — secured, governed, and observable by Azure. Clients • Foundry Agents • Copilot Studio • GitHub Copilot • VS Code Servers/ Tools • Azure Services (Search, Cosmos DB, Power Platform, etc.) • Third-party APIs (Salesforce, ServiceNow, REST APIs) • On-prem data MCP Protocol • One standard interface • Security: Entra ID • Observability: Azure Monitor
11. MCP Model Context Protocol - Easier to give context to models - Optimized communication between LLMs, external tools, data sources and applications - Uses a client-server model for interactions
12. Model Context Protocol At Microsoft Model Context Protocol MCP enables seamless integration between LLM Apps and external data sources Azure API Management Build Local & Remote MCPs MCP SDK Microsoft Connectors Azure Functions Tools Etc. … Community Windows Registry VS Code BYO Enterprise or MCP Clients & Host Copilot Studio Foundry agents Visual Studio Windows VS Code GitHub Copilot Entra ID
13. Enhancing Security for MCP Servers Use credential manager to authorize access to your backend MCP servers Protect your remote MCP servers with OAuth 2.0 and API Management Blog aka.ms/remote-mcp-apim-auth-blog Generally Available Azure API Management Safety Policies Azure OpenAI MCP enabled Tools Azure Monitor Logs & Metrics … Credential Manager Agents …
14. AI development natively embedded in your VS Code workflow AI Toolkit Extension
15. MCP Servers AI Toolkit
16. GitHub Copilot Agent Mode Develop alongside AI agents that can automatically review and update your code
17. https://bethany-jep.com/events/alx-workshop/ Let’s build an AI Agent
18. Security, monitoring and evaluations. Agents in production…. 19
19. Generative AI Development Lifecycle Hypo thesis Find LLMs Try prompts Manage Customize Design Managing PREPARE FOR APP DEPLOYMENT ADVANCE PROJECT BUSINESS NEED Deploy LLM App/UI Quo ta and cost management REVERT PROJECT C ontent Filtering M onitoring SEND FEEDBACK Prompt Engine ering or Fine-tuning Evalua tion Exceptio n Handling Retrieval Augmente d Generation Prote ct & govern
20. How do you typically test if a model or application is working as intended?
21. The value of evaluating AI systems Quality assurance Performance metrics Bias detection User trust
22. Mitigation layers Application Platform
23. • Violence • Sexual • Self-harm • Hate and Unfairness • Ungrounded Attributes • Code Vulnerability • Protected Materials • Content Safety Evaluators in Azure AI Foundry Safety & Security • Fluency • QA • Coherence • Intent Resolution • Task Adherence • Tool call accuracy General Purpose • Retrieval • Document Retrieval • Groundedness • Relevance • Response Completeness RAG Agents
24. Evaluate agentic workflows User query “Weather tomorrow.” Intent resolution User wants to know the local weather and the time to forecast. Tool calls Call location and time API Call weather API Final response “The temperature will be 30 degrees. Rain will…” Intent resolution evaluation • User intent identification • Clarification for ambiguity Tool call accuracy evaluation • Correct tool call selection • Correct parameter extraction Final response evaluation • Task adherence • Response completeness Quality and Safety evaluation • RAG quality (Groundedness/Relevance/Retrieval) • Risk and Safety (Jailbreak/Code Vulnerabilities/Ungrounded Attributes
25. Tips for meaningful agent evaluation Change one variable at a time Use identical prompts for fair tests Score with a consistent rubric Keep a changelog of changes Note trade-offs (e.g., speed vs. quality) Learn why a version wins
26. Production Ready
27. Azure AI Foundry Foundry Models Foundry Agents Azure AI Search Azure AI Services Azure Machine Learning Azure AI Content Safety Foundry Observability Security • Identity • Management Copilot Studio Visual Studio GitHub Foundry SDK Cloud Azure Azure Arc Foundry Local Edge Serverless Control Azure Kubernetes Service Azure Container Apps Azure App Service Azure Functions
28. Foundry Observability Dashboard Azure AI Foundry Observability, integrated with Azure Monitor Application Insights, enables you to continuously monitor your AI applications
29. Rapidly prototype with GitHub-hosted models using the AI Toolkit Design and evaluate agents in VS Code using Agent Builder Use GitHub Copilot Agent Mode to integrate your agent code to an existing application Deploy to Azure AI Foundry for scale, security, and observability From idea to impact In this session, you saw how to: 4 3 2 1 What we’ve built in VS Code… powers production app.
30. Your tools are ready. What will you build? Access this session’s resources https://bethany-jep.com/events/alx-workshop/
31. Thank You