Agents with Prompty and Azure AI Foundry

An exciting session on building intelligent multi-agent AI applications using Prompty and Azure AI Foundry. We'll start with an overview of Azure AI Foundry and Prompty, and explore how agent systems are designed, and deployed.

Next, I’ll guide you through a hands-on project demo showing these tools and components in action. You’ll see how agents use knowledge and available tools to perform actions based on user query. Along the way, I’ll share practical tips on handling tricky error scenarios, debugging, and evaluating your AI systems effectively.

Transcript

1. Agents with Prompty and Azure AI Foundry Bethany Jepchumba Cloud Advocate, Microsoft @bethanyjep
2. Agentic AI
3. AI Agents Unlike existing automation, AI Agents can automate requests as soon as they come in by… Reasoning over the request Proactively retrieving more context Performing required actions with APIs
4. What are agents? AI designed to perform a task Tasks can vary in level of complexity and capabilities depending on your need Simple Advanced Generation Generate summaries, images, audio, and more with an AI model and inputs. Retrieval Retrieve information from grounding data, reason, summarize, and answer user questions Action Take actions to automate workflows, and replace repetitive tasks for users
5. With AI agents, you can make your processes faster than ever VALUE Yesterday – manual Forms over data Other apps HR manager New account New employee form Automation HR manager New employee API Automation Today – RAG Chatbots Tomorrow Automated agents Approve? New account Agent HR manager New account New employee form Automation Chatbot Improve employee productivity Improve process efficiency
6. Agentic AI capabilities Agent Planning Memory Tools Action Web Search Code Execution Calculator Calendar ...more ReAct Self-critique Collaboration Subgoal decomposition ...many more design patterns
7. Common Design Patterns RAG Agent Code Generation Agent Multi-agent Systems Multi-domain Agent Systems
8. Retrieval Augmented Generation Agent Search Tools discussion D Intelligent RAG Agent Plan Query Observe Update plan Compile answer Knowledge Graph Translates questions into a research problem with human in the loop to produce high quality answers to complex questions within the scope of its domain question final answer
9. Session and memory management Dynamic context look-up Planning & tracking Toolsets & coding interface Human interaction Coordinator Multi-turn reasoning and action (ReAct) Code Generation Agent Generates code based on natural language requirements, leveraging existing code base, templates, guidelines, libraries to match policies and best practices while interacting with humans to clarify, validate and deliver functionality as intended. Code Executor Constraints Existing Codebase Coding Guidelines (docs) Internal SDKs Dev Task (spec / bug / feature) Code w/ Tests, Doc, DevOps code, etc
10. MS #3 MS #2 MS #1 Multi-Agent System A complex problem is decomposed into smaller, manageable parts, each addressed by specialized agents, effectively a micro-service (MS). These agents work together in a coordinated manner within a workflow to efficiently solve the overall problem. conversation data query docker Coding Guidelines history of work Critical Design Elements Adaptive planning within scope of existing tightly scoped skills (agents) Handles ambiguity by discussing and refining requirements with human Memory to handle complex long running execution of a plan Effective inter agent communications Test, monitor, release & maintain each agent independently to quickly handle quality & safety issues
11. Multi-Domain Agents System Multiple domain-specific agents are orchestrated by an Agent Runner to scale across multiple domains while appearing as a single agent to users. Agent pool Agent Runner Active Agent Agent 1 Revaluate agent assignment Run Transfer Run + Back-off Role/goals + skills Agent 2 Run + Back-off Role/goals + skills Agent n Run + Back-off Role/goals + skills Shared context memory Critical Design Elements Agents capability descriptors Scalable Agent Runner able to manage 10s to 100s of agents Ability to manage domain switching with proper memory management Avoid single interceptor problem as individual agents maintain direct communication with user and can hand off when needed
12. How does your agent work? User Travel Booking Agent “Help me book a trip to New York for a client meeting? I need to fly out next Monday and return on Friday.” Knowledge Sources (search, files, databases, storage etc.) Models (Azure OpenAI Service, Models-as-a- Service) Actions (Pre-built or custom tools to automate processes) “I’ve booked your trip to New York as requested. Here are details:…”
13. How does your agent work? Step 1: Create an Agent Step 2: Create a Thread Step 3: Run the Agent Step 5: Check the Run status Step 6: Display the Agent’s Response Agent Travel Planning Agent Instructions You are a travel booking and expense management assistant designed to help employees plan, book, and manage business travel. Run 2 Model Tools (optional) File Search Code Interpreter Function Calling Bing Search Microsoft SharePoint (coming soon) Microsoft Fabric (coming soon) Azure Logic Apps (coming soon) Azure Functions OpenAPI 3.0 specified tools User’s message I need to book a hotel in New York for 2 stays. Agent’s message Here are some suggestions: Run 1 1 Use Tripadvisor API to search the nearest hotel Create message 2 Your data (optional) User’s message What’s the daily meal allowance for the business trip? Agent’s message The daily allowance for your business trip is $75, as per company policy. Use Microsoft SharePoint to query the company travel policy Create message 2 1 Thread Travel Planning Azure AI Search Files (local or Azure Blob)
14. Model Catalog GitHub Models Use GitHub Models to find and experiment with AI models for free. Once you are ready to bring your application to production, you can switch to a token from a paid Azure account. Azure AI Foundry Model Catalog Discover and use a wide range of models for building generative AI applications. The model catalog features hundreds of models across model providers such as Azure OpenAI Service, Mistral, Meta, Cohere, NVIDIA, and Hugging Face, including models that Microsoft trained.
15. Prompty as a language-agnostic prompt asset Easy to start Code first Intuitive to develop Multiple programming language will be supported Simplification deployment Agency with Observability Choose the framework
16. Scenario A creative writing assistant that will help write well researched, product specific articles for Contoso Outdoor Company.
17. Multi-agent creative writing assistant Processing services User input Azure Monitor Researcher Agent Azure OpenAI Service Azure OpenAI Service Azure OpenAI Service Grounding with Bing Azure AI Search Azure Managed Identity Azure Application Container (ACA) Response Azure AI Foundry Product Agent Writer Agent Editor Agent Azure AI Search Azure OpenAI Service https://aka.ms/ai-apps
18. Tracing and debugging with Prompty benefits Provides a detailed view of the execution flow of the application Support both local and cloud runs Detail every LLM call including message, prompt, parameters, token, and latency Collect & attach user feedback using OpenTelemetry's semantic conventions
19. AI Agent workflows can enhance efficiency, accuracy, and customer satisfaction Key Use Cases Across Industries • Assists employees in booking business trips • Integrates with Tripadvisor, Outlook, and SharePoint • Books via Teams chat or email • Uses OCR to gather receipts • Automates expense report submission and tracking Travel Booking & Expense Management • Personalized onboarding assistant for new hires • Uses LLMs grounded in HR data from SharePoint • Provide relevant training materials • Schedule orientations and set up software accounts • Monitor task completion and ensure efficient onboarding Employee Onboarding • Diagnoses issues by referencing history and product manuals • Provides tailored solutions or escalates through automated workflows • Creates tickets and schedules follow-ups • Updates CRM records, enhancing future support Personalized Customer Support • Analytics data from data lake and data warehouse • Responds to user requests in natural language • Generates insights, visualization, and sends via Teams or email • Automates data handling for real-time, effortless decision-making Data Analytics and Reporting
20. Thank you Resources https://aka.ms/GenAI/LLMOps https://aka.ms/ai-apps https://aka.ms/azureai/discord Check out the workshop! Build your first agent with Azure AI Agent Service