Build enterprise- grade AI agents with Azure AI Agent Service SATO Naoki (Neo) Principal Software Engineer Microsoft Corporation

Every app will be reinvented with AI New apps will be built that weren’t possible before

Copilot & AI stack Copilot devices Copilot

Visual Studio + Copilot Studio GitHub + Azure Cloud to Edge Azure AI Foundry Data Infrastructure Trustworthy AI Copilot & AI stack

Agentic AI A new frontier:

Agentic AI use AI to automate and execute business processes, nearly autonomously AI agents can plan, adapt to new information, and execute tasks independently, making them capable of handling complex, dynamic environments, decision-making and autonomous action. They can learn from feedback, adjust their strategies, and operate with minimal human oversight. Reason over a provided business process Retrieve context to complete the process Perform an action for the end-user

K Inefficiency and High Operational Costs Organizations face significant challenges with complex, manual, and time-consuming workflows and minimal automation Lack of Visibility & Control Human Error Scalability Issues

Promising even more efficiency, value, and advantage VALUE Yesterday Today User Chatbot Able to answer questions “What is the dress code in the office?” User Single AI Agent Able to take action “Order a laptop for a new employee” Human Supervisor User HR Agent Able to collaboratively solve complex tasks “Onboard 5 employees by Monday” Human Supervisor IT Agent Training Agent User RPA Able to complete repetitive tasks “Inputs new hire information into HR System”

Sends welcome email HR Manager IT Manager Training Manager New Hire Manually sets up user accounts and provisions access Gets new hire forms Has questions Answers questions Completes paperwork Confirm computer preference Confirmed Initiates IT process Missing applications submits IT ticket. Contacts procureme nt to obtain license. Initiates training process Schedule conflict Reviews training schedule and reschedules Sends feedback survey Waits for more data before analyzing and optimizing the entire process. Completes the training still has questions TODAY Existing solutions can only automate very specific tasks that have clear inputs and outputs Initial Interaction Document Verification Access & Technology Provisioning Training Onboarding Feedback Loop Survey is completed RPA automates the verification of completion and accuracy with predefined rules. RPA automates assigning predefined training modules but cannot adjust based on user feedback

• Provides a personalized onboarding journey based on interaction IT Agent Training Agent New Hire TOMORROW With AI Agents, these steps can be automated for the first time, while keeping human in the loop. HR Agent • Assesses documents and learns from interactions • Analyzes role, experience, and learning preferences to recommend training • Gathers real-time feedback • Identifies patterns makes informed decisions Provides adaptive training • Sets up user accounts, adapts to troubleshoot unexpected issues, and learns from errors. Fills out forms and has no questions The new hire has questions during the training. Orchestrates the additional processes Human in the Loop, Supervisor and Approver Initial Interaction Document Verification Access & Technology Provisioning Training Onboarding Feedback Loop

Key Applications of AI Agents

This cutting-edge shift promises to ignite a fresh wave of productivity and efficiency Intelligent Decision making Increased Productivity Enhanced Efficiency Cost Reduction Automation is undergoing a thrilling transformation with autonomous agents now being designed to proactively plan actions and perform tasks for users.

AI Agent workflows can enhance efficiency, accuracy, and customer satisfaction Key Use Cases Across Industries • Assists employees in booking business trips • Integrates with Tripadvisor, Outlook, and SharePoint • Books via Teams chat or email • Uses OCR to gather receipts • Automates expense report submission and tracking Travel Booking & Expense Management • Personalized onboarding assistant for new hires • Uses LLMs grounded in HR data from SharePoint • Provide relevant training materials • Schedule orientations and set up software accounts • Monitor task completion and ensure efficient onboarding Employee Onboarding • Diagnoses issues by referencing history and product manuals • Provides tailored solutions or escalates through automated workflows • Creates tickets and schedules follow-ups • Updates CRM records, enhancing future support Personalized Customer Support • Analytics data from data lake and data warehouse • Responds to user requests in natural language • Generates insights, visualization, and sends via Teams or email • Automates data handling for real-time, effortless decision-making Data Analytics and Reporting

Challenges in Development with AI Agents

4 primary considerations K Providing agents with the right context Knowledge Evaluation Ensuring they complete their tasks correctly Actions Giving them the tools to complete their tasks Security Ensuring they only have access to what they should Successfully developing AI Agents requires

So far, building agents from scratch been difficult to do Tool Integration Creating a cohesive system through complex integration of various tools and APIs that have different interfaces, data formats, and requirements. Interoperability Achieving interoperability between different tools and platforms to ensure that data can be shared and understood across different systems. Scalability Handling increased data volumes, more complex computations, and higher user loads without degrading performance. Real-time Processing Ensuring tools can handle real-time requirements without significant latency. Maintenance Making labor–intensive updates to integrated tools for compatibility with new versions and prevention of obsolescence and security vulnerabilities. Flexibility Modifying or customizing existing tools or developing new ones to meet unique requirements. Error Handling Ensuring errors are handled gracefully and continue functioning despite tool failures or unexpected inputs is critical for reliability. Security Implementing robust encryption, access controls, and compliance with privacy regulations to protect sensitive data.

Organizations need platforms that enable rapid development of performant, secure AI Agents Current Frameworks Security and data privacy risks What’s Needed Secure, responsible AI that protects sensitive information and behaves compliantly Lack of integrated tools, insecure data grounding, challenging orchestration Ineffective deployment of AI across websites, applications, and production environments Restrictive pre-defined models that are challenging to customize Flexible models that enable processing and integration of information from multiple modalities or types of data Connected complex workflow automation grounded by seamless connection to enterprise data Tools and APIs that seamlessly integrate across enterprise applications

Visual Studio Copilot Studio GitHub Azure AI Foundry SDK Azure AI Foundry Model Catalog Open-source models Foundational models Task models Industry models Azure AI Content Safety Azure AI Search Azure AI Agent Service Azure OpenAI Service Observability Customization Evaluations Governance Monitoring Announcing

Azure AI Agent Service Public Preview Empower developers to securely build, deploy, and scale AI agents with ease Flexible Model Selection Extensive Data Connections Enterprise-grade Security Rapid Development and Automation AI.Azure.com

The full package Built-in enterprise readiness BYO-file storage (coming soon) BYO-search index Azure AI Foundry SDK – Agent Service OBO Authorization Support Enhanced Observability Extensive Ecosystem of Tools Knowledge Microsoft Fabric* SharePoint* Grounding with Bing Search Azure AI Search Your own licensed data* Files (local or Azure Blob) File Search Code Interpreter Actions Azure Logic Apps* OpenAPI 3.0 Specified Tools Azure Functions Model Catalog Azure OpenAI Service (GPT-4o, GPT-4o mini) Models-as-a-Service Llama 3.1-405B-Instruct Mistral Large Cohere-Command-R-Plus *Indicates feature is coming soon

Introducing Azure AI Foundry SDK – Agent Service Azure OpenAI Assistants API + + Extensive Ecosystem of Tools Model Catalog Actions Azure AI Search Microsoft Fabric (coming soon) SharePoint (coming soon) Grounding with Bing Search Azure Logic Apps (coming soon) OpenAPI 3.0 Specified Tools Azure Functions Your own licensed data (coming soon) Files (local or Azure Blob) File Search Code Interpreter Azure OpenAI Service (GPT-4o, GPT-4o mini) Models-as-a-Service Llama 3.1-405B-Instruct Mistral Large Cohere-Command-R-Plus Built-In Enterprise Readiness BYO-file storage (coming soon) BYO-search index BYO-thread storage BYO-virtual network (coming soon) OBO Authorization Support Enhanced Observability Knowledge

How does your agent work? User Travel Booking Agent “Help me book a trip to New York for a client meeting? I need to fly out next Monday and return on Friday.” Knowledge Sources (search, files, databases, storage etc.) Models (Azure OpenAI Service, Models-as-a- Service) Actions (Pre-built or custom tools to automate processes) “I’ve booked your trip to New York as requested. Here are details:…”

How does your agent work? Step 1: Create an Agent Step 2: Create a Thread Step 3: Run the Agent Step 5: Check the Run status Step 6: Display the Agent’s Response Agent Travel Planning Agent Instructions You are a travel booking and expense management assistant designed to help employees plan, book, and manage business travel. Run 2 Model Tools (optional) File Search Code Interpreter Function Calling Bing Search Microsoft SharePoint (coming soon) Microsoft Fabric (coming soon) Azure Logic Apps (coming soon) Azure Functions OpenAPI 3.0 specified tools User’s message I need to book a hotel in New York for 2 stays. Agent’s message Here are some suggestions: Run 1 1 Use Tripadvisor API to search the nearest hotel Create message 2 Your data (optional) User’s message What’s the daily meal allowance for the business trip? Agent’s message The daily allowance for your business trip is $75, as per company policy. Use Microsoft SharePoint to query the company travel policy Create message 2 1 Thread Travel Planning Azure AI Search Files (local or Azure Blob)

Chat Completions API vs Assistants API vs Agents Assistants API Chat Completions API • Lightweight and powerful • Inherently stateless • Assistants API features, plus • Build with a model of your choice (OpenAI, Llama, Mistral, Cohere…) • Real time web-grounding with Bing • Secure grounding on enterprise data in SharePoint and Fabric • Bring Your Own Licensed data (Tripadvisor) • Connect to 1400+ data sources and services with Azure Logic Apps • Long running, event driven actions with Azure Functions • Standardized OpenAPI 3.0 tools • Bring Your Own Storage • Bring Your Own Private Network • Bring Your Own AI Search Resource • Limitless scaling with PTUs • Open Telemetry based tracing vs Azure AI Agent Service • Build with OpenAI models • Stateful (inbuilt conversation state management) • Access persistent threads • Automatic management of the model’s context window • Access files in several formats • Utilized Microsoft-managed storage • File Search (API handles chunking, embeddings storage and creation, and implementing vector search) • Code Interpreter • Function Calling vs

Customers are innovating with Azure AI Agent Service

Demo https://youtu.be/ltt7JNd30Ag?si=Y2Cbdx8YZOPFWJzB&t=1506

No content

Rapid Development and Automation

Rapid Development and Automation Out-of-the-box connectors Connection with 3P APIs Custom code Code Interpreter Client-side functions

Automation 1,400 built-in connectors through Azure Logic Apps (coming soon) • Create and run automated workflows with Azure Logic Apps with little to no code using the visual designer and selecting from prebuilt operations. • Simply define the business logic for your workflow to connect your agent to external systems, tools, and APIs. • Microsoft connectors include Microsoft products such as Azure App Service, Dynamics365 Customer Voice, Microsoft Teams, M365 Excel • Non-Microsoft connectors include enterprise services such as MongoDB, Dropbox, Jira, Gmail, Twilio, SAP, Stripe, ServiceNow and many more. Playground experience in Azure AI Foundry portal is coming soon in January

Automation Build standardized custom tools with OpenAPI 3.0 specifications • Connect your AI agent to an external API using an OpenAPI 3.0 specified tool, allowing for scalable interoperability with various applications • Build an organization-side tools library specified with the OpenAPI 3.0 standard • Enable your custom tools to authenticate access and connections with managed identities (Microsoft Entra ID) for added security, making it ideal for integrating with existing infrastructure or web services Screenshot Playground experience in Azure AI Foundry portal is coming soon in January

Automation Implement stateless or stateful code- based actions with Azure Functions • Enable your agent to perform external interactions with the world, such as calling APIs or asynchronously sending and waiting for events. • Azure Functions and Azure Durable Actions enable you to execute serverless code for synchronous, asynchronous, long-running, and event-driven actions such as approving invoices with human-in-the- loop, monitor an end-to-end product supply chain over long periods of time, and many more. Screenshot Playground experience in Azure AI Foundry portal is coming soon in January

Automation Perform advanced data analysis with Code Interpreter • Code Interpreter allows your agent to write and run Python code in a sandboxed execution environment, handling diverse data formats and generate files with data and visuals. • With Code Interpreter enabled, your agent can run code iteratively to solve more challenging code, math, and data analysis problems. • Unlike the Assistants API, you can integrate data in your storage to use with this tool. Screenshot Playground experience in Azure AI Foundry portal is coming soon in January

Automation Extend Llama Stack agents with cloud- based tools • Supports the agent protocol for developers that are using Llama Stack SDKs • Natively offer a scalable, cloud-hosted, enterprise grade tools • Wireline compatible with Llama Stack • Protocol adapter will be a standalone distributable, discoverable from the llama-stack OSS repo directly Screenshot Playground experience in Azure AI Foundry portal is coming soon in January

Extensive Data Connections

Extensive Knowledge Connections Public web data Corporate data Structured data Unstructured data Licensed data

Data Connections Real-time, web data grounding with Bing Search • Create more reliable and trustworthy applications with real-time public information from Bing • Grounding with Bing Search allows your agents to integrate real-time public web data, ensuring their response is accurate and up to date. • By including supporting URLs and search query links, Grounding with Bing Search enhances trust and transparency, empowering the users to verify responses with the original sources. Playground experience in Azure AI Foundry portal is coming soon in January

Data Connections Grounding with private data in Microsoft SharePoint (coming soon) • Grounding with SharePoint makes your SharePoint content more accessible to your end users. • Enterprise-grade security features, such as On-behalf-of (OBO) authentication for SharePoint, ensure secure and controlled access for end users. Playground experience in Azure AI Foundry portal is coming soon in January

Data Connections Chat with your structured data in Microsoft Fabric (coming soon) • Integrate your agents with Fabric AI Skill to unlock powerful data analysis capabilities. • Fabric AI Skills can transform enterprise data into conversational Q&A systems, allowing users to interact with data through chat and uncover data-driven and actionable insights effortlessly. • On-behalf-of (OBO) authentication simplifies access to enterprise data in Fabric while maintaining robust security, ensuring proper access control and enterprise-grade protection. Playground experience in Azure AI Foundry portal is coming soon in January

Data Connections Incorporate private data in Azure AI Search, Blob, and your local files • Bring your existing Azure AI Search index or create a new one using the improved File Search tool. • This tool leverages a built-in ingestion pipeline to process files from your local system or Azure Blob Storage. • Maintain complete control over your data as your files remain in your own storage, and your Azure AI Search resource ingests them Playground experience in Azure AI Foundry portal is coming soon in January

Data Connections Gain a competitive advantage with licensed data (private preview) • Empower your agents to deliver nuanced, informed answers tailored to specific use cases • Integrate licensed data from specialized data providers, such as Tripadvisor, to enrich agent responses • Tripadvisor enhances the quality of your agent’s responses to travel related queries with high-quality, fresh data, such as travel guidance and reviews (coming soon) Playground experience in Azure AI Foundry portal is coming soon in January

Flexible Model Selection

Flexible Model Selection Azure OpenAI Service models Models as a Service (Serverless API) Azure AI Services models Multi-modal support Fine-tuned models

Flexible Model Selection Leverage Models as a Service (MaaS) • Support a variety of models from Azure AI Foundry model catalog to power your agent’s reasoning. The supported models include: • GPT-4o • GPT-4o mini • Llama 3.1 • Mistral Large • Cohere Command R+ • Leverage Model Inference API to easily swap models and compare performance to find the best model for your specific needs.

Flexible Model Selection Use fine-tuned models for your agents • Customize AI models to address your unique business needs with fine-tuning, enabling you to build task-specific agents while optimizing token costs. • Through collaborations with partners like Scale AI, you can efficiently label training data and create fine-tuned models that integrate seamlessly with Azure AI Agent Service. • This process enhances the performance of AI agents, streamlining development and reducing time to production. Azure AI Agent Service supports fine-tuned gpt-35-turbo (0125) as of Jan 2025

Flexible Model Selection Multi-modal support across text, image, and audio • Unlock new scenarios with multi-modal support, enabling AI agents to process and respond to diverse data formats beyond text, expanding the potential use cases. • Support for GPT-4o’s image and audio modalities so that you can analyze and combine data from various formats to deliver comprehensive insights, make decisions, and provide relevant outputs tailored to specific user needs Playground experience in Azure AI Foundry portal is coming soon in January

Enterprise-grade Security

Enterprise-grade Security Bring your own storage Keyless setup and authentication Private Virtual Network support Tracing/ monitoring Content filters

Security Bring your own storage and search • Bring your own storage account (coming soon) and Azure AI Search resource for custom data handling. • All files you upload will be stored in your storage account, ensuring you maintain complete control over your data. • All indexes get created using your connected Azure AI Search resource. Playground experience in Azure AI Foundry portal is coming soon in January

Security Bring your own virtual network (coming soon) • No public egress foundational infrastructure ensures the right authentication and security for your agents and tools, without you having to do trusted service bypass • Container injection allows the platform network to host APIs and inject a subnet into your network, enabling local communication of your Azure resources within the same virtual network (VNet). • If your resources are marked as private and non-discoverable from the Internet, the platform network can still access them, provided the necessary credentials and authorization are in place

Security Monitor agent performance with OpenTelemetry-based training (coming soon) • Gain insights into your AI agent's performance and reliability • OpenTelemetry-compatible metrics for offline and online evaluation of agent outputs through the Azure AI Foundry SDK • Add local variables and intermediate results to trace decorator for detailed tracing capabilities for user defined functions • Options to prevent sensitive or large data logging as per OpenTelemetry standards

Security Build responsibly with content filters and XPIA mitigation (coming soon) • Supports prebuilt and custom content filters that detect harmful content at varying severity levels. • Prompt shields protect agents against cross-prompt injection attacks from malicious actors. • As with Azure OpenAI Service, prompts and completions processed by the Azure AI Agent Service are not used to train, retrain, or improve Microsoft or 3rd party products or services without your permission. Customers can delete their stored data when they see fit.

Basic agent setup • Overview: All agents created in this project use multi- tenant search and storage resources fully managed by Microsoft. You don't have visibility or control over these underlying Azure resources. • Required customer resources: • AI hub • AI project • AI Services/AOAI • How to use: Deploy the basic setup template • Tool Implications: • File search and code interpreter • Uploaded files get stored in Microsoft managed storage • Vector stores get created using a Microsoft managed search resource • Azure AI Search tool is not supported • Azure Blob Storage with file search is not supported

Standard agent setup • Overview: Agents use customer-owned, single-tenant search and storage resources. With this setup, you have full control and visibility over these resources, but you incur costs based on your usage. • Required customer resources: • AI hub • AI project • AI Services/AOAI • Key Vault • Storage account • Azure AI Search • How to use: Deploy the standard setup template • Tool Implications: • File search and code interpreter • Uploaded files get stored in your Azure Blob Storage account • Vector stores get created using your Azure AI Search resource

Start building agents with Copilot Studio… Build, deploy, realize value and prototype faster for simple scenarios Build agents for business processes without setting up & managing your own infrastructure… Copilot Studio Azure AI Foundry ...and scale with Azure AI Bring your vectorized indices from Azure AI Search Access 1800+ models from Azure AI model catalog

...or you can start with code first solutions Control and customize with options to manage your own infrastructure Copilot Studio Build highly customized apps, Agents, and APIs using Azure AI Foundry... ...and integrate Copilot Studio components with Microsoft 365 Agent SDK including the ability to customize and deploy app UI to 15+ channels Visual Studio/ GitHub Learn more about the better-together with Copilot Studio and Azure AI Foundry Better together: Copilot Studio and Azure AI

Deploy agents with Azure AI Foundry Orchestrate them together with AutoGen and Semantic Kernel Single-agent Multi-agent 1 2 State-of-the-art research SDK Production-ready and stable SDK Managed agent micro-services Ideation Production

Agent Sample #1 Web Search and Extraction Web Search Query Decomposition Retrieve Web Sites Search Query Site Analyzer Extract HTML Body Screenshot Analysis Verifier Memory Thoughts Error Handling Page Transition Convert to Markdown Page 1 Page 2 Text Text Screenshot Generator Web Result 1 * Azure AI Content Understanding Azure AI Search Doc Search * We can use the Azure AI Content Understanding service to extract images and videos, and transform them from unstructured to structured data Azure AI Agent Service Azure AI Agent Service Azure AI Agent Service

Agent Sample #2 Call Center Agent Assist Query Parsing C u s t o m e r Q u e r y My SmartHome thermostat isn’t connecting to Wi-Fi, and I keep seeing an error code E-22. How can I fix this? LLM { Wi-Fi } { Error: E-22 } { Thermostat X10 } Planner Agent Azure AI Agent Service Video Agent Doc Agent Web Agent 1. Doc Agent → Search internal support documentation for error code "E-22" and Wi-Fi troubleshooting. 2. Video Agent → Locate and timestamp troubleshooting video content related to Wi-Fi connection and error code E-22. 3. Web Search Agent → Search external sources for recent discussions, tips, or product-specific issues regarding Wi-Fi and the error code. D y n a m i c P l a n Gathers internal troubleshooting steps. Summarizes video clips related to Wi-Fi and error code E-22. Finds the latest external resources related to the issue * Abstraction Recent Online Findings from Web Internal Steps from Documents Video Link and Timestamps from Videos * We can use the Azure AI Content Understanding service to extract images and videos, and transform them from unstructured to structured data Azure AI Agent Service Azure AI Agent Service Azure AI Agent Service

Agent Sample #3 AI Teammates powered by multi-agents Marketing Director Role: Social Media Manager Qualification: Expert Skills: Content creation Coding... AI Teammate Marketing Director AI Teammate Publish to Linked In Create an AI Teammate Collaborate with AI Teammate Invite AI Teammate to Teams meeting Conversation AI Coding Agent Content Gen Azure AI Agent Service Azure AI Agent Service Azure AI Agent Service Create AI Teammate

Get started with Agent Service on Microsoft Learn Docs https://aka.ms/MicrosoftLearn_Agents Get started with Azure AI Agents through Azure AI Foundry SDK Watch the Microsoft Ignite sessions on- demand  Building intelligent multi-agent systems with Azure AI Read the Techcommunity announcement

Thank you