Building RAG-Based Copilots With The Azure AI Platform Nitya Narasimhan, PhD AI Advocacy @Microsoft Leah Bar-On Simmons Azure AI Experiences @ Microsoft

Agenda Define: Application Scenario (Contoso Chat)  Provision Azure Infrastructure with AZD  Build RAG-Based Copilot (with your data)  Evaluate Copilot (process & metrics)  Deploy Copilot (hosting & monitoring) Deliver: Responsible & Observable Copilot (Azure AI Studio) https://github.com/microsoft/RAG_Hack/discussions/26

Start: Describe the Application Scenario Understand the Application Lifecycle Introduce the core Developer Tools Demo: Contoso Outdoors (UI) with Contoso Chat (AI)

Application Scenario: What are we trying to build? Frontend: Contoso Outdoors This is the retailer’s website (chat UI). Customers can ask questions about products or get recommendations Backend: Contoso Chat This is the copilot endpoint (chat AI). It receives chat requests from the UI & returns relevant responses for display.

Demo Retail copilot (chat AI) with natural language queries from users Response grounded in product catalog data Response contextual to customer data, history Responsible AI with content safety filters

Saf e Rollout/Staging Retrieval Augmente d Generation Hypo thesis Find LLMs Try prompts Operationalizing Building/ augmenting Ideating/ exploring Managing SEND FEEDBACK PREPARE FOR APP DEPLOYMENT ADVANCE PROJECT BUSINESS NEED Deploy LLM App/UI Quo ta and cost management REVERT PROJECT Prompt Engine ering or Fine-tuning Evalua tion Exceptio n Handling C ontent Filtering M onitoring Application Lifecycle: How do we build this in reality?

Identify business use case Run flow against sample data Evaluate prompt flow Satisfied? Run flow against larger dataset Evaluate prompt flow Satisfied? Deploy endpoint No No Yes Yes Modify flow (prompts and tools, etc.) Integrate into application Connect to your data and build LLM flows Discover your model Test sample prompt Add monitoring and alerts 1. Ideating/exploring 2. Building/augmenting 3. Operationalizing

Developer Experience: What do we care about? Does my app work correctly with diverse user inputs? Evaluation techniques, prompt engineering How can I debug and understand failures in complex workflows? Tracing and debugging, local and production How can I simplify infrastructure & environment setup & maintenance? Infrastructure-as-code templates, Configuration-as-code environments, Unified developer CLI

T Provision & Manage Azure Infrastructure Setup & Maintain Local Environment Rapid Prototyping & Prompt Engineering AI-Assisted Flows & Evaluation Metrics Support for Hosting, Safety & Monitoring Azure Developer CLI GitHub Codespaces Prompty & Prompt flow Promptflow CLI & Evaluators Azure AI Studio UI Developer Tooling: Azure AI Platform & Workflow Tool Task

Getting Started: Understand the Design Architecture (RAG) Setup Local Environment (with Codespaces) Provision Azure (with Azure Developer CLI) Demo: Contoso Chat Sample Quickstart Provision & Manage Azure Infrastructure Setup & Maintain Local Environment Rapid Prototyping & Prompt Engineering AI-Assisted Flows & Evaluation Metrics Support for Hosting, Safety & Monitoring Azure Developer CLI GitHub Codespaces Prompty & Prompt flow Promptflow CLI & Evaluators Azure AI Studio UI

Retrieval Augmented Generation: Quick Overview

Azure AI Platform: Unified Tooling for Generative AI Best-in-class AI foundation models Azure AI Services Pre-trained, turnkey solutions for intelligent applications Responsible AI Tooling Build and manage apps that are trustworthy by design Azure Machine Learning Full-lifecycle tools for designing and managing AI models Azure AI Studio A comprehensive platform to develop and deploy custom copilots

Azure Developer CLI: Code to Cloud with 1 command azd up Azure Developer CLI azd commands Your app The cloud

Infrastructure As Code: AI Project Templates for AZD Customer benefits • Quickly deploy applications with the command “azd up” • Common starter AI Scenarios to implement solutions • GitHub Actions for CI/CD deployment and evaluation to test model performance • The CLI allows you to work consistently in a repeatable way across the terminal, your editor/integrated development environment, GitHub Actions pipeline, and more. https://aka.ms/azd-ai-templates

Contoso Chat Architecture: An AZD AI Template https://aka.ms/azd-ai-templates

Demo Fork Repo – Launch Codespaces – Provision Infra – Validate Setup

Build: Prompt engineering with Prompty Orchestration with Prompt flow Demo: Local Development Of Basic Copilot Provision & Manage Azure Infrastructure Setup & Maintain Local Environment Rapid Prototyping & Prompt Engineering AI-Assisted Flows & Evaluation Metrics Support for Hosting, Safety & Monitoring Azure Developer CLI GitHub Codespaces Prompty & Promptflow Promptflow CLI & Evaluators Azure AI Studio UI

Prompty Prompty is a language agnostic prompt asset for creating prompts and engineering the responses.

Prompty is – a specification + tooling + runtime • Easy to start • Intuitive • Code-first • Language and framework agnostic • Agency with Observability

Demo Create Prompty – Configure Prompty – Run Prompty – Iterate Once

Contoso Chat Prompty Template – chat.json The “sample” input works for rapid prototyping and testing … but needs orchestration for real data sources

End-to-End Development With Prompty @MSBuild

Build: Reference multiple data sources (in template) Orchestrate complex flows with Promptys (in code) Demo: Execute chat.prompty with pf CLI – review code Provision & Manage Azure Infrastructure Setup & Maintain Local Environment Rapid Prototyping & Prompt Engineering AI-Assisted Flows & Evaluation Metrics Support for Hosting, Safety & Monitoring Azure Developer CLI GitHub Codespaces Prompty & Prompt flow Promptflow CLI & Evaluators Azure AI Studio UI

Contoso Chat Flow Orchestration: Demo https://aka.ms/azd-ai-templates

Demo Execute flow – Explore codebase Template – chat.prompty, chat.json Flow -- chat_request.py, ai_search.py, flex.flow.yaml

Contoso Chat local testing – using Prompt flow CLI 1. Test locally with promptflow CLI – specify flow and inputs as parameters 2. Generates a default trace view that can be viewed locally, via browser. 3. On Codespaces, view by clicking the Preview icon for port specified in URL 4. UI for Trace view maintains history of runs, drill-down for per-run details

Contoso Chat Trace View – for successful run 1. Open Run View to dive into detailed traces of flow execution .. 2. Integrates traces from openai – get insight into LLM tool execution 3. Get insight into Prompt execution 4. Debug model configuration issues (e.g., LLM Parameters).. And more

Contoso Chat Prompty Template – chat.prompty Develop more complex templates with context from multiple data sources, add instructions & safety rules

Contoso Chat code – chat_request.py, ai_search.py Complex workflow with multiple models (chat, embedding) & tools (functions) = orchestrated using Azure AI SDKs

Evaluate: Understand AI-Assisted Evaluation Flow Understand Quality vs. Safety Metrics Demo: Evaluate copilot with custom quality evaluators Provision & Manage Azure Infrastructure Setup & Maintain Local Environment Rapid Prototyping & Prompt Engineering AI-Assisted Flows & Evaluation Metrics Support for Hosting, Safety & Monitoring Azure Developer CLI GitHub Codespaces Prompty & Prompt flow Promptflow CLI & Evaluators Azure AI Studio UI

AI-Assisted Evaluation – one LLM “grades” the other

No content

No content

Evaluation (Preview) – What’s New In Azure AI Studio https://ai.azure.com/build/evaluation

No content

Demo (to be updated) Eval dataset - Eval prompty – Eval flow - Eval metrics - Evaluators

Deploy: Deploy to Azure AI Studio with AZD Explore Built-in Testing, Integrations Demo: Test the Copilot from AI Studio & from Web App Provision & Manage Azure Infrastructure Setup & Maintain Local Environment Rapid Prototyping & Prompt Engineering AI-Assisted Flows & Evaluation Metrics Support for Hosting, Safety & Monitoring Azure Developer CLI GitHub Codespaces Prompty & Prompt flow Promptflow CLI & Evaluators Azure AI Studio UI

Demo Use `azd deploy` to deploy the application to Azure AI Studio endpoint Use built-in testing with sample inputs to validate

Demo Use `azd deploy` to deploy the application to Azure AI Studio endpoint Use built-in testing with sample inputs to validate Benefit from integrated dashboards for monitoring and logging to help debug

Demo Use `azd deploy` to deploy the application to Azure AI Studio endpoint Use built-in testing with sample inputs to validate Benefit from integrated dashboards for monitoring and logging to help debug Integrate it with Contoso Web UI (Static Web App)

Cleanup Use `azd down` to cleanup your resources effectively (with AZD AI templates) Some Azure AI resources have soft delete enabled limiting reuse of names or available quota for models Purge any soft-deleted resources to allow reuse in iterative or collaborative dev environments

Let’s Recap

What we learned today Provision & Manage Azure Infrastructure Setup & Maintain Local Environment Rapid Prototyping & Prompt Engineering AI-Assisted Flows & Evaluation Metrics Support for Hosting, Safety & Monitoring Azure Developer CLI GitHub Codespaces Prompty & Prompt flow Promptflow CLI & Evaluators Azure AI Studio UI  What is the Application Lifecycle? (LLM Ops)  What is the Azure AI Platform? (Dev Tools)  What is Contoso Chat? (Retail Copilot Sample)  How to build, evaluate, deploy RAG copilot on Azure https://github.com/microsoft/RAG_Hack/discussions/26

What you can do to skill up  Explore Other Azure AI Templates https://aka.ms/azd-ai-templates  Fork Contoso Chat and explore/extend it! https://aka.ms/aitour/contoso-chat  Join us on the AI Tour at a city near you! https://aka.ms/aitour https://github.com/microsoft/RAG_Hack/discussions/26