LLMOps with Azure Machine Learning prompt flow SATO Naoki (Neo) Senior Software Engineer, Microsoft

Access to thousands of LLMs from OpenAI, Meta, Hugging Face Azure Machine Learning for Generative AI Prompt engineering/ evaluation Built-in safety and responsible AI Continuous monitoring for LLMs Purpose-built AI infrastructure

The paradigm shift (MLOps vs LLMOps) Traditional MLOps LLMOps Target audiences Assets to share Metrics/evaluations ML models ML Engineers Data Scientists ML Engineers App developers Model, data, environments, features LLM, agents, plugins, prompts, chains, APIs Accuracy Accuracy, fairness, groundedness, relevance, coherence Build from scratch Pre-built, fine-tune

Operationalize LLM app development with prompt flow LLMOps is a complex process. Customers want: • Private data access and controls • Prompt engineering • CI/CD • Iterative experimentation • Versioning and reproducibility • Deployment and optimization • Safe and Responsible AI Design and development Develop flow based on prompt to extend the capability Debug, run, and evaluate flow with small data Modify flow (prompts and tools etc.) No If satisfied Yes Evaluation and refinement No Evaluate flow against large dataset with different metrics (quality, relevance, safety, etc.) If satisfied Yes Optimization and production Optimize flow Deploy and monitor flow Get end user feedback

Streamline prompt engineering projects Azure Machine Learning prompt flow Customer Benefits • Create AI workflows that connect various language models, APIs, and data sources to ground LLMs on your data. • One platform to design, construct, tune, evaluate, test, and deploy LLM workflows • Evaluate the quality of workflows with rich set of pre-built metrics and safety system. • Easy prompt tuning, comparison of prompt variants, and version-control. Documentation: https://aka.ms/prompt_flow

Studio UI

Azure Machine Learning prompt flow (1/7) Capabilities Overview • Develop workflows • Develop flows that connect to various language models, external data sources, tools, and custom code • Test and evaluate • Test flows with large datasets in parallel • Evaluate the AI quality of the workflows with metrics like performance, groundedness, and accuracy • Prompt tuning • Easily tune prompts​ with variants and versions • Compare and deploy • Visually compare across experiments • One-click deploy to a managed endpoint for rapid integration

Azure Machine Learning prompt flow (2/7) Prompt flow authoring Develop your LLM flow from scratch • Construct a flow using pre-built tools • Support custom code • Clone flows from samples • Track run history

Azure Machine Learning prompt flow (3/7) Connections Manage APIs and external data sources • Seamless integration with pre-built LLMs like Azure OpenAI Service • Built-in safety system with Azure AI Content Safety • Effectively manage credentials or secrets for APIs • Create your own connections in Python tools

Azure Machine Learning prompt flow (4/7) Variants • Create dynamic prompts using external data and few shot samples • Edit your complex prompts in full screen • Quickly tune prompt and LLM configuration with variants

Azure Machine Learning prompt flow (5/7) Evaluation • Evaluate flow performance with your own data • Use pre-built evaluation flows • Build your own custom evaluation flows Tune Variant 0 Tune Variant 1 Tune Variant 2 Flow variants Evaluation Bulk Test

Azure Machine Learning prompt flow (6/7) Evaluation • Compare multiple variants or runs to pick best flow • Add new evaluations to a finished run • Ensure accuracy by scaling the size of data in evaluation Tune Variant 0 Tune Variant 1 Tune Variant 2 Flow variants Evaluation Bulk Test

Azure Machine Learning prompt flow (7/7) Deploy • Seamless transition from development to production with AzureML’s managed online endpoints Production Tune Variant 0 Tune Variant 1 Tune Variant 2 Flow variants Test App

What is prompt flow code experience ？ Use code to define flow File based flow, organized in a well-defined folder structure​ Support CLI/SDK​ Smooth transition between cloud and local Download flow to local, import flow to cloud​ Develop, test, debug, deploy on local ​ Submit run from local to cloud​ From local deploy to cloud​ Manage runs/evaluation in cloud Integrate with your CI/CD automation SDK/CLI to init, execute, evaluate, visualize flow and metrics VS Code Extension Flow editor​ Local connection management​ Run history​ Collaboration or share cross workspace Submit flow runs to cloud from your repo (anywhere)

Demo - Azure Machine Learning prompt flow 1. Upload PDF files and create a vector search index in Azure AI Search 2. Create a new chat flow in prompt flow 3. Configure Azure OpenAI Service (LLM) and Azure AI Search (vector search) 4. Run the chat flow 5. Evaluate the chat flow 6. Deploy the chat flow as a REST API

No content

No content

No content

No content

No content

No content

No content

No content

No content

No content

No content

No content

No content

No content

No content

No content

No content

No content

No content

No content

No content

Learn More • What is Azure Machine Learning prompt flow - Azure Machine Learning | Microsoft Learn • Prompt flow — Prompt flow documentation (microsoft.github.io)