Slide 1

Slide 1 text

Microsoft Build 2023 Updates – Copilot Stack and Azure OpenAI Service SATO Naoki (Neo) Senior Software Engineer, Microsoft

Slide 2

Slide 2 text

+

Slide 3

Slide 3 text

World’s most powerful supercomputers World’s most capable foundation models, from hosted to open source World’s best AI dev infrastructure

Slide 4

Slide 4 text

Copilot

Slide 5

Slide 5 text

Announcing Windows Copilot

Slide 6

Slide 6 text

Announcing Copilot extensibility and plugins ChatGPT Bing Chat Microsoft Copilots Windows Copilot Your Copilots

Slide 7

Slide 7 text

Augment AI systems to access APIs Retrieve useful information Perform new computations Safely act on the user’s behalf Plugins

Slide 8

Slide 8 text

The digital world Plugins Copilot

Slide 9

Slide 9 text

Copilot stack AI infrastructure Foundation models AI orchestration Orchestration Copilots Plugin extensibility Apps

Slide 10

Slide 10 text

Copilot stack AI infrastructure Foundation models AI safety BYO models Hosted fine-tuned foundation models Hosted foundation models Plugin extensibility Plugin execution Grounding Metaprompt Prompt & response filtering Orchestration Copilot frontend UX

Slide 11

Slide 11 text

Foundation models AI safety BYO models Hosted fine-tuned foundation models Hosted foundation models Plugin extensibility Copilot frontend UX Orchestration frameworks Copilot stack Plugin execution Grounding Orchestration Metaprompt Prompt & response filtering

Slide 12

Slide 12 text

Foundation models AI safety Plugin extensibility Plugin execution Grounding Orchestration Copilot frontend UX Metaprompt Prompt & response filtering Prompt and Metaprompt Copilot stack

Slide 13

Slide 13 text

Metaprompt Foundation models AI safety BYO models Hosted fine-tuned foundation models Hosted foundation models Plugin extensibility Plugin execution Orchestration UX AI infrastructure Vector databases • Web APIs • Plugins Prompt & response filtering Copilot stack Retrieval Augmented Generation (RAG) Grounding

Slide 14

Slide 14 text

Foundation models AI safety Grounding Metaprompt Prompt & response filtering Orchestration Copilot frontend UX Plugin execution Plugin extensibility Copilot stack Plugins

Slide 15

Slide 15 text

AI safety Plugin execution Grounding Metaprompt Prompt & response filtering AI infrastructure Foundation models BYO models Hosted fine-tuned foundation models Hosted foundation models Copilot stack Foundation models and fine-tuning

Slide 16

Slide 16 text

AI infrastructure Foundation models AI safety BYO models Hosted fine-tuned foundation models Hosted foundation models Plugin extensibility Plugin execution Grounding Metaprompt Prompt & response filtering Orchestration Copilot frontend UX Copilot stack

Slide 17

Slide 17 text

Copilots New development pattern Unique architecture Will be everywhere

Slide 18

Slide 18 text

Azure AI Applied AI Services Bot Service Cognitive Search Form Recognizer Video Indexer Metrics Advisor Immersive Reader Cognitive Services Vision Speech Language Decision Azure OpenAI Service Azure Machine Learning Prepare & Preprocess Build, Train & Consume Deploy & Scale Manage & Monitor AI Infrastructure

Slide 19

Slide 19 text

Announcing Azure AI Studio Build and train your own models Ground Azure OpenAI Service and OSS models using your data Built-in vector indexing Retrieval augmented generation made easy Create prompt workflows AI safety built-in

Slide 20

Slide 20 text

Generative AI Applications Azure Machine Learning Native OSS Model Catalog Prompt Eng/Eval Prompt flow Responsible AI Azure Content Safety High Scale Gen AI App Deployment Gen AI Model Monitoring

Slide 21

Slide 21 text

Radically changing the art of possible with Azure OpenAI Service Large pretrained foundation AI models custom-tunable with your parameters and your data Summarization Reasoning over data Writing tools Code generation ChatGPT The Era of Copilots GPT-3 (GA) DALL•E 2 (preview) ChatGPT (GA) GPT-4 (GA) Foundation of enterprise security, privacy and compliance

Slide 22

Slide 22 text

Enterprise innovation on Azure OpenAI Service

Slide 23

Slide 23 text

https://azure.microsoft.com/en-us/blog/mercedes-benz-enhances-drivers-experience-with-azure-openai-service/

Slide 24

Slide 24 text

Update Bookmark ‘what's new’ in Azure OpenAI Service Sign up for Azure OpenAI Service updates Azure OpenAI Service GPT-3 (GA) DALL·E 2 (preview) ChatGPT (GA) GPT-4 (GA) Apply your own data Available in Preview early June now Plugins for Azure OpenAI Service Coming soon Configurable Content Filters Available in Preview early June now Provisioned Throughput Limited Availability early June

Slide 25

Slide 25 text

RAG: LLMs + your data Retrieval Augmented Generation

Slide 26

Slide 26 text

Anatomy of a RAG app App UX Orchestrator Retriever over Knowledge Base Query → Knowledge Prompt + Knowledge → Response Large Language Model Build your own experience UX, orchestration, calls to retriever and LLM e.g., Copilots, in-app chat Extend other app experiences Plugins for retrieval, symbolic math, app integration, etc. e.g., plugins for OpenAI ChatGPT

Slide 27

Slide 27 text

Retrievers: Externalizing Knowledge “Find the most relevant snippets in a large data collection, using unstructured input as query” == search engine App UX Orchestrator Azure OpenAI Azure Cognitive Search Data Sources (files, databases, etc.) Query → Knowledge Prompt + Knowledge → Response Azure Cognitive Search  Azure’s complete retrieval solution  Data ingestion, enterprise-grade security, partitioning and replication for scaling, support for 50+ written languages, and more

Slide 28

Slide 28 text

Retrieving Using Semantic Similarity Vector representations (or embeddings)  Learned such that “close” vectors represent items with similar meaning  May encode words, sentences, images, audio, etc.  Some map multiple media types into the same space  Azure OpenAI embeddings API, OSS embeddings (e.g., SBERT, CLIP)

Slide 29

Slide 29 text

Vector-based Retrieval Encoding (vectorizing)  Pre-process and encode content during ingestion  Encode queries during search/retrieval Vector indexing  Store and index lots of n-dimensional vectors  Quickly retrieve K closest to a “query” vector  Exhaustive search impractical in most cases  Approximate nearest neighbor (ANN) search Embedding [0.023883354, 0.021508986, 0.044205155, 0.019588541, 0.031198505, …]

Slide 30

Slide 30 text

Vector Search in Azure Cognitive Search New vector type for index fields  Users indicate vector size, distance function, algorithm and algo-specific parameters Pure Vector Search & Hybrid Search  Filters, faceting, etc. all works with vectors  Integrates with existing search indexes  Existing data ingestion and augmentation machinery entirely applicable Combines well with L2 re-ranker powered by Bing’s models  Enables improved ranking for hybrid search scenarios  L1: keywords + vector retrieval  L2: Bing’s ranker refreshed with GPT-enhanced work Enterprise-grade  Scalability (partitioning, replication)  Security: network isolation, managed identities, RBAC, etc.

Slide 31

Slide 31 text

Revolutionizing Indexing and Retrieval for LLM-powered Apps Power your retrieval-augmented generation applications Images Audio Video Graphs Documents • Use vector or hybrid search • Use Azure OpenAI embeddings or bring your own • Deeply integrate with Azure • Scale with replication and partitioning • Build generative AI apps and retrieval plugins Sign up today https://aka.ms/VectorSearchSignUp

Slide 32

Slide 32 text

Azure OpenAI Service on your data

Slide 33

Slide 33 text

Chat Completions API Versatile interface use for all scenarios—not just chat Model adheres to instructions in “system” message It sets the behavioral guidelines for the model, including responsible AI steering Examples go into “User”, “Assistant” fields import openai openai.api_type = "azure” response = openai.ChatCompletion.create( engine ="gpt-4", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Who won the world series in 2020?"}, {"role": "assistant", "content": "The LA Dodgers won the World Series in 2020."}, {"role": "user", "content": "Where was it played?"} ] ) …

Slide 34

Slide 34 text

Using your data Challenge I want to reason over my very long documents, but the token limits are not sufficient Conventional solution Build a vector database Retrieve relevant data and construct prompt at run-time

Slide 35

Slide 35 text

Introducing: Azure OpenAI Service on your data (Preview) Data Sources (search, files, databases, storage etc.) Additional 3P Data Sources (future capabilities) Azure OpenAI Service on your data API & SDK App or Copilot agent

Slide 36

Slide 36 text

https://techcommunity.microsoft.com/t5/ai-cognitive-services-blog/introducing-azure-openai-service-on-your-data-in-public-preview/ba-p/3847000 https://www.youtube.com/watch?v=6SNfeVop4zM

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

Plugins Azure OpenAI Service Plugins

Slide 39

Slide 39 text

Expanding potential Challenges we wanted to address Accurate translation for wider range of languages—improve support for Asian and African languages Integrate vector databases and cloud data stores Use up-to-date information from the web

Slide 40

Slide 40 text

Introducing: Azure OpenAI Service Plugins (coming soon) Build powerful AI Copilots with secure access to Microsoft services Retrieve data with Azure Cognitive Search Translate >100 languages with Azure Translator Ground with recent info with Bing Search Extract structured data from Azure SQL Azure OpenAI Plugins • Securely access your data in various data stores, vector databases and the web • Data path access controlled via Azure AD and Managed Identities • Admin roles to choose what plugins to enable

Slide 41

Slide 41 text

Recap Bookmark ‘what's new’ in Azure OpenAI Service Sign up for Azure OpenAI Service updates Azure OpenAI Service GPT-3 (GA) DALL·E 2 (preview) ChatGPT (GA) GPT-4 (GA) Apply your own data Available in Preview early June Plugins for Azure OpenAI Service Coming soon Configurable Content Filters Available in Preview early June Provisioned Throughput Limited Availability early June

Slide 42

Slide 42 text

Resources  Microsoft Build (2023/05/23-25)  https://build.microsoft.com/  DEEP LEARNING LAB - [Recap] Microsoft Build 2023 最新アップデー トAnalytics&AI (2023/06/08)  https://dllab.connpass.com/event/284571/  Microsoft Build 2023 Azure AI&ML 最新アップデート - Speaker Deck  https://speakerdeck.com/shohei1029/microsoft-build-2023-azure-ai-and-ml-zui-xin-atupudeto  Microsoft Build Japan (2023/06/27‐28)  https://info.microsoft.com/JA-ADAI-CATALOG-FY23-06Jun-28-Microsoft-Build-Japan-Day2- SREVM14500_Catalog-Display-Page.html

Slide 43

Slide 43 text

© Copyright Microsoft Corporation. All rights reserved.