Microsoft Build 2023 Updates – Copilot Stack and Azure OpenAI Service (Machine Learning 15minutes! Broadcast #78)

第78回 Machine Learning 15minutes! Broadcast (2023/06/24)
https://machine-learning15minutes.connpass.com/event/285147/

S/N Ratio > Microsoft Build 2023 Updates – Copilot Stack and Azure OpenAI Service (Machine Learning 15minutes! Broadcast #78)
https://satonaoki.wordpress.com/2023/06/24/build-2023-updates-copilot-stack-azure-openai/

Transcript

1. Microsoft Build 2023 Updates – Copilot Stack and Azure OpenAI Service SATO Naoki (Neo) Senior Software Engineer, Microsoft
2. +
3. World’s most powerful supercomputers World’s most capable foundation models, from hosted to open source World’s best AI dev infrastructure
4. Copilot
5. Announcing Windows Copilot
6. Announcing Copilot extensibility and plugins ChatGPT Bing Chat Microsoft Copilots Windows Copilot Your Copilots
7. Augment AI systems to access APIs Retrieve useful information Perform new computations Safely act on the user’s behalf Plugins
8. The digital world Plugins Copilot
9. Copilot stack AI infrastructure Foundation models AI orchestration Orchestration Copilots Plugin extensibility Apps
10. Copilot stack AI infrastructure Foundation models AI safety BYO models Hosted fine-tuned foundation models Hosted foundation models Plugin extensibility Plugin execution Grounding Metaprompt Prompt & response filtering Orchestration Copilot frontend UX
11. Foundation models AI safety BYO models Hosted fine-tuned foundation models Hosted foundation models Plugin extensibility Copilot frontend UX Orchestration frameworks Copilot stack Plugin execution Grounding Orchestration Metaprompt Prompt & response filtering
12. Foundation models AI safety Plugin extensibility Plugin execution Grounding Orchestration Copilot frontend UX Metaprompt Prompt & response filtering Prompt and Metaprompt Copilot stack
13. Metaprompt Foundation models AI safety BYO models Hosted fine-tuned foundation models Hosted foundation models Plugin extensibility Plugin execution Orchestration UX AI infrastructure Vector databases • Web APIs • Plugins Prompt & response filtering Copilot stack Retrieval Augmented Generation (RAG) Grounding
14. Foundation models AI safety Grounding Metaprompt Prompt & response filtering Orchestration Copilot frontend UX Plugin execution Plugin extensibility Copilot stack Plugins
15. AI safety Plugin execution Grounding Metaprompt Prompt & response filtering AI infrastructure Foundation models BYO models Hosted fine-tuned foundation models Hosted foundation models Copilot stack Foundation models and fine-tuning
16. AI infrastructure Foundation models AI safety BYO models Hosted fine-tuned foundation models Hosted foundation models Plugin extensibility Plugin execution Grounding Metaprompt Prompt & response filtering Orchestration Copilot frontend UX Copilot stack
17. Copilots New development pattern Unique architecture Will be everywhere
18. Azure AI Applied AI Services Bot Service Cognitive Search Form Recognizer Video Indexer Metrics Advisor Immersive Reader Cognitive Services Vision Speech Language Decision Azure OpenAI Service Azure Machine Learning Prepare & Preprocess Build, Train & Consume Deploy & Scale Manage & Monitor AI Infrastructure
19. Announcing Azure AI Studio Build and train your own models Ground Azure OpenAI Service and OSS models using your data Built-in vector indexing Retrieval augmented generation made easy Create prompt workflows AI safety built-in
20. Generative AI Applications Azure Machine Learning Native OSS Model Catalog Prompt Eng/Eval Prompt flow Responsible AI Azure Content Safety High Scale Gen AI App Deployment Gen AI Model Monitoring
21. Radically changing the art of possible with Azure OpenAI Service Large pretrained foundation AI models custom-tunable with your parameters and your data Summarization Reasoning over data Writing tools Code generation ChatGPT The Era of Copilots GPT-3 (GA) DALL•E 2 (preview) ChatGPT (GA) GPT-4 (GA) Foundation of enterprise security, privacy and compliance
22. Enterprise innovation on Azure OpenAI Service
23. https://azure.microsoft.com/en-us/blog/mercedes-benz-enhances-drivers-experience-with-azure-openai-service/
24. Update Bookmark ‘what's new’ in Azure OpenAI Service Sign up for Azure OpenAI Service updates Azure OpenAI Service GPT-3 (GA) DALL·E 2 (preview) ChatGPT (GA) GPT-4 (GA) Apply your own data Available in Preview early June now Plugins for Azure OpenAI Service Coming soon Configurable Content Filters Available in Preview early June now Provisioned Throughput Limited Availability early June
25. RAG: LLMs + your data Retrieval Augmented Generation
26. Anatomy of a RAG app App UX Orchestrator Retriever over Knowledge Base Query → Knowledge Prompt + Knowledge → Response Large Language Model Build your own experience UX, orchestration, calls to retriever and LLM e.g., Copilots, in-app chat Extend other app experiences Plugins for retrieval, symbolic math, app integration, etc. e.g., plugins for OpenAI ChatGPT
27. Retrievers: Externalizing Knowledge “Find the most relevant snippets in a large data collection, using unstructured input as query” == search engine App UX Orchestrator Azure OpenAI Azure Cognitive Search Data Sources (files, databases, etc.) Query → Knowledge Prompt + Knowledge → Response Azure Cognitive Search  Azure’s complete retrieval solution  Data ingestion, enterprise-grade security, partitioning and replication for scaling, support for 50+ written languages, and more
28. Retrieving Using Semantic Similarity Vector representations (or embeddings)  Learned such that “close” vectors represent items with similar meaning  May encode words, sentences, images, audio, etc.  Some map multiple media types into the same space  Azure OpenAI embeddings API, OSS embeddings (e.g., SBERT, CLIP)
29. Vector-based Retrieval Encoding (vectorizing)  Pre-process and encode content during ingestion  Encode queries during search/retrieval Vector indexing  Store and index lots of n-dimensional vectors  Quickly retrieve K closest to a “query” vector  Exhaustive search impractical in most cases  Approximate nearest neighbor (ANN) search Embedding [0.023883354, 0.021508986, 0.044205155, 0.019588541, 0.031198505, …]
30. Vector Search in Azure Cognitive Search New vector type for index fields  Users indicate vector size, distance function, algorithm and algo-specific parameters Pure Vector Search & Hybrid Search  Filters, faceting, etc. all works with vectors  Integrates with existing search indexes  Existing data ingestion and augmentation machinery entirely applicable Combines well with L2 re-ranker powered by Bing’s models  Enables improved ranking for hybrid search scenarios  L1: keywords + vector retrieval  L2: Bing’s ranker refreshed with GPT-enhanced work Enterprise-grade  Scalability (partitioning, replication)  Security: network isolation, managed identities, RBAC, etc.
31. Revolutionizing Indexing and Retrieval for LLM-powered Apps Power your retrieval-augmented generation applications Images Audio Video Graphs Documents • Use vector or hybrid search • Use Azure OpenAI embeddings or bring your own • Deeply integrate with Azure • Scale with replication and partitioning • Build generative AI apps and retrieval plugins Sign up today https://aka.ms/VectorSearchSignUp
32. Azure OpenAI Service on your data
33. Chat Completions API Versatile interface use for all scenarios—not just chat Model adheres to instructions in “system” message It sets the behavioral guidelines for the model, including responsible AI steering Examples go into “User”, “Assistant” fields import openai openai.api_type = "azure” response = openai.ChatCompletion.create( engine ="gpt-4", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Who won the world series in 2020?"}, {"role": "assistant", "content": "The LA Dodgers won the World Series in 2020."}, {"role": "user", "content": "Where was it played?"} ] ) …
34. Using your data Challenge I want to reason over my very long documents, but the token limits are not sufficient Conventional solution Build a vector database Retrieve relevant data and construct prompt at run-time
35. Introducing: Azure OpenAI Service on your data (Preview) Data Sources (search, files, databases, storage etc.) Additional 3P Data Sources (future capabilities) Azure OpenAI Service on your data API & SDK App or Copilot agent
36. https://techcommunity.microsoft.com/t5/ai-cognitive-services-blog/introducing-azure-openai-service-on-your-data-in-public-preview/ba-p/3847000 https://www.youtube.com/watch?v=6SNfeVop4zM
38. Plugins Azure OpenAI Service Plugins
39. Expanding potential Challenges we wanted to address Accurate translation for wider range of languages—improve support for Asian and African languages Integrate vector databases and cloud data stores Use up-to-date information from the web
40. Introducing: Azure OpenAI Service Plugins (coming soon) Build powerful AI Copilots with secure access to Microsoft services Retrieve data with Azure Cognitive Search Translate >100 languages with Azure Translator Ground with recent info with Bing Search Extract structured data from Azure SQL Azure OpenAI Plugins • Securely access your data in various data stores, vector databases and the web • Data path access controlled via Azure AD and Managed Identities • Admin roles to choose what plugins to enable
41. Recap Bookmark ‘what's new’ in Azure OpenAI Service Sign up for Azure OpenAI Service updates Azure OpenAI Service GPT-3 (GA) DALL·E 2 (preview) ChatGPT (GA) GPT-4 (GA) Apply your own data Available in Preview early June Plugins for Azure OpenAI Service Coming soon Configurable Content Filters Available in Preview early June Provisioned Throughput Limited Availability early June
42. Resources  Microsoft Build (2023/05/23-25)  https://build.microsoft.com/  DEEP LEARNING LAB - [Recap] Microsoft Build 2023 最新アップデー トAnalytics&AI (2023/06/08)  https://dllab.connpass.com/event/284571/  Microsoft Build 2023 Azure AI&ML 最新アップデート - Speaker Deck  https://speakerdeck.com/shohei1029/microsoft-build-2023-azure-ai-and-ml-zui-xin-atupudeto  Microsoft Build Japan (2023/06/27‐28)  https://info.microsoft.com/JA-ADAI-CATALOG-FY23-06Jun-28-Microsoft-Build-Japan-Day2- SREVM14500_Catalog-Display-Page.html
43. © Copyright Microsoft Corporation. All rights reserved.