Microsoft Build 2023 Updates – Copilot Stack and Azure OpenAI Service (Machine Learning 15minutes! Broadcast #78)

第78回 Machine Learning 15minutes! Broadcast (2023/06/24)
https://machine-learning15minutes.connpass.com/event/285147/

S/N Ratio > Microsoft Build 2023 Updates – Copilot Stack and Azure OpenAI Service (Machine Learning 15minutes! Broadcast #78)
https://satonaoki.wordpress.com/2023/06/24/build-2023-updates-copilot-stack-azure-openai/

Transcript

1. Microsoft Build 2023
   Updates –
   Copilot Stack and
   Azure OpenAI Service
   SATO Naoki (Neo)
   Senior Software Engineer, Microsoft
   

   View Slide

2. +
   

   View Slide

3. World’s most
   powerful
   supercomputers
   World’s most capable
   foundation models, from
   hosted to open source
   World’s best
   AI dev
   infrastructure
   

   View Slide

4. Copilot
   

   View Slide

5. Announcing
   Windows Copilot
   

   View Slide

6. Announcing
   Copilot extensibility
   and plugins
   ChatGPT Bing Chat Microsoft Copilots Windows Copilot Your Copilots
   

   View Slide

7. Augment AI systems to access APIs
   Retrieve useful information
   Perform new computations
   Safely act on the user’s behalf
   Plugins
   

   View Slide

8. The digital world
   Plugins
   Copilot
   

   View Slide

9. Copilot
   stack
   AI infrastructure
   Foundation models
   AI orchestration
   Orchestration
   Copilots
   Plugin extensibility
   Apps
   

   View Slide

10. Copilot
    stack
    AI infrastructure
    Foundation models
    AI
    safety
    BYO models
    Hosted fine-tuned foundation models
    Hosted foundation models
    Plugin extensibility
    Plugin execution
    Grounding
    Metaprompt
    Prompt & response filtering
    Orchestration
    Copilot frontend
    UX
    

    View Slide

11. Foundation models
    AI
    safety
    BYO models
    Hosted fine-tuned foundation models
    Hosted foundation models
    Plugin extensibility
    Copilot frontend
    UX
    Orchestration frameworks
    Copilot stack
    Plugin execution
    Grounding
    Orchestration
    Metaprompt
    Prompt & response filtering
    

    View Slide

12. Foundation models
    AI
    safety
    Plugin extensibility
    Plugin execution
    Grounding
    Orchestration
    Copilot frontend
    UX
    Metaprompt
    Prompt & response filtering
    Prompt and Metaprompt
    Copilot stack
    

    View Slide

13. Metaprompt
    Foundation models
    AI
    safety
    BYO models
    Hosted fine-tuned foundation models
    Hosted foundation models
    Plugin extensibility
    Plugin execution
    Orchestration
    UX
    AI infrastructure
    Vector databases • Web APIs • Plugins
    Prompt & response filtering
    Copilot stack
    Retrieval Augmented Generation
    (RAG)
    Grounding
    

    View Slide

14. Foundation models
    AI
    safety
    Grounding
    Metaprompt
    Prompt & response filtering
    Orchestration
    Copilot frontend
    UX
    Plugin execution
    Plugin extensibility
    Copilot stack
    Plugins
    

    View Slide

15. AI
    safety
    Plugin execution
    Grounding
    Metaprompt
    Prompt & response filtering
    AI infrastructure
    Foundation models
    BYO models
    Hosted fine-tuned foundation models
    Hosted foundation models
    Copilot stack
    Foundation models and fine-tuning
    

    View Slide

16. AI infrastructure
    Foundation models
    AI
    safety
    BYO models
    Hosted fine-tuned foundation models
    Hosted foundation models
    Plugin extensibility
    Plugin execution
    Grounding
    Metaprompt
    Prompt & response filtering
    Orchestration
    Copilot frontend
    UX
    Copilot
    stack
    

    View Slide

17. Copilots
    New development
    pattern
    Unique
    architecture
    Will be
    everywhere
    

    View Slide

18. Azure AI
    Applied AI Services
    Bot Service Cognitive Search Form Recognizer Video Indexer Metrics Advisor Immersive Reader
    Cognitive Services
    Vision Speech Language Decision Azure OpenAI Service
    Azure Machine Learning
    Prepare & Preprocess Build, Train & Consume Deploy & Scale Manage & Monitor
    AI Infrastructure
    

    View Slide

19. Announcing
    Azure AI Studio
    Build and
    train your
    own models
    Ground Azure
    OpenAI Service
    and OSS models
    using your data
    Built-in
    vector
    indexing
    Retrieval
    augmented
    generation
    made easy
    Create
    prompt
    workflows
    AI safety
    built-in
    

    View Slide

20. Generative AI Applications
    Azure Machine Learning
    Native OSS
    Model Catalog
    Prompt Eng/Eval
    Prompt flow
    Responsible AI
    Azure Content Safety
    High Scale Gen AI
    App Deployment
    Gen AI Model
    Monitoring
    

    View Slide

21. Radically changing the art of possible
    with Azure OpenAI Service
    Large pretrained foundation AI models
    custom-tunable with your parameters and your data
    Summarization
    Reasoning over data
    Writing tools
    Code generation
    ChatGPT
    The Era of Copilots
    GPT-3 (GA)
    DALL•E 2 (preview)
    ChatGPT (GA)
    GPT-4 (GA)
    Foundation of enterprise security, privacy and compliance
    

    View Slide

22. Enterprise innovation on Azure OpenAI Service
    

    View Slide

23. https://azure.microsoft.com/en-us/blog/mercedes-benz-enhances-drivers-experience-with-azure-openai-service/
    

    View Slide

24. Update
    Bookmark ‘what's
    new’ in Azure OpenAI
    Service
    Sign up for Azure
    OpenAI Service
    updates
    Azure OpenAI Service
    GPT-3 (GA) DALL·E 2 (preview)
    ChatGPT (GA) GPT-4 (GA)
    Apply your
    own data
    Available in Preview
    early June now
    Plugins for Azure
    OpenAI Service
    Coming soon
    Configurable
    Content Filters
    Available in Preview
    early June now
    Provisioned
    Throughput
    Limited Availability
    early June
    

    View Slide

25. RAG: LLMs + your data
    Retrieval Augmented Generation
    

    View Slide

26. Anatomy of a RAG app
    App UX Orchestrator
    Retriever over Knowledge Base
    Query →
    Knowledge
    Prompt + Knowledge →
    Response
    Large Language Model
    Build your own experience
    UX, orchestration, calls to retriever and LLM
    e.g., Copilots, in-app chat
    Extend other app experiences
    Plugins for retrieval, symbolic math,
    app integration, etc.
    e.g., plugins for OpenAI ChatGPT
    

    View Slide

27. Retrievers: Externalizing Knowledge
    “Find the most relevant snippets
    in a large data collection, using
    unstructured input as query”
    == search engine
    App UX Orchestrator
    Azure OpenAI
    Azure Cognitive Search
    Data Sources
    (files, databases, etc.)
    Query →
    Knowledge
    Prompt + Knowledge
    → Response
    Azure Cognitive Search
     Azure’s complete retrieval solution
     Data ingestion, enterprise-grade
    security, partitioning and replication
    for scaling, support for 50+ written
    languages, and more
    

    View Slide

28. Retrieving Using Semantic Similarity
    Vector representations
    (or embeddings)
     Learned such that “close” vectors
    represent items with similar meaning
     May encode words, sentences, images,
    audio, etc.
     Some map multiple media types into the same space
     Azure OpenAI embeddings API, OSS
    embeddings (e.g., SBERT, CLIP)
    

    View Slide

29. Vector-based Retrieval
    Encoding (vectorizing)
     Pre-process and encode
    content during ingestion
     Encode queries during search/retrieval
    Vector indexing
     Store and index lots of n-dimensional vectors
     Quickly retrieve K closest to a “query” vector
     Exhaustive search impractical in most cases
     Approximate nearest neighbor (ANN) search
    Embedding
    [0.023883354, 0.021508986,
    0.044205155, 0.019588541,
    0.031198505, …]
    

    View Slide

30. Vector Search in Azure Cognitive Search
    New vector type for index fields
     Users indicate vector size, distance function, algorithm and algo-specific parameters
    Pure Vector Search & Hybrid Search
     Filters, faceting, etc. all works with vectors
     Integrates with existing search indexes
     Existing data ingestion and augmentation machinery entirely applicable
    Combines well with L2 re-ranker powered by Bing’s models
     Enables improved ranking for hybrid search scenarios
     L1: keywords + vector retrieval
     L2: Bing’s ranker refreshed with GPT-enhanced work
    Enterprise-grade
     Scalability (partitioning, replication)
     Security: network isolation, managed identities, RBAC, etc.
    

    View Slide

31. Revolutionizing Indexing and
    Retrieval for LLM-powered Apps
    Power your retrieval-augmented generation applications
    Images Audio Video Graphs Documents
    • Use vector or hybrid search
    • Use Azure OpenAI embeddings or bring your own
    • Deeply integrate with Azure
    • Scale with replication and partitioning
    • Build generative AI apps and retrieval plugins
    Sign up today https://aka.ms/VectorSearchSignUp
    

    View Slide

32. Azure OpenAI Service on your data
    

    View Slide

33. Chat Completions API
    Versatile interface use for all scenarios—not just chat
    Model adheres to instructions in “system” message
    It sets the behavioral guidelines for the model, including responsible AI steering
    Examples go into “User”, “Assistant” fields
    import openai
    openai.api_type = "azure”
    response = openai.ChatCompletion.create(
    engine ="gpt-4",
    messages=[
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Who won the world series in 2020?"},
    {"role": "assistant", "content": "The LA Dodgers won the World Series in 2020."},
    {"role": "user", "content": "Where was it played?"}
    ]
    )
    …
    

    View Slide

34. Using your data
    Challenge I want to reason over my very
    long documents, but the token
    limits are not sufficient
    Conventional solution Build a vector database
    Retrieve relevant data and
    construct prompt at run-time
    

    View Slide

35. Introducing:
    Azure OpenAI Service on your data (Preview)
    Data Sources
    (search, files, databases, storage etc.)
    Additional 3P Data Sources
    (future capabilities)
    Azure OpenAI Service
    on your data
    API & SDK
    App or
    Copilot agent
    

    View Slide

36. https://techcommunity.microsoft.com/t5/ai-cognitive-services-blog/introducing-azure-openai-service-on-your-data-in-public-preview/ba-p/3847000 https://www.youtube.com/watch?v=6SNfeVop4zM
    

    View Slide

37. View Slide

38. Plugins
    Azure OpenAI Service Plugins
    

    View Slide

39. Expanding potential
    Challenges we wanted to address
    Accurate translation
    for wider range of
    languages—improve
    support for Asian and
    African languages
    Integrate vector
    databases and cloud
    data stores
    Use up-to-date
    information from
    the web
    

    View Slide

40. Introducing:
    Azure OpenAI Service Plugins (coming soon)
    Build powerful AI Copilots with secure access to Microsoft services
    Retrieve data with Azure
    Cognitive Search
    Translate >100 languages
    with Azure Translator
    Ground with recent info
    with Bing Search
    Extract structured data
    from Azure SQL
    Azure OpenAI
    Plugins
    • Securely access your data in
    various data stores, vector
    databases and the web
    • Data path access controlled via
    Azure AD and Managed Identities
    • Admin roles to choose
    what plugins to enable
    

    View Slide

41. Recap
    Bookmark ‘what's
    new’ in Azure OpenAI
    Service
    Sign up for Azure
    OpenAI Service
    updates
    Azure OpenAI Service
    GPT-3 (GA) DALL·E 2 (preview)
    ChatGPT (GA) GPT-4 (GA)
    Apply your
    own data
    Available in Preview
    early June
    Plugins for Azure
    OpenAI Service
    Coming soon
    Configurable
    Content Filters
    Available in Preview
    early June
    Provisioned
    Throughput
    Limited Availability
    early June
    

    View Slide

42. Resources
     Microsoft Build (2023/05/23-25)
     https://build.microsoft.com/
     DEEP LEARNING LAB - [Recap] Microsoft Build 2023 最新アップデー
    トAnalytics&AI (2023/06/08)
     https://dllab.connpass.com/event/284571/
     Microsoft Build 2023 Azure AI&ML 最新アップデート - Speaker Deck
     https://speakerdeck.com/shohei1029/microsoft-build-2023-azure-ai-and-ml-zui-xin-atupudeto
     Microsoft Build Japan (2023/06/27‐28)
     https://info.microsoft.com/JA-ADAI-CATALOG-FY23-06Jun-28-Microsoft-Build-Japan-Day2-
    SREVM14500_Catalog-Display-Page.html
    

    View Slide

43. © Copyright Microsoft Corporation. All rights reserved.
    

    View Slide