Upgrade to Pro — share decks privately, control downloads, hide ads and more …

.NET Day 2023: LLMs, Data & APIs: Integrating AI power into your .NET applications with Semantic Kernel

dotnetday
August 31, 2023

.NET Day 2023: LLMs, Data & APIs: Integrating AI power into your .NET applications with Semantic Kernel

Let's look beyond the ChatGPT hype! In this session, Boris & Christian delve into the world of Large Language Models (LLMs), data, and APIs - focusing on how to integrate AI capabilities seamlessly into your .NET-based applications. We will explore practical use cases that demonstrate the potential of LLMs (such as GPT) and discuss how to incorporate AI techniques into existing applications and architectures. Attendees will gain insights into Microsoft's Semantic Kernel tooling for .NET developers, as well as a glimpse into alternative solutions. Lastly, we will address the importance of not only leveraging cloud-based and closed-source systems (like OpenAI) but also considering on-premises and open-source options to accommodation.

dotnetday

August 31, 2023
Tweet

More Decks by dotnetday

Other Decks in Technology

Transcript

  1. LLMs, Data & APIs:
    Integrating AI power into your applications
    (with Semantic Kernel & LangChain)
    Christian Weyer
    @christianweyer
    CTO, Technology Catalyst

    View full-size slide

  2. § What & why?
    § LLM integration aspects
    § Some use cases
    § Agents & LLMs
    § LLM landscape
    § Current state & Outlook
    LLMs, Data & APIs
    Integrating AI power into your applications
    Talking points
    2

    View full-size slide

  3. § Technology catalyst
    § AI-powered solutions
    § Pragmatic end-to-end architectures
    § Microsoft Regional Director
    § Microsoft MVP for Developer Technologies & Azure
    ASPInsider, AzureInsider
    § Google GDE for Web Technologies
    [email protected] @christianweyer https://www.thinktecture.com
    LLMs, Data & APIs
    Integrating AI power into your applications
    Christian Weyer
    Co-Founder & CTO @ Thinktecture AG
    3

    View full-size slide

  4. LLMs, Data & APIs
    Integrating AI power into your applications
    Intro
    4

    View full-size slide

  5. Speech
    OpenAI Whisper
    tortoise-tts

    LLMs, Data & APIs
    Integrating AI power into your applications
    Generative AI all-the-things
    Images
    Midjourney
    DALL·E

    Audio/Music
    Musico
    Soundraw

    Text
    OpenAI GPT
    Anthropic Claude
    Codex
    Llama

    5

    View full-size slide

  6. Speech
    OpenAI Whisper
    tortoise-tts

    LLMs, Data & APIs
    Integrating AI power into your applications
    Generative AI all-the-things
    Images
    Midjourney
    DALL·E

    Audio/Music
    Musico
    Soundraw

    Text
    OpenAI GPT
    Anthropic Claude
    Codex
    Llama

    6

    View full-size slide

  7. § Large Language Models (LLMs) generate text based on input
    § Plus, LLMs can understand text – this changes a lot
    § Prompts are the universal interface (“UI”) → unstructured
    text with semantics
    § Human language as a first-class citizen in
    software architecture 🤯
    * LLMs are not perfect – errors may occur, caveats like non-determinism & hallucination – these are topics to be dealt with
    LLMs, Data & APIs
    Integrating AI power into your applications
    Text… – really, just text?
    7
    Yeah: Large Language Models

    View full-size slide

  8. “Hm… what … how… erm… who could know this…? 🤔”
    Learning about my company’s policies in Slack
    LLMs, Data & APIs
    Integrating AI power into your applications
    DEMO
    8
    Demos are using OpenAI GPT-3.5-turbo or GPT-4

    View full-size slide

  9. § LLMs are programs
    § LLMs are highly specialized neural networks
    § LLMs use(d) lots of data
    § LLMs need a lot of resources to be operated
    § LLMs have an API to be used through
    LLMs, Data & APIs
    Integrating AI power into your applications
    Large Language Models demystified
    9

    View full-size slide

  10. LLMs, Data & APIs
    Integrating AI power into your applications
    Integration
    10

    View full-size slide

  11. LLMs, Data & APIs
    Integrating AI power into your applications
    Using LLMs: It’s just APIs !
    11

    View full-size slide

  12. LLMs, Data & APIs
    Integrating AI power into your applications
    The best tool for .NET developers to talk to LLMs!
    12
    🙈

    View full-size slide

  13. § Most popular open-source framework for developing applications
    powered by LLMs
    § Several releases per week
    § > 1000 contributors
    § Python and Typescript versions
    § Chains as a central concept
    § Sequences of actions in code
    § Abstractions for
    § Loading text from a wide range of sources
    § Vector stores
    § Prompts & LLMs (local and remote)
    § Tools and Agents
    LLMs, Data & APIs
    Integrating AI power into your applications
    LangChain
    13

    View full-size slide

  14. § Microsoft’s open-source framework to integrate LLMs into
    applications
    § .NET, Python, and Java versions
    § Plugins encapsulate AI capabilities into a single unit of functionality
    § Semantic functions for prompting
    § Native functions to run local code
    § Not as broad feature set as LangChain
    § E.g., no concept/abstraction for loading text
    § .NET version has no support to run a model locally (yet?)
    § Can only call (Azure) OpenAI API and Hugging Face Inference API
    LLMs, Data & APIs
    Integrating AI power into your applications
    Semantic Kernel
    14

    View full-size slide

  15. LLMs, Data & APIs
    Integrating AI power into your applications
    Use cases
    15

    View full-size slide

  16. Text generation
    § LLMs are good in generating text
    § Regular text
    § Code
    § SQL (beware!)
    § JSON
    § etc.
    LLMs, Data & APIs
    Integrating AI power into your applications
    Typical LLM use cases
    16

    View full-size slide

  17. Understanding text
    § LLMs understand a given text
    § LLM can be instructed to e.g., extract information out of the
    text
    § Tools like Kor or Open AI Functions can be used to extract
    information in a structured way
    § JSON, Python classes, etc.
    LLMs, Data & APIs
    Integrating AI power into your applications
    Typical LLM use cases
    17

    View full-size slide

  18. Extracting structured data (LangChain)
    LLMs, Data & APIs
    Integrating AI power into your applications
    DEMO
    18

    View full-size slide

  19. Answering Questions / Chat with documents
    § Mixed usage of text embedding and a large language model
    § Retrieval-augmented generation (RAG)
    § Index one or more documents
    § Use text embedding to index documents
    § Store embeddings in a vector database
    § User asks question
    § Create embedding for the question
    § Search vector database to find relevant text
    § Use LLM with relevant input text & question to create an answer
    LLMs, Data & APIs
    Integrating AI power into your applications
    Typical LLM use cases
    19

    View full-size slide

  20. LLMs, Data & APIs
    Integrating AI power into your applications
    Answering Questions / Chat with documents - Architecture
    Cleanup & Split Text Embedding
    Question
    Text Embedding
    Save
    Query
    Relevant Text
    Question
    Answer
    LLM
    20
    Vector DB
    Embedding
    model
    Embedding
    model 💡
    Indexing QA

    View full-size slide

  21. Chat with web site documents (Semantic Kernel)
    LLMs, Data & APIs
    Integrating AI power into your applications
    DEMO
    21

    View full-size slide

  22. LLMs, Data & APIs
    Integrating AI power into your applications
    Agents
    22

    View full-size slide

  23. § Involve an LLM making decisions (via ReAct – Reasoning & Acting)
    § Which actions to take (thought)
    § Taking that action (executed via your code)
    § Seeing an observation
    § Repeating until done
    § Tools or functions to interact with the workflow from the LLM
    § LangChain tools and agents
    § Semantic Kernel native functions
    § OpenAI function calling
    § Reliable (structured) way to interact from the LLM with your code
    § Proprietary to OpenAI models (fine-tuned)
    LLMs, Data & APIs
    Integrating AI power into your applications
    Agents: Multi-steps workflows with LLM interaction
    23

    View full-size slide

  24. LLMs, Data & APIs
    Integrating AI power into your applications
    ReAct – with function calling
    24
    LLM
    My code
    Query
    Some API Some database
    Prompt Function
    Definitions
    Final answer
    Answer



    💡

    View full-size slide

  25. Stock market agent
    with OpenAI function calling (LangChain)
    LLMs, Data & APIs
    Integrating AI power into your applications
    DEMO
    25

    View full-size slide

  26. LLMs, Data & APIs
    Integrating AI power into your applications
    Landscape
    26

    View full-size slide

  27. § Open AI
    § Azure Open AI Service
    § Google Model Garden on Vertex AI
    § Amazon Sagemaker
    § Hugging Face
    § Self-hosting (does not really scale)
    § Device / browser
    LLMs, Data & APIs
    Integrating AI power into your applications
    LLM providers
    27

    View full-size slide

  28. LLMs, Data & APIs
    Integrating AI power into your applications
    Wrap-Up
    28

    View full-size slide

  29. § Fast moving and changing field
    § Every week something “big”
    happens in LLM space
    § Frameworks are evolving together
    with LLMs
    § Closed vs open LLMs
    § Competition drives invention &
    advancement
    § Bias of the models
    § Which data was the model trained
    on?
    § Finding the right prompt can be
    tedious
    § Prompting differs between LLMs
    § Prompt injection as a threat
    § Will improve over time
    § Context size is currently limited
    § Will likely diminish over time
    § SISO (sh*t in, sh*t out)
    § Quality of results heavily depends
    on your data
    LLMs, Data & APIs
    Integrating AI power into your applications
    Current state
    29

    View full-size slide

  30. Huge potential for
    LLM-driven
    human-machine workflows
    via a UI (universal interface)
    LLMs, Data & APIs
    Integrating AI power into your applications
    Outlook
    30

    View full-size slide

  31. Thank you!
    Christian Weyer
    https://thinktecture.com/christian-weyer
    31

    View full-size slide

  32. § Semantic Kernel
    § https://learn.microsoft.com/en-us/semantic-kernel/overview/
    § LangChain
    § https://www.langchain.com/
    § LangChain Agents
    § https://python.langchain.com/docs/modules/agents/
    § ReAct: Synergizing Reasoning and Acting in Language Models
    § https://react-lm.github.io/
    § Prompt Engineering Guide
    § https://www.promptingguide.ai/
    § OpenAI API reference
    § https://platform.openai.com/docs/api-reference
    § Azure OpenAI Service REST API reference
    § https://learn.microsoft.com/en-us/azure/ai-services/openai/reference
    § Hugging Face Inference Endpoints (for various OSS LLMs)
    § https://huggingface.co/docs/inference-endpoints/api_reference
    § Llama2 on Hugging Face
    § https://huggingface.co/blog/llama2
    LLMs, Data & APIs
    Integrating AI power into your applications
    Links
    32

    View full-size slide