Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Large Language Models, Daten & APIs: Integration von Generative AI Power – mit Python & .NET

Large Language Models, Daten & APIs: Integration von Generative AI Power – mit Python & .NET

Jenseits des ChatGPT-Hypes taucht Christian in die Welt der Large Language Models (LLMs), Daten und APIs ein und konzentriert sich darauf, wie Sie AI-Funktionalität sinnvoll in Ihre Anwendungenintegrieren können. Wir werden pragmatische Szenarien und Use Cases untersuchen, die das Potenzial von LLMs (wie GPT oder Llama) demonstrieren - und erörtern, wie AI-Techniken in bestehende Architekturen einbezogen werden können. Die Teilnehmer erhalten erste Einblicke sowohl in LangChain als Python-Toolkit als auch Semantic Kernel als .NET-basiertes SDK.
Zudem werden wir darauf eingehen, nicht nur Closed-Source-Systeme (wie OpenAI) zu nutzen, sondern auch Open-Source-Optionen in Betracht zu ziehen, um unterschiedlichen Anforderungen gerecht werden zu können.

Christian Weyer

October 05, 2023
Tweet

More Decks by Christian Weyer

Other Decks in Programming

Transcript

  1. Large Language Models, Daten & APIs:
    Integration von Generative AI Power
    – mit Python & .NET
    Christian Weyer
    @christianweyer
    CTO, Technology Catalyst

    View full-size slide

  2. § What & why?
    § LLM integration aspects
    § Selected use cases
    § Agents & LLMs
    § LLM landscape
    § Current state & Outlook
    Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    Talking points
    2

    View full-size slide

  3. § Technology catalyst
    § AI-powered solutions
    § Pragmatic end-to-end architectures
    § Microsoft Regional Director
    § Microsoft MVP for Developer Technologies & Azure
    ASPInsider, AzureInsider
    § Google GDE for Web Technologies
    [email protected] @christianweyer https://www.thinktecture.com
    Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    Christian Weyer
    Co-Founder & CTO @ Thinktecture AG
    3

    View full-size slide

  4. Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    Intro
    4

    View full-size slide

  5. Speech
    OpenAI Whisper
    tortoise-tts

    Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    Generative AI all-the-things
    Images
    Midjourney
    DALL·E

    Audio/Music
    Musico
    Soundraw

    Text
    OpenAI GPT
    Anthropic Claude
    Codex
    Llama

    5

    View full-size slide

  6. Speech
    OpenAI Whisper
    tortoise-tts

    Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    Generative AI all-the-things
    Images
    Midjourney
    DALL·E

    Audio/Music
    Musico
    Soundraw

    Text
    OpenAI GPT
    Anthropic Claude
    Codex
    Llama

    6
    P

    View full-size slide

  7. § LLMs generate text based on input
    § LLMs can understand text – this changes a lot
    § Prompts are the universal interface (“UI”) →
    unstructured text with semantics
    § Human language evolves as a first-class citizen in
    software architecture 🤯
    * LLMs are not “perfect” – errors may occur, caveats like non-determinism & hallucination – these are topics to be dealt with
    Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    Text… – really, just text?
    7
    Yeah: Large Language Models (LLMs)

    View full-size slide

  8. “Hm… what … how… erm… who could know this…? 🤔”
    Learning about my company’s policies in Slack
    Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    DEMO
    8
    Demos are using OpenAI GPT-3.5-turbo or GPT-4

    View full-size slide

  9. § LLMs are programs
    § LLMs are highly specialized neural networks
    § LLMs use(d) lots of data
    § LLMs need a lot of resources to be operated
    § LLMs have an API to be used through
    Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    Large Language Models demystified
    9

    View full-size slide

  10. Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    Integration
    10

    View full-size slide

  11. Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    Using LLMs: It’s just APIs !
    Inference, FTW.
    11

    View full-size slide

  12. GPT-4 API access via OpenAI Playground
    Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    DEMO
    12

    View full-size slide

  13. Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    The best tool for .NET developers to talk to LLMs!
    13
    🙈

    View full-size slide

  14. § Most popular open-source framework for developing applications
    powered by LLMs
    § Several releases per week
    § > 1000 contributors
    § Python and Typescript versions
    § Chains as a central concept
    § Sequences of actions in code
    § Abstractions for
    § Loading text from a wide range of sources
    § Vector stores
    § Prompts & LLMs (local and remote)
    § Tools and Agents
    Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    LangChain
    14

    View full-size slide

  15. § Microsoft’s open-source framework to integrate LLMs into
    applications
    § .NET, Python, and Java versions
    § Plugins encapsulate AI capabilities into a single unit of functionality
    § Semantic functions for prompting
    § Native functions to run local code
    § Chain is collection of Plugins
    § Planners are similar to Agents in LangChain
    § Not as broad feature set as LangChain
    § E.g., no concept/abstraction for loading data
    Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    Semantic Kernel
    15

    View full-size slide

  16. Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    Use cases
    16

    View full-size slide

  17. Text generation
    § LLMs are good in generating text
    § Regular text
    § Code
    § SQL (beware!)
    § JSON
    § etc.
    Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    Typical LLM use cases
    17

    View full-size slide

  18. Generating text & images (multimodal) (Semantic Kernel)
    Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    DEMO
    18

    View full-size slide

  19. Understanding text
    § LLM can be instructed to e.g.
    § do sentiment analysis
    § extract information from text
    § Tools (like Kor, Open AI Functions – or TypeChat) for
    extracting information in a structured way reliably
    § JSON, Python classes, TypeScript types etc.
    Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    Typical LLM use cases
    19

    View full-size slide

  20. Extracting structured data (LangChain + Kor)
    Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    DEMO
    20

    View full-size slide

  21. Answering Questions / Chat with documents
    § Mixed usage of text embedding and a large language model
    § Retrieval-augmented generation (RAG)
    § Index one or more documents
    § Use text embedding to index documents
    § Store embeddings in a vector database
    § User asks question
    § Create embedding for the question
    § Search vector database to find relevant text
    § Use LLM with relevant input text & question to create an answer
    Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    Typical LLM use cases
    21

    View full-size slide

  22. Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    Answering Questions / Chat with documents - RAG
    Cleanup & Split Text Embedding
    Question
    Text Embedding
    Save
    Query
    Relevant Text
    Question
    Answer
    LLM
    22
    Vector DB
    Embedding
    model
    Embedding
    model 💡
    Indexing / Embedding QA

    View full-size slide

  23. Chat with web site documents (Semantic Kernel)
    Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    DEMO
    23

    View full-size slide

  24. Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    Agents
    24

    View full-size slide

  25. § Involve an LLM making decisions (via ReAct – Reasoning & Acting)
    § Which actions to take (thought)
    § Taking that action (executed via your code)
    § Seeing an observation
    § Repeating until done
    § Tools to interact with the workflow from the LLM
    § LangChain tools and agents
    § Semantic Kernel native functions
    § OpenAI function calling
    § Reliable (structured) way to interact from the LLM with your code
    § Proprietary to OpenAI models (fine-tuned)
    Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    Agents: Multi-steps workflows with LLM interaction
    25

    View full-size slide

  26. Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    ReAct – with function calling
    26
    LLM
    My code
    Query
    Some API Some database
    Prompt Function
    definitions
    Final answer
    Answer



    💡

    View full-size slide

  27. Stock market agent
    with OpenAI function calling (Python OpenAI SDK)
    Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    DEMO
    27

    View full-size slide

  28. Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    Landscape
    28

    View full-size slide

  29. § Open AI
    § Azure Open AI Service
    § Google Model Garden on Vertex AI
    § Amazon Sagemaker
    § Hugging Face
    § Fermyon Serverless AI
    § Self-hosting (hard to scale)
    § Device / browser
    Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    LLM providers
    29

    View full-size slide

  30. Running an open-source LLM offline in the browser
    (with Wasm)
    (web-llm)
    Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    DEMO
    30

    View full-size slide

  31. Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    Wrap-Up
    31

    View full-size slide

  32. § Fast moving and changing field
    § Every week something “big”
    happens in LLM space
    § Frameworks are evolving together
    with LLMs
    § Closed vs open LLMs
    § Competition drives invention &
    advancement
    § Bias of the models
    § Which data was the model trained
    on?
    § Finding the right prompt can be
    tedious
    § Prompting differs between LLMs
    § Prompt injection as a threat
    § Will improve over time
    § Context size is currently limited
    § Will likely diminish over time
    § SISO (sh*t in, sh*t out)
    § Quality of results heavily depends
    on your data
    Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    Current state
    32

    View full-size slide

  33. Huge potential for
    LLM-driven
    human-machine workflows
    via UI (universal interface) agents
    Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    Outlook
    33
    P

    View full-size slide

  34. https://www.thinktecture.com/ueber-uns/karriere/
    Dankeschön!
    Christian Weyer
    https://thinktecture.com/christian-weyer
    34

    View full-size slide

  35. § Semantic Kernel
    § https://learn.microsoft.com/en-us/semantic-kernel/overview/
    § LangChain
    § https://www.langchain.com/
    § LangChain Agents
    § https://python.langchain.com/docs/modules/agents/
    § ReAct: Synergizing Reasoning and Acting in Language Models
    § https://react-lm.github.io/
    § Prompt Engineering Guide
    § https://www.promptingguide.ai/
    § OpenAI API reference
    § https://platform.openai.com/docs/api-reference
    § Azure OpenAI Service REST API reference
    § https://learn.microsoft.com/en-us/azure/ai-services/openai/reference
    § Hugging Face Inference Endpoints (for various OSS LLMs)
    § https://huggingface.co/docs/inference-endpoints/api_reference
    § Llama2 on Hugging Face
    § https://huggingface.co/blog/llama2
    § OWASP Top 10 for LLM Applications
    § https://owasp.org/www-project-top-10-for-large-language-model-applications/assets/PDF/OWASP-Top-10-for-LLMs-2023-slides-v1_0_1.pdf
    Large Language Models, Daten & APIs
    Integration von Generative AI Power - mit Python & .NET
    Links
    35

    View full-size slide