Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Large Language Models, Daten & APIs: Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.

Large Language Models, Daten & APIs: Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.

Jajaja, der ChatGPT-Hype... Jenseits dessen taucht Christian in die Welt der Large Language Models (LLMs), Daten und APIs ein und konzentriert sich darauf, wie Sie AI-Funktionalität sinnvoll in Ihre Anwendungen integrieren können. Wir werden pragmatische Anwendungsfälle untersuchen, die das Potenzial von LLMs (wie GPT) demonstrieren - und erörtern, wie AI-Techniken in bestehende Architekturen einbezogen werden können. Die Teilnehmer erhalten Einblicke in LangChain als Python-Toolkit und die Möglichkeiten der Integration von LLMs. Zudem werden wir uns damit befassen, nicht nur Cloud-basierte und Closed-Source-Systeme (wie OpenAI) zu nutzen, sondern auch On-Premises- und Open-Source-Optionen in Betracht zu ziehen, um unterschiedlichen organisatorischen Anforderungen gerecht zu werden. Come in and find out!

Christian Weyer

November 08, 2023
Tweet

More Decks by Christian Weyer

Other Decks in Programming

Transcript

  1. Large Language Models, Daten & APIs:
    Integration von Generative-AI-Power
    in Ihre Anwendungen
    mit LangChain & Co.
    Christian Weyer
    @christianweyer
    CTO, Technology Catalyst

    View full-size slide

  2. § What & why?
    § LLM integration aspects
    § Selected use cases
    § Agents & LLMs
    § LLM landscape
    § Current state & Outlook
    Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    Talking points
    2

    View full-size slide

  3. § Technology catalyst
    § AI-powered solutions
    § Pragmatic end-to-end architectures
    § Microsoft Regional Director
    § Microsoft MVP for Developer Technologies & Azure
    ASPInsider, AzureInsider
    § Google GDE for Web Technologies
    [email protected] @christianweyer https://www.thinktecture.com
    Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    Christian Weyer
    Co-Founder & CTO @ Thinktecture AG
    3

    View full-size slide

  4. Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    Intro
    4

    View full-size slide

  5. Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    What’s in a name?
    5

    View full-size slide

  6. § LLMs generate text based on input
    § LLMs can understand text – this changes a lot
    § Prompts are the universal interface (“UI”) →
    unstructured text with semantics
    § Human language evolves as a first-class citizen in
    software architecture 🤯
    * LLMs are not “perfect” – errors may occur, caveats like non-determinism & hallucination – these are topics to be dealt with
    Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    Text… – really, just text?
    6
    Yeah: Large Language Models (LLMs)

    View full-size slide

  7. § LLMs are programs
    § LLMs are highly specialized neural networks
    § LLMs use(d) lots of data
    § LLMs need a lot of resources to be operated
    § LLMs have an API to be used through
    Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    Large Language Models demystified
    7

    View full-size slide

  8. Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    Integration
    8

    View full-size slide

  9. Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    Using LLMs: It’s just APIs!
    Inference, FTW.
    9

    View full-size slide

  10. GPT-4 API access via OpenAI Playground
    Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    DEMO
    10
    Demos are using OpenAI GPT-3.5-turbo or GPT-4(-Turbo)

    View full-size slide

  11. Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    The best tool for Java developers to talk to LLMs!
    11
    🙈

    View full-size slide

  12. § Open-source framework for developing applications powered by LLMs
    § > 1000 contributors
    § Python and Typescript versions
    § Chains as a central concept
    § Sequences of LLM-related actions in code
    § Abstractions for
    § Prompts & LLMs (local and remote)
    § Memory
    § Vector stores
    § Tools
    § Loading text from a wide range of sources
    Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    LangChain
    12

    View full-size slide

  13. § No official Java version of LangChain available
    § LangChain4J is community-driven
    § Works with Java 8 or higher
    § Supports Spring Boot 2 and 3
    Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    LangChain4J
    13

    View full-size slide

  14. Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    Use cases
    14

    View full-size slide

  15. Text generation
    § LLMs are good in generating text
    § Regular text
    § Code
    § SQL (beware!)
    § JSON
    § etc.
    Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    Typical LLM use cases
    15

    View full-size slide

  16. Understanding text
    § LLM can be instructed to e.g.
    § do sentiment analysis
    § extract information from text
    § Tools (like Kor, Open AI Functions – or TypeChat) for extracting
    information in a structured way reliably
    § JSON, Python classes, TypeScript types etc.
    Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    Typical LLM use cases
    16

    View full-size slide

  17. Extracting structured data (LangChain + Kor)
    Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    DEMO
    17

    View full-size slide

  18. Answering Questions / Chat with documents
    § Mixed usage of text embedding and a large language model
    § Retrieval-augmented generation (RAG)
    § Index one or more documents
    § Use text embedding to index documents
    § Store embeddings in a vector database
    § User asks question
    § Create embedding for the question
    § Search vector database to find relevant text
    § Use LLM with relevant input text & question to create an answer
    Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    Typical LLM use cases
    18

    View full-size slide

  19. Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    Answering Questions / Chat with documents - RAG
    Cleanup & Split Text Embedding
    Question
    Text Embedding
    Save
    Query
    Relevant Text
    Question
    Answer
    LLM
    19
    Vector DB
    Embedding
    model
    Embedding
    model 💡
    Indexing / Embedding QA

    View full-size slide

  20. Chat with my company’s policies (LangChain)
    Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    DEMO
    20

    View full-size slide

  21. Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    Agents
    21

    View full-size slide

  22. § Involve an LLM making decisions (via ReAct – Reasoning & Acting)
    § Which actions to take (thought)
    § Taking that action (executed via your code)
    § Seeing an observation
    § Repeating until done
    § Tools to interact with the workflow from the LLM
    § LangChain tools and agents
    § Semantic Kernel native functions
    § OpenAI function calling
    § Reliable (structured) way to interact from the LLM with your code
    § Proprietary to OpenAI models (fine-tuned)
    Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    Agents: Multi-steps workflows with LLM interaction
    22

    View full-size slide

  23. Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    ReAct – with function calling
    23
    LLM
    My code
    Query
    Some API Some database
    Prompt Function
    definitions
    Final answer
    Answer



    💡

    View full-size slide

  24. Stock market agent
    with OpenAI function calling (Python OpenAI SDK)
    Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    DEMO
    24

    View full-size slide

  25. Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    Landscape
    25

    View full-size slide

  26. Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    LLMs everywhere
    26
    OpenAI-related (cloud)
    OpenAI
    Azure OpenAI
    Service
    Big cloud providers
    Google Model Garden
    on Vertex AI
    Amazon Bedrock
    Other providers
    Antrophic
    Cohere
    HuggingFace
    … Open-source
    Edge
    IoT Server
    Desktop Mobile
    Web
    Open-source

    View full-size slide

  27. § Open-source community drives
    innovation in Generative AI
    § HuggingFace is central place for it
    § Literally, every week a new and
    “better” LLM shows up 🤓
    § Important factors
    § Use case
    § Parameter size
    § Quantization
    § Processing power needed
    § Mistral-based family shows big
    potential for local use cases (7B params)
    Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    Open-source LLMs thrive
    27

    View full-size slide

  28. Mistral OSS LLM locally on Apple M1 Max
    (llama.cpp & LM Studio)
    Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    DEMO
    28

    View full-size slide

  29. Running OSS LLM offline in the browser
    - with Wasm & WebGPU
    (web-llm)
    Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    DEMO
    29

    View full-size slide

  30. Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    Wrap-Up
    30

    View full-size slide

  31. § Fast moving and changing field
    § Every week something “big” happens in LLM space
    § Frameworks & ecosystem are evolving together with LLMs
    § Closed vs open LLMs
    § Competition drives invention & advancement
    § SISO (sh*t in, sh*t out)
    § Quality of results heavily depends on your data
    Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    Current state
    31

    View full-size slide

  32. Huge potential for
    LLM-driven
    human-machine workflows
    via universal interface agents
    Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    Outlook
    32

    View full-size slide

  33. Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    Outlook
    33

    View full-size slide

  34. § Prompt Engineering Guide
    § https://www.promptingguide.ai/
    § OpenAI API reference
    § https://platform.openai.com/docs/api-reference
    § Azure OpenAI Service REST API reference
    § https://learn.microsoft.com/en-us/azure/ai-
    services/openai/reference
    § Hugging Face Inference Endpoints
    § https://huggingface.co/docs/inference-
    endpoints/api_reference
    § LangChain
    § https://www.langchain.com/
    § LangChain4J
    § https://github.com/langchain4j/langchain4j
    § ReAct: Synergizing Reasoning and Acting in
    Language Models
    § https://react-lm.github.io/
    § Llama.cpp
    § https://github.com/ggerganov/llama.cpp
    § Web-llm
    § https://webllm.mlc.ai/
    § The Dawn of LMMs: Preliminary Explorations
    with GPT-4V(ision)
    § https://arxiv.org/pdf/2309.17421.pdf
    § OWASP Top 10 for LLM Applications
    § https://owasp.org/www-project-top-10-for-
    large-language-model-
    applications/assets/PDF/OWASP-Top-10-for-
    LLMs-2023-slides-v1_0_1.pdf
    Large Language Models, Daten & APIs
    Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.
    Links
    34

    View full-size slide