Large Language Models Szenarien, Use Cases & Patterns für Business-Anwendungen - in Action Christian Weyer | Co-Founder & CTO | Thinktecture AG

§ Technology catalyst § AI-powered solutions § Pragmatic end-to-end architectures § Microsoft Regional Director § Microsoft MVP for Developer Technologies & Azure ASPInsider, AzureInsider § Google GDE for Web Technologies [email protected] @christianweyer https://www.thinktecture.com Large Language Models Szenarien, Use Cases & Patterns für Business-Anwendungen - in Action Christian Weyer Co-Founder & CTO @ Thinktecture AG 2

The image part with relationship ID rId3 was not found in the file. The image part with relationship ID rId3 was not found in the file. Special Day Generative AI für Business-Anwendungen Thema Sprecher Datum, Uhrzeit Large Language Models: Szenarien, Use Cases & Patterns für Business- Anwendungen - in Action Christian Weyer DI, 17. September 2024, 10.45 bis 11.45 Real-World RAG: Eigene Daten & Dokumente mit semantischer Suche & LLMs erschließen Sebastian Gingter DI, 17. September 2024, 12.15 bis 13.15 Von 0 zu Smart: SPAs mit Generative AI aufwerten Max Marschall DI, 17. September 2024, 15.30 bis 16.30 Deep Dive in OpenAI Hosted Tools Rainer Stropek DI, 17. September 2024, 17.00 bis 18.00

Large Language Models Szenarien, Use Cases & Patterns für Business-Anwendungen - in Action Our journey today 4 AI all-the- things? LLMs in your Solutions Talk to your Data Recap & Outlook Generative AI everywhere Talk to your Systems

Large Language Models Szenarien, Use Cases & Patterns für Business-Anwendungen - in Action AI all-the-things? 5

Large Language Models Szenarien, Use Cases & Patterns für Business-Anwendungen - in Action AI all-the-things? 6 Data Science Artificial Intelligence Machine Learning Unsupervised, supervised, reinforcement learning Deep Learning ANN, CNN, RNN etc. NLP (Natural Language Processing) Generative AI GAN, VAE, Transformers etc. Image / Video Generation GAN, VAE Large Language Models Transformers

§ LLMs generate text based on input § LLMs can understand text – this changes a lot § Without having to train them on domains or use cases § Prompts are the universal interface (“UI”) → unstructured text with semantics § Human language evolves as a first-class citizen in software architecture 🤯 Large Language Models Szenarien, Use Cases & Patterns für Business-Anwendungen - in Action Large Language Models (LLMs) – like GPT powering ChatGPT 7

§ LLMs are programs § LLMs are highly specialized neural networks § LLMs use(d) lots of data § LLMs need a lot of resources to be operated § LLMs have an API to be used through Large Language Models Szenarien, Use Cases & Patterns für Business-Anwendungen - in Action Large Language Models demystified 🔍 8

Large Language Models Szenarien, Use Cases & Patterns für Business-Anwendungen - in Action LLMs in your Solutions 9

§ LLMs are always part of end-to-end architectures § Client apps (Web, desktop, mobile) § Services with APIs § Databases § etc. § An LLM is ‘just’ an additional asset in your architecture § Enabling human language understanding & generation § It is not the Holy Grail for everything Large Language Models Szenarien, Use Cases & Patterns für Business-Anwendungen - in Action End-to-end architectures with LLMs 10 Clients Services LLMs Desktop Web Mobile Service A Service B Service C API Gateway Monitoring LLM 1 LLM 2

Large Language Models Szenarien, Use Cases & Patterns für Business-Anwendungen - in Action Using LLMs: It’s just HTTP APIs Inference, FTW 11

GPT-4 API access OpenAI Playground Large Language Models Szenarien, Use Cases & Patterns für Business-Anwendungen - in Action DEMO 12

Large Language Models Szenarien, Use Cases & Patterns für Business-Anwendungen - in Action Generative AI everywhere 13

Large Language Models Szenarien, Use Cases & Patterns für Business-Anwendungen - in Action LLMs everywhere OpenAI-related (cloud) OpenAI Azure OpenAI Service Big cloud providers Google Model Garden on Vertex AI Amazon Bedrock Edge 14 Other providers Anthropic Cohere Mistral AI Hugging Face Groq

§ Open-source community drives innovation in Generative AI § Llama- & Mistral-based families show big potential § Success factors § Use case § Parameter size § Quantization § Processing power needed § CPU optimization on its way Large Language Models Szenarien, Use Cases & Patterns für Business-Anwendungen - in Action Open-weights LLMs thrive 15 § Local inference runtimes with APIs § E.g. llama.cpp, ollama, VLLM § Local UIs § E.g. Open WebUI

Local open-source models, APIs & UIs Ollama, Open WebUI Large Language Models Szenarien, Use Cases & Patterns für Business-Anwendungen - in Action DEMO 16

Barebones SDKs § E.g. Open AI SDK § Available for any programming language § Basic abstraction over HTTP APIs § Lot of inference runtimes offer Open AI API compatible APIs § Also available from other providers § Mistral § Anthropic § Cohere § Ollama § Etc. Frameworks – e.g. LangChain, Semantic Kernel § Provide abstractions – typically for § Prompts & LLMs § Memory § Vector stores § Tools § Loading data from a wide range of sources Large Language Models Szenarien, Use Cases & Patterns für Business-Anwendungen - in Action Building LLM-based end-to-end applications 17

Large Language Models Szenarien, Use Cases & Patterns für Business-Anwendungen - in Action Talk to your Data 18

Large Language Models Szenarien, Use Cases & Patterns für Business-Anwendungen - in Action Answering questions on data Retrieval-augmented generation (RAG) Cleanup & Split Text Embedding Question Text Embedding Save Query Relevant Text Question Answer LLM 19 Embedding model Embedding model 💡 Indexing / Embedding Question Answering Vector DB

RAG: Learning about company’s policies via Slack LangChain, Weaviate – GPT-4o Large Language Models Szenarien, Use Cases & Patterns für Business-Anwendungen - in Action DEMO 20

Local RAG: Llama 3.1 open-source LLM llama.cpp, ollama, LangChain, StreamLit Large Language Models Szenarien, Use Cases & Patterns für Business-Anwendungen - in Action DEMO 21

Large Language Models Szenarien, Use Cases & Patterns für Business-Anwendungen - in Action Talk to your Systems 22

§ Write or speak your input → get structured data for your programs & systems § Clever & strict prompting § Schema description: Custom format, JSON, TypeScript types, etc. § Framework or tools support § Pydantic, Instructor, TypeChat, etc. § OpenAI Function / Tool Calling Large Language Models Szenarien, Use Cases & Patterns für Business-Anwendungen - in Action 1. Extract structured data from textual information 23

Extracting structured data from text & voice: Form filling Data extraction prompt, OpenAI JS SDK, Angular Forms – Mixtral-8x7B on Groq Large Language Models Szenarien, Use Cases & Patterns für Business-Anwendungen - in Action DEMO 24

§ Integrate LLM-external systems to aid LLMs § Tool / function calling standard established by OpenAI § LLM outputs JSON conforming to a schema § LLM does not call a function § All major libs support tool calling § OpenAI SDKs § LangChain § Semantic Kernel § etc. § You wire up the logic in your code Large Language Models Szenarien, Use Cases & Patterns für Business-Anwendungen - in Action 2. Extending LLM capabilities 25 curl https://api.openai.com/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "model": "gpt-4o", "messages": [ { "role": "user", "content": "What is the weather like in Boston?" } ], "tools": [ { "type": "function", "function": { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"] } }, "required": ["location"] } } } ], "tool_choice": "auto" }'

Ask for experts availability in my company systems Angular, Speech-to-text, internal HTTP API, node.js OpenAI SDK + Tool Calling, Text-to-speech – GPT-4-turbo Large Language Models Szenarien, Use Cases & Patterns für Business-Anwendungen - in Action DEMO 26

Large Language Models Szenarien, Use Cases & Patterns für Business-Anwendungen - in Action Talking to internal APIs – Ask for expert’s availability 27 Angular PWA Open AI Speech-to-Text Internal Systems Gateway Open AI GPT-4 Open AI Text-to-Speech Transcribe spoken text Transcribed text Check for experts availability with text Extract { experts, booking times } from text Structured JSON data (Tool calling) Generate response with availability Response Response with experts availability 🗣 🔉 Speech-to-text for response Response audio Internal Company API Query Availability API Availability When is CL…? CL will be…

Large Language Models Szenarien, Use Cases & Patterns für Business-Anwendungen - in Action Exciting Times 🫱 🫲 28

§ LLMs enable new scenarios & use cases to incorporate human language into software solutions § Fast moving and changing field § Every week something “big” happens in LLM space § Frameworks & ecosystem are evolving together with LLMs § Closed vs open LLMs § Competition drives invention & advancement § SLMs: specialized, fine-tuned for domains § Running local models in production is hard! § SISO (sh*t in, sh*t out) § Quality of results heavily depends on your data & input Large Language Models Szenarien, Use Cases & Patterns für Business-Anwendungen - in Action Current state 29

Large Language Models Szenarien, Use Cases & Patterns für Business-Anwendungen - in Action The rise of SLMs & CPU inference 30

Thank you! Christian Weyer https://thinktecture.com/christian-weyer 31