Generative AI Large Language Models: Szenarien, Use Cases und Patterns für Businessanwendungen Christian Weyer @christianweyer CTO, Technology Catalyst

The image part with relationship ID rId12 was not found in the file. The image part with relationship ID rId12 was not found in the file. Special Day Generative AI für Business-Anwendungen Thema Sprecher Datum, Uhrzeit Generative AI: Large Language Models – Szenarien, Use Cases und Patterns für Business-Anwendungen Christian Weyer DI, 13. Februar 2024, 10.45 bis 11.45 Generative AI: A Story About LLM Prompting (and how Tools like TypeChat Can Help) Rainer Stropek DI, 13. Februar 2024, 12.15 bis 13.15 Generative AI: Semantische Suche und LLMs jenseits des Hello World- RAG-Tutorials Sebastian Gingter DI, 13. Februar 2024, 15.30 bis 16.30 Generative AI: Optimierte Informationssuche durch AI-gesteuerte Datenquellenwahl Marco Frodl DI, 13. Februar 2024, 17.00 bis 18.00 Generative AI: Private GPT LLMs: Azure OpenAI Service sicher deployen mit Terraform Kenny Pflug DI, 13. Februar 2024, 19.00 bis 20.00

§ Technology catalyst § AI-powered solutions § Pragmatic end-to-end architectures § Microsoft Regional Director § Microsoft MVP for Developer Technologies & Azure ASPInsider, AzureInsider § Google GDE for Web Technologies [email protected] @christianweyer https://www.thinktecture.com Generative AI - Large Language Models Szenarien, Use Cases und Patterns für Business-Anwendungen Christian Weyer Co-Founder & CTO @ Thinktecture AG 3

Generative AI - Large Language Models Szenarien, Use Cases und Patterns für Business-Anwendungen Our journey 4 AI all-the- things? LLMs in your Solutions Selected Scenarios Exciting Times… Democratizing Generative AI

Generative AI - Large Language Models Szenarien, Use Cases und Patterns für Business-Anwendungen AI all-the-things? 5

Generative AI - Large Language Models Szenarien, Use Cases und Patterns für Business-Anwendungen AI all-the-things? 6 Data Science Artificial Intelligence Machine Learning Unsupervised, supervised, reinforcement learning Deep Learning ANN, CNN, RNN etc. NLP Generative AI GAN, VAE, Transformers etc. Image / Video Generation GAN, VAE Large Language Models Transformers

§ LLMs generate text based on input § LLMs can understand text – this changes a lot § Prompts are the universal interface (“UI”) → unstructured text with semantics § Human language evolves as a first-class citizen in software architecture 🤯 Generative AI - Large Language Models Szenarien, Use Cases und Patterns für Business-Anwendungen Large Language Models (LLMs) 7 Text… – really, just text?

§ LLMs are programs § LLMs are highly specialized neural networks § LLMs use(d) lots of data § LLMs need a lot of resources to be operated § LLMs have an API to be used through Generative AI - Large Language Models Szenarien, Use Cases und Patterns für Business-Anwendungen Large Language Models demystified 8

Generative AI - Large Language Models Szenarien, Use Cases und Patterns für Business-Anwendungen LLMs in your Solutions 9

Generative AI - Large Language Models Szenarien, Use Cases und Patterns für Business-Anwendungen Using LLMs: It’s just APIs ! Inference, FTW. 10

GPT-4 API access OpenAI Playground Generative AI - Large Language Models Szenarien, Use Cases und Patterns für Business-Anwendungen DEMO 11

Generative AI - Large Language Models Szenarien, Use Cases und Patterns für Business-Anwendungen The best tool for .NET developers to talk to LLMs! 12 🙈

§ OSS framework for developing applications powered by LLMs § > 1000 contributors § Python and Typescript versions § Chains for sequences of LLM-related actions in code § Abstractions for § Prompts & LLMs (local and remote) § Memory § Vector stores § Tools § Loading text from a wide range of sources Generative AI - Large Language Models Szenarien, Use Cases und Patterns für Business-Anwendungen LangChain - building LLM-based applications 13

§ Microsoft’s OSS framework to integrate LLMs into applications § .NET, Python, and Java versions § Plugins encapsulate AI capabilities § Semantic functions for prompting § Native functions to run local code § Planners are orchestrating LLM interactions § Not as broad feature set as LangChain § E.g., no concept/abstraction for loading data Generative AI - Large Language Models Szenarien, Use Cases und Patterns für Business-Anwendungen Semantic Kernel - building LLM-based applications 14

Generative AI - Large Language Models Szenarien, Use Cases und Patterns für Business-Anwendungen Talk to your Data 15

Real-World Generative AI Sprachzentrierte Anwendungen mit Large Language Models, Python & .NET Answering Questions on Data Retrieval-augmented generation (RAG) Cleanup & Split Text Embedding Question Text Embedding Save Query Relevant Text Question Answer LLM 16 Vector DB Embedding model Embedding model 💡 Indexing / Embedding QA Intro

Learning about my company’s policies via Slack LangChain (Python), Weaviate, Open AI GPT Generative AI - Large Language Models Szenarien, Use Cases und Patterns für Business-Anwendungen DEMO 17

Chat with web site documents Semantic Kernel (.NET), SqLite, Open AI GPT Generative AI - Large Language Models Szenarien, Use Cases und Patterns für Business-Anwendungen DEMO 18

Generative AI - Large Language Models Szenarien, Use Cases und Patterns für Business-Anwendungen Talk to your Systems 19

§ LLM can be instructed to, e.g. § Do sentiment analysis § Extract information from text § Extract structured information § Schema description: JSON, TypeScript types, etc. § Implement via tools like Kor, TypeChat, or Open AI Function / Tool Calling Generative AI - Large Language Models Szenarien, Use Cases und Patterns für Business-Anwendungen Extracting meaning in text for structured data 20

Extracting structured data OpenAI Tool Calling with LangChain Generative AI - Large Language Models Szenarien, Use Cases und Patterns für Business-Anwendungen DEMO 21

§ Tool / function calling standard established by OpenAI § Functions for interactions § LLM chooses to output JSON object containing arguments to call one or many functions § LLM does not call the function § You do it in your code § All major libs support tool calling with abstractions § OpenAI SDKs § Langchain § Semantic Kernel Generative AI - Large Language Models Szenarien, Use Cases und Patterns für Business-Anwendungen Extending LLM capabilities 22 curl https://api.openai.com/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "model": "gpt-3.5-turbo", "messages": [ { "role": "user", "content": "What is the weather like in Boston?" } ], "tools": [ { "type": "function", "function": { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"] } }, "required": ["location"] } } } ], "tool_choice": "auto" }'

Extending LLM capabilities OpenAI Tool Calling with Semantic Kernel Generative AI - Large Language Models Szenarien, Use Cases und Patterns für Business-Anwendungen DEMO 23

Generative AI - Large Language Models Szenarien, Use Cases und Patterns für Business-Anwendungen Democratizing Generative AI 24

Generative AI - Large Language Models Szenarien, Use Cases und Patterns für Business-Anwendungen LLMs everywhere OpenAI-related (cloud) OpenAI Azure OpenAI Service Big cloud providers Google Model Garden on Vertex AI Amazon Bedrock Open-source Edge IoT Server Desktop Mobile Web 25 Other providers Antrophic Cohere Mistral AI Hugging Face Open-source

§ Open-source community drives innovation in Generative AI § HuggingFace is central place for it § Literally, every week a new and “better” LLM shows up 🤓 § Important factors § Use case § Parameter size § Quantization § Processing power needed § Mistral-based family shows big potential for local use cases (7B params) Generative AI - Large Language Models Szenarien, Use Cases und Patterns für Business-Anwendungen Open-source LLMs thrive 26

Local RAG with Zephyr open-source LLM llama.cpp, ollama, LangChain Generative AI - Large Language Models Szenarien, Use Cases und Patterns für Business-Anwendungen DEMO 27

Generative AI - Large Language Models Szenarien, Use Cases und Patterns für Business-Anwendungen End-to-End 28

Ask for expert availability in my company systems Angular, Speech-to-text, internal HTTP API, node.js OpenAI SDK + Tool Calling, Open AI GPT, Text-to-speech Generative AI - Large Language Models Szenarien, Use Cases und Patterns für Business-Anwendungen DEMO 29

30 Angular PWA OpenAI Speech-to-Text TT Panorama Gateway OpenAI GPT-4 OpenAI Text-to-Speech Transcribe spoken text Transcribed text Check for experts availability with text Extract { experts, booking times } from text Structured JSON data Generate response with availability Response Response with experts availability 🗣 🔉 Speech-to-text for response Response audio TT Panorama Query Panorama API Availability Generative AI - Large Language Models Szenarien, Use Cases und Patterns für Business-Anwendungen

Generative AI - Large Language Models Szenarien, Use Cases und Patterns für Business-Anwendungen Exciting Times… 31

§ LLMs enable new scenarios & use cases to incorporate human language into software solutions § Fast moving and changing field § Every week something “big” happens in LLM space § Frameworks & ecosystem are evolving together with LLMs § Closed vs open LLMs § Competition drives invention & advancement § Security topics need to be addressed § SISO (sh*t in, sh*t out) § Quality of results heavily depends on your data & input Generative AI - Large Language Models Szenarien, Use Cases und Patterns für Business-Anwendungen Current state 32

§ Beginning of a long way Generative AI - Large Language Models Szenarien, Use Cases und Patterns für Business-Anwendungen Generative to Interactive 33 https://www.technologyreview.com/2023/09/15/1079624/deepmind-inflection-generative-ai-whats-next-mustafa-suleyman

Thank you! Christian Weyer https://thinktecture.com/christian-weyer 34