„User Interfaces“ mal anders Sprache als Zugang zur eigenen Software – mit Large Language Models Jenseits von ChatGPT Christian Weyer Thinktecture AG @christianweyer

§ Technology catalyst § AI-powered solutions § Pragmatic end-to-end architectures § Microsoft Regional Director § Microsoft MVP for Developer Technologies & Azure ASPInsider, AzureInsider § Google GDE for Web Technologies [email protected] @christianweyer https://www.thinktecture.com User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models Christian Weyer Co-Founder & CTO @ Thinktecture AG 2

User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models Our journey today 3 AI all-the- things? LLMs in your Solutions Talk to your Data Exciting Times… Democratizing Generative AI Talk to your Systems

User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models AI all-the-things? 4

User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models A.I. 5

User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models AI all-the-things? 6 Data Science Artificial Intelligence Machine Learning Unsupervised, supervised, reinforcement learning Deep Learning ANN, CNN, RNN etc. Generative AI GAN, VAE, Transformers etc. Image / Video Generation GAN, VAE Large Language Models Transformers NLP (Natural Language Processing)

§ LLMs generate text based on input § LLMs can understand text – this changes a lot § Without having to train them on data, domains, or use cases § Prompts are the universal interface (“UI”): unstructured text with semantics as input § Human language evolves as a first-class citizen in software architecture 🤯 User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models Large Language Models (LLMs) – like GPT powering ChatGPT 7 Text… – really, just text?

§ LLMs are programs § LLMs are highly specialized neural networks § LLMs use(d) lots of data § LLMs need a lot of resources to be operated § LLMs have an API to be used through User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models Large Language Models demystified 🔍 8

User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models LLMs in your Solutions 9

User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models Using LLMs: It’s just HTTP APIs Inference, FTW. 10

GPT API access OpenAI Playground User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models DEMO 11

User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models Talk to your Data 12

User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models Answering Questions on Data Retrieval-augmented generation (RAG) Cleanup & Split Text Embedding Question Text Embedding Save Query Relevant Text Question Answer LLM 13 Embedding model Embedding model 💡 Indexing / Embedding Question Answering Vector DB

RAG: Learning about company’s policies via Slack LangChain Python, Weaviate – Mixtral-8x7B on Groq User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models DEMO 14

User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models Talk to your Systems 15

§ Write or speak your input → get structured data for your programs & systems § Clever & strict prompting § Schema description: JSON, TypeScript types, etc. § Framework or tools support § Kor, TypeChat, etc. § Open AI Function Calling User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models Extract structured data from textual information 16

Extracting structured data from text: Flight search OpenAI Function Calling, LangChain Python, FastAPI, JS - GPT-4o User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models DEMO 17

Extracting structured data from voice: Form filling JSON extraction, Angular Forms - Mixtral-8x7B on Groq User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models DEMO 18

§ Integrate LLM-external systems to aid LLMs § Tool / function calling standard established by OpenAI § LLM outputs JSON containing arguments to call one or many functions § LLM does not call the function § All major libs support tool calling § OpenAI SDKs § LangChain § Semantic Kernel § etc. User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models Extending LLM capabilities 19 curl https://api.openai.com/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "model": "gpt-3.5-turbo", "messages": [ { "role": "user", "content": "What is the weather like in Boston?" } ], "tools": [ { "type": "function", "function": { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"] } }, "required": ["location"] } } } ], "tool_choice": "auto" }'

User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models Talking to internal APIs – Ask for experts availability 20 Angular PWA Open AI Speech-to-Text Internal Systems Gateway Open AI GPT-4-T Open AI Text-to-Speech Transcribe spoken text Transcribed text Check for experts availability with text Extract { experts, booking times } from text Structured JSON data (Tool calling) Generate response with availability Response Response with experts availability 🗣 🔉 Speech-to-text for response Response audio Internal Company API Query Availability API Availability When is CL…? CL will be…

Ask for experts availability in my company systems Angular, Speech-to-text, internal HTTP API, node.js OpenAI SDK + Tool Calling, Text-to-speech – GPT-4-Turbo User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models DEMO 21

User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models Democratizing Generative AI 22

User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models LLMs everywhere OpenAI-related (cloud) OpenAI Azure OpenAI Service Big cloud providers Google Model Garden on Vertex AI Amazon Bedrock Open-source Edge IoT Server Desktop Mobile Web 23 Other providers Antrophic Cohere Mistral AI Hugging Face Open-source

§ Open-source community drives innovation in Generative AI § Important factors § Use case § Parameter size § Quantization § Processing power needed § Llama- & Mistral-based families show big potential for local use cases User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models Open-source LLMs thrive 24

Local RAG: Mistral-7B open-source LLM llama.cpp, ollama, LangChain Python, StreamLit User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models DEMO 25

User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models Exciting Times… 26

User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models 2024: The year of SLMs & CPU inference 27

Vielen Dank für Ihre Aufmerksamkeit! Christian Weyer https://www.thinktecture.com