Slide 1

Slide 1 text

„User Interfaces“ mal anders Sprache als Zugang zur eigenen Software – mit Large Language Models Jenseits von ChatGPT Christian Weyer Thinktecture AG @christianweyer

Slide 2

Slide 2 text

§ Technology catalyst § AI-powered solutions § Pragmatic end-to-end architectures § Microsoft Regional Director § Microsoft MVP for Developer Technologies & Azure ASPInsider, AzureInsider § Google GDE for Web Technologies [email protected] @christianweyer https://www.thinktecture.com User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models Christian Weyer Co-Founder & CTO @ Thinktecture AG 2

Slide 3

Slide 3 text

User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models Our journey today 3 AI all-the- things? LLMs in your Solutions Talk to your Data Exciting Times… Democratizing Generative AI Talk to your Systems

Slide 4

Slide 4 text

User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models AI all-the-things? 4

Slide 5

Slide 5 text

User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models A.I. 5

Slide 6

Slide 6 text

User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models AI all-the-things? 6 Data Science Artificial Intelligence Machine Learning Unsupervised, supervised, reinforcement learning Deep Learning ANN, CNN, RNN etc. Generative AI GAN, VAE, Transformers etc. Image / Video Generation GAN, VAE Large Language Models Transformers NLP (Natural Language Processing)

Slide 7

Slide 7 text

§ LLMs generate text based on input § LLMs can understand text – this changes a lot § Without having to train them on data, domains, or use cases § Prompts are the universal interface (“UI”): unstructured text with semantics as input § Human language evolves as a first-class citizen in software architecture 🤯 User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models Large Language Models (LLMs) – like GPT powering ChatGPT 7 Text… – really, just text?

Slide 8

Slide 8 text

§ LLMs are programs § LLMs are highly specialized neural networks § LLMs use(d) lots of data § LLMs need a lot of resources to be operated § LLMs have an API to be used through User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models Large Language Models demystified 🔍 8

Slide 9

Slide 9 text

User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models LLMs in your Solutions 9

Slide 10

Slide 10 text

User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models Using LLMs: It’s just HTTP APIs Inference, FTW. 10

Slide 11

Slide 11 text

GPT API access OpenAI Playground User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models DEMO 11

Slide 12

Slide 12 text

User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models Talk to your Data 12

Slide 13

Slide 13 text

User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models Answering Questions on Data Retrieval-augmented generation (RAG) Cleanup & Split Text Embedding Question Text Embedding Save Query Relevant Text Question Answer LLM 13 Embedding model Embedding model 💡 Indexing / Embedding Question Answering Vector DB

Slide 14

Slide 14 text

RAG: Learning about company’s policies via Slack LangChain Python, Weaviate – Mixtral-8x7B on Groq User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models DEMO 14

Slide 15

Slide 15 text

User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models Talk to your Systems 15

Slide 16

Slide 16 text

§ Write or speak your input → get structured data for your programs & systems § Clever & strict prompting § Schema description: JSON, TypeScript types, etc. § Framework or tools support § Kor, TypeChat, etc. § Open AI Function Calling User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models Extract structured data from textual information 16

Slide 17

Slide 17 text

Extracting structured data from text: Flight search OpenAI Function Calling, LangChain Python, FastAPI, JS - GPT-4o User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models DEMO 17

Slide 18

Slide 18 text

Extracting structured data from voice: Form filling JSON extraction, Angular Forms - Mixtral-8x7B on Groq User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models DEMO 18

Slide 19

Slide 19 text

§ Integrate LLM-external systems to aid LLMs § Tool / function calling standard established by OpenAI § LLM outputs JSON containing arguments to call one or many functions § LLM does not call the function § All major libs support tool calling § OpenAI SDKs § LangChain § Semantic Kernel § etc. User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models Extending LLM capabilities 19 curl https://api.openai.com/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "model": "gpt-3.5-turbo", "messages": [ { "role": "user", "content": "What is the weather like in Boston?" } ], "tools": [ { "type": "function", "function": { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"] } }, "required": ["location"] } } } ], "tool_choice": "auto" }'

Slide 20

Slide 20 text

User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models Talking to internal APIs – Ask for experts availability 20 Angular PWA Open AI Speech-to-Text Internal Systems Gateway Open AI GPT-4-T Open AI Text-to-Speech Transcribe spoken text Transcribed text Check for experts availability with text Extract { experts, booking times } from text Structured JSON data (Tool calling) Generate response with availability Response Response with experts availability 🗣 🔉 Speech-to-text for response Response audio Internal Company API Query Availability API Availability When is CL…? CL will be…

Slide 21

Slide 21 text

Ask for experts availability in my company systems Angular, Speech-to-text, internal HTTP API, node.js OpenAI SDK + Tool Calling, Text-to-speech – GPT-4-Turbo User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models DEMO 21

Slide 22

Slide 22 text

User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models Democratizing Generative AI 22

Slide 23

Slide 23 text

User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models LLMs everywhere OpenAI-related (cloud) OpenAI Azure OpenAI Service Big cloud providers Google Model Garden on Vertex AI Amazon Bedrock Open-source Edge IoT Server Desktop Mobile Web 23 Other providers Antrophic Cohere Mistral AI Hugging Face Open-source

Slide 24

Slide 24 text

§ Open-source community drives innovation in Generative AI § Important factors § Use case § Parameter size § Quantization § Processing power needed § Llama- & Mistral-based families show big potential for local use cases User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models Open-source LLMs thrive 24

Slide 25

Slide 25 text

Local RAG: Mistral-7B open-source LLM llama.cpp, ollama, LangChain Python, StreamLit User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models DEMO 25

Slide 26

Slide 26 text

User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models Exciting Times… 26

Slide 27

Slide 27 text

User Interfaces mal anders Sprache als Zugang zur eigenen Software mit Large Language Models 2024: The year of SLMs & CPU inference 27

Slide 28

Slide 28 text

Vielen Dank für Ihre Aufmerksamkeit! Christian Weyer https://www.thinktecture.com