AI in Action: 'UIs' with a difference Language as access to your own software with Large Language Models (LLMs) Christian Weyer Co-Founder & CTO @christianweyer 

§ Technology catalyst § AI-powered solutions § Pragmatic end-to-end architectures § Microsoft Regional Director § Microsoft MVP for Developer Technologies & Azure ASPInsider, AzureInsider § Google GDE for Web Technologies [email protected] @christianweyer https://www.thinktecture.com AI in Action: 'UIs' with a difference Language as access to your own software with Large Language Models (LLMs) Christian Weyer Co-Founder & CTO @ Thinktecture AG 2 

AI in Action: 'UIs' with a difference Language as access to your own software with Large Language Models (LLMs) Our journey today AI all-the- things? LLMs in your Solutions Talk to your Data Exciting Times… Democratizing Generative AI Talk to your Systems 3 

AI in Action: 'UIs' with a difference Language as access to your own software with Large Language Models (LLMs) AI all-the-things? 4 

AI in Action: 'UIs' with a difference Language as access to your own software with Large Language Models (LLMs) A.I. 5 

AI in Action: 'UIs' with a difference Language as access to your own software with Large Language Models (LLMs) AI all-the-things? Data Science Artificial Intelligence Machine Learning Unsupervised, supervised, reinforcement learning Deep Learning ANN, CNN, RNN etc. Generative AI GAN, VAE, Transformers etc. Image / Video Generation GAN, VAE Large Language Models Transformers NLP (Natural Language Processing) 6 

§ LLMs generate text based on input § LLMs can understand text – this changes a lot § Without having to train them on data, domains, or use cases § Prompts are the universal interface (“UI”): unstructured text with semantics as input § Human language evolves as a first-class citizen in software architecture 🤯 AI in Action: 'UIs' with a difference Language as access to your own software with Large Language Models (LLMs) Large Language Models (LLMs) – like GPT powering ChatGPT Text… – really, just text? 7 

§ LLMs are programs § LLMs are highly specialized neural networks § LLMs use(d) lots of data § LLMs need a lot of resources to be operated § LLMs have an API to be used through AI in Action: 'UIs' with a difference Language as access to your own software with Large Language Models (LLMs) Large Language Models demystified 🔍 8 

AI in Action: 'UIs' with a difference Language as access to your own software with Large Language Models (LLMs) LLMs in your Solutions 9 

AI in Action: 'UIs' with a difference Language as access to your own software with Large Language Models (LLMs) Using LLMs: It’s just HTTP APIs Inference, FTW. 10 

GPT API access OpenAI Playground AI in Action: 'UIs' with a difference Language as access to your own software with Large Language Models (LLMs) DEMO 11 

Barebones SDKs § E.g. Open AI SDK § Available for any programming language § Basic abstraction over HTTP APIs § Also available from other providers § Mistral § Cohere § Etc. Frameworks – e.g. LangChain § Large community § Python and Typescript versions § Chains for sequences of LLM-related actions in code § Abstractions for § Prompts & LLMs § Memory § Vector stores § Tools § Loading text from a wide range of sources AI in Action: 'UIs' with a difference Language as access to your own software with Large Language Models (LLMs) Building LLM-based end-to-end applications 12 

AI in Action: 'UIs' with a difference Language as access to your own software with Large Language Models (LLMs) Talk to your Data 13 

AI in Action: 'UIs' with a difference Language as access to your own software with Large Language Models (LLMs) Answering Questions on Data Retrieval-augmented generation (RAG) Cleanup & Split Text Embedding Question Text Embedding Save Query Relevant Text Question Answer LLM Embedding model Embedding model 💡 Indexing / Embedding Question Answering Vector DB 14 

RAG: Learning about company’s policies via Slack LangChain Python, Weaviate – Mixtral-8x7B on Groq AI in Action: 'UIs' with a difference Language as access to your own software with Large Language Models (LLMs) DEMO 15 

AI in Action: 'UIs' with a difference Language as access to your own software with Large Language Models (LLMs) Talk to your Systems 16 

§ Write or speak your input → get structured data for your programs & systems § Clever & strict prompting § Schema description: JSON, TypeScript types, etc. § Framework or tools support § Kor, TypeChat, etc. § Open AI Function Calling AI in Action: 'UIs' with a difference Language as access to your own software with Large Language Models (LLMs) Extract structured data from textual information 17 

Extracting structured data from text: Flight search OpenAI Function Calling, LangChain Python, FastAPI, JS - GPT-4o AI in Action: 'UIs' with a difference Language as access to your own software with Large Language Models (LLMs) DEMO 18 

Extracting structured data from voice: Form filling JSON extraction, Angular Forms - Mixtral-8x7B on Groq AI in Action: 'UIs' with a difference Language as access to your own software with Large Language Models (LLMs) DEMO 19 

§ Integrate LLM-external systems to aid LLMs § Tool / function calling standard established by OpenAI § LLM outputs JSON containing arguments to call one or many functions § LLM does not call the function § All major libs support tool calling § OpenAI SDKs § LangChain § Semantic Kernel § etc. AI in Action: 'UIs' with a difference Language as access to your own software with Large Language Models (LLMs) Extending LLM capabilities curl https://api.openai.com/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "model": "gpt-3.5-turbo", "messages": [ { "role": "user", "content": "What is the weather like in Boston?" } ], "tools": [ { "type": "function", "function": { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"] } }, "required": ["location"] } } } ], "tool_choice": "auto" }' 20 

AI in Action: 'UIs' with a difference Language as access to your own software with Large Language Models (LLMs) Talking to internal APIs – Ask for experts availability Angular PWA Open AI Speech-to-Text Internal Systems Gateway Open AI GPT-4-T Open AI Text-to-Speech Transcribe spoken text Transcribed text Check for experts availability with text Extract { experts, booking times } from text Structured JSON data (Tool calling) Generate response with availability Response Response with experts availability 🗣 🔉 Speech-to-text for response Response audio Internal Company API Query Availability API Availability When is CL…? CL will be… 21 

Ask for experts availability in my company systems Angular, Speech-to-text, internal HTTP API, node.js OpenAI SDK + Tool Calling, Text-to-speech – GPT-4-Turbo AI in Action: 'UIs' with a difference Language as access to your own software with Large Language Models (LLMs) DEMO 22 

AI in Action: 'UIs' with a difference Language as access to your own software with Large Language Models (LLMs) Democratizing Generative AI 23 

AI in Action: 'UIs' with a difference Language as access to your own software with Large Language Models (LLMs) LLMs everywhere OpenAI-related (cloud) OpenAI Azure OpenAI Service Big cloud providers Google Model Garden on Vertex AI Amazon Bedrock Open-source Edge IoT Server Desktop Mobile Web Other providers Antrophic Cohere Mistral AI Hugging Face Open-source 24 

§ Open-source community drives innovation in Generative AI § Important factors § Use case § Parameter size § Quantization § Processing power needed § Llama- & Mistral-based families show big potential for local use cases AI in Action: 'UIs' with a difference Language as access to your own software with Large Language Models (LLMs) Open-source LLMs thrive 25 

Local RAG: Mistral-7B open-source LLM llama.cpp, ollama, LangChain Python, StreamLit AI in Action: 'UIs' with a difference Language as access to your own software with Large Language Models (LLMs) DEMO 26 

AI in Action: 'UIs' with a difference Language as access to your own software with Large Language Models (LLMs) Exciting Times… 27 

AI in Action: 'UIs' with a difference Language as access to your own software with Large Language Models (LLMs) 2024: The year of SLMs & CPU inference 28 

Thank you! Christian Weyer [email protected] https://thinktecture.com/christian-weyer 29