Slide 1

Slide 1 text

AI++ Multimodale Large Language Models als Kern moderner Business-Anwendungen – in Action Christian Weyer Co-Founder & CTO @christianweyer

Slide 2

Slide 2 text

§ Technology catalyst § AI-powered solutions § Pragmatic end-to-end architectures § Microsoft Regional Director § Microsoft MVP for Developer Technologies & Azure ASPInsider, AzureInsider § Google GDE for Web Technologies [email protected] @christianweyer https://www.thinktecture.com Multimodale Large Language Models (LMMs) als Kern moderner Business-Anwendungen – in Action Christian Weyer Co-Founder & CTO @ Thinktecture AG 2

Slide 3

Slide 3 text

Multimodale Large Language Models (LMMs) als Kern moderner Business-Anwendungen – in Action Our journey today 3 AI all-the- things? LLMs in your Solutions Talk to your Data Exciting Times… Democratizing Generative AI Talk to your Systems

Slide 4

Slide 4 text

Multimodale Large Language Models (LMMs) als Kern moderner Business-Anwendungen – in Action AI all-the-things? 4

Slide 5

Slide 5 text

Multimodale Large Language Models (LMMs) als Kern moderner Business-Anwendungen – in Action AI all-the-things? 5 Data Science Artificial Intelligence Machine Learning Unsupervised, supervised, reinforcement learning Deep Learning ANN, CNN, RNN etc. NLP (Natural Language Processing) Generative AI GAN, VAE, Transformers etc. Image / Video Generation GAN, VAE Large Language Models Transformers

Slide 6

Slide 6 text

§ LLMs generate text based on input § LLMs can understand text – this changes a lot § Without having to train them on domains or use cases § Prompts are the universal interface (“UI”) → unstructured text with semantics § Human language evolves as a first-class citizen in software architecture 🤯 Multimodale Large Language Models (LMMs) als Kern moderner Business-Anwendungen – in Action Large Language Models (LLMs) – like GPT powering ChatGPT 6 Text… – really, just text?

Slide 7

Slide 7 text

§ LLMs are programs § LLMs are highly specialized neural networks § LLMs use(d) lots of data § LLMs need a lot of resources to be operated § LLMs have an API to be used through Multimodale Large Language Models (LMMs) als Kern moderner Business-Anwendungen – in Action Large Language Models demystified 🔍 7

Slide 8

Slide 8 text

Multimodale Large Language Models (LMMs) als Kern moderner Business-Anwendungen – in Action LLMs in your Solutions 8

Slide 9

Slide 9 text

Multimodale Large Language Models (LMMs) als Kern moderner Business-Anwendungen – in Action Using LLMs: It’s just HTTP APIs Inference, FTW. 9

Slide 10

Slide 10 text

GPT-4 API access OpenAI Playground Multimodale Large Language Models (LMMs) als Kern moderner Business-Anwendungen – in Action DEMO 10

Slide 11

Slide 11 text

§ Barebones SDKs § Abstraction over HTTP APIs § E.g. Open AI SDK, Mistral SDK § Available for any programming language § Also available from other LLM providers § Or: Abstracing frameworks Multimodale Large Language Models (LMMs) als Kern moderner Business-Anwendungen – in Action Building LLM-based end-to-end applications 11

Slide 12

Slide 12 text

§ OSS framework for developing applications powered by LLMs § > 1000 contributors § Python and Typescript versions § Chains for sequences of LLM-related actions in code § Abstractions for § Prompts & LLMs (local and remote) § Memory § Vector stores § Tools § Loading text from a wide range of sources § Alternatives like LlamaIndex, Haystack, etc. Multimodale Large Language Models (LMMs) als Kern moderner Business-Anwendungen – in Action LangChain - building LLM-based applications 12

Slide 13

Slide 13 text

§ Microsoft’s open-source framework to integrate LLMs into applications § .NET, Python, and Java versions § Plugins encapsulate AI capabilities § Semantic functions for prompting § Native functions to run local code § Chain is collection of Plugins § Planners are similar to Agents in LangChain § Not as broad feature set as LangChain § E.g., no concept/abstraction for loading & working with data Multimodale Large Language Models (LMMs) als Kern moderner Business-Anwendungen – in Action Semantic Kernel 13

Slide 14

Slide 14 text

Multimodale Large Language Models (LMMs) als Kern moderner Business-Anwendungen – in Action Talk to your Data 14

Slide 15

Slide 15 text

Multimodale Large Language Models (LMMs) als Kern moderner Business-Anwendungen – in Action Answering Questions on Data Retrieval-augmented generation (RAG) Cleanup & Split Text Embedding Question Text Embedding Save Query Relevant Text Question Answer LLM 15 Embedding model Embedding model 💡 Indexing / Embedding Question Answering Vector DB

Slide 16

Slide 16 text

RAG: Learning about company’s policies via Slack LangChain, Weaviate – GPT-4o Multimodale Large Language Models (LMMs) als Kern moderner Business-Anwendungen – in Action DEMO 16

Slide 17

Slide 17 text

Multimodale Large Language Models (LMMs) als Kern moderner Business-Anwendungen – in Action Talk to your Systems 17

Slide 18

Slide 18 text

§ Write or speak your input → get structured data for your programs & systems § Clever & strict prompting § Schema description: JSON, TypeScript types, etc. § Framework or tools support § Pydantic, Kor, TypeChat, etc. § Open AI Function / Tool Calling Multimodale Large Language Models (LMMs) als Kern moderner Business-Anwendungen – in Action Extract structured data from textual information 18

Slide 19

Slide 19 text

Extracting structured data from text & voice: Form filling JSON extraction, OpenAI JS SDK, Angular Forms - Mixtral-8x7B on Groq Multimodale Large Language Models (LMMs) als Kern moderner Business-Anwendungen – in Action DEMO 19

Slide 20

Slide 20 text

Extracting structured data from PDF Python SDK, OpenAI Tool Calling - GPT-4o Multimodale Large Language Models (LMMs) als Kern moderner Business-Anwendungen – in Action DEMO 20 Multimodal

Slide 21

Slide 21 text

§ Integrate LLM-external systems to aid LLMs § Tool / function calling standard established by OpenAI § LLM outputs JSON containing arguments to call one or many functions § LLM does not call the function § All major libs support tool calling § OpenAI SDKs § LangChain § Semantic Kernel § etc. Multimodale Large Language Models (LMMs) als Kern moderner Business-Anwendungen – in Action Extending LLM capabilities 21 curl https://api.openai.com/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "model": "gpt-3.5-turbo", "messages": [ { "role": "user", "content": "What is the weather like in Boston?" } ], "tools": [ { "type": "function", "function": { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"] } }, "required": ["location"] } } } ], "tool_choice": "auto" }'

Slide 22

Slide 22 text

Ask for experts availability in my company systems Angular, Speech-to-text, internal HTTP API, node.js OpenAI SDK + Tool Calling, Text-to-speech – GPT-4o Multimodale Large Language Models (LMMs) als Kern moderner Business-Anwendungen – in Action DEMO 22

Slide 23

Slide 23 text

Multimodale Large Language Models (LMMs) als Kern moderner Business-Anwendungen – in Action Talking to internal APIs – Ask for experts availability 23 Angular PWA Open AI Speech-to-Text Internal Systems Gateway Open AI GPT-4 Open AI Text-to-Speech Transcribe spoken text Transcribed text Check for experts availability with text Extract { experts, booking times } from text Structured JSON data (Tool calling) Generate response with availability Response Response with experts availability 🗣 🔉 Speech-to-text for response Response audio Internal Company API Query Availability API Availability When is CL…? CL will be…

Slide 24

Slide 24 text

Multimodale Large Language Models (LMMs) als Kern moderner Business-Anwendungen – in Action Democratizing Generative AI 24

Slide 25

Slide 25 text

Multimodale Large Language Models (LMMs) als Kern moderner Business-Anwendungen – in Action LLMs everywhere OpenAI-related (cloud) OpenAI Azure OpenAI Service Big cloud providers Google Model Garden on Vertex AI Amazon Bedrock Open-source Edge IoT Server Desktop Mobile Web 25 Other providers Antrophic Cohere Mistral AI Hugging Face Open-source

Slide 26

Slide 26 text

§ Open-source community drives innovation in Generative AI § Important factors § Use case § Parameter size § Quantization § Processing power needed § Llama- & Mistral-based families show big potential for local use cases Multimodale Large Language Models (LMMs) als Kern moderner Business-Anwendungen – in Action Open-source LLMs thrive 26

Slide 27

Slide 27 text

Local RAG: Mistral-7B open-source LLM llama.cpp, ollama, LangChain, StreamLit Multimodale Large Language Models (LMMs) als Kern moderner Business-Anwendungen – in Action DEMO 27

Slide 28

Slide 28 text

Multimodale Large Language Models (LMMs) als Kern moderner Business-Anwendungen – in Action Exciting Times… 28

Slide 29

Slide 29 text

§ LLMs & LMMs enable new scenarios & use cases to incorporate human language into software solutions § Fast moving and changing field § Every week something “big” happens in LLM space § Frameworks & ecosystem are evolving together with LLMs § Closed vs open LLMs § Competition drives invention & advancement § SLMs: specialized, fine-tuned for domains § SISO (sh*t in, sh*t out) § Quality of results heavily depends on your data & input Multimodale Large Language Models (LMMs) als Kern moderner Business-Anwendungen – in Action Current state 29

Slide 30

Slide 30 text

Multimodale Large Language Models (LMMs) als Kern moderner Business-Anwendungen – in Action 2024: The year of SLMs & CPU inference 30

Slide 31

Slide 31 text

Thank you! Christian Weyer https://thinktecture.com/christian-weyer 31