‘Talk to your systems’ Strukturierte Daten & Tool-Integration - LLM-Anbindung einfach gemacht Christian Weyer | Co-Founder & CTO | Thinktecture AG | [email protected]

§ Technology catalyst § AI-powered solutions § Pragmatic end-to-end architectures § Microsoft Regional Director § Microsoft MVP for AI § Google GDE for Web Technologies [email protected] @christianweyer https://www.thinktecture.com 'Talk to your systems' Strukturierte Daten & Tool-Integration - LLM-Anbindung einfach gemacht Christian Weyer Co-Founder & CTO @ Thinktecture AG 2

'Talk to your systems' Strukturierte Daten & Tool-Integration - LLM-Anbindung einfach gemacht Talk to your systems Why? What? How? 3

'Talk to your systems' Strukturierte Daten & Tool-Integration - LLM-Anbindung einfach gemacht TALK TO YOUR SYSTEMS WHY? 4

'Talk to your systems' Strukturierte Daten & Tool-Integration - LLM-Anbindung einfach gemacht Human language rocks Extending access to software 5

'Talk to your systems' Strukturierte Daten & Tool-Integration - LLM-Anbindung einfach gemacht A classical UI – strong with certain use cases 6

One possible UX pattern 'Talk to your systems' Strukturierte Daten & Tool-Integration - LLM-Anbindung einfach gemacht Language-enabled “UIs” 7

'Talk to your systems' Strukturierte Daten & Tool-Integration - LLM-Anbindung einfach gemacht LLMs Enabling new scenarios 8

§ LLMs are always part of end-to-end architectures § Client apps (Web, desktop, mobile) § Services with APIs § Databases § etc. § An LLM is ‘just’ an additional asset in your architecture § Enabling human language understanding & generation § It is not the Holy Grail for everything § Enable human language as a first-class citizen 🤯 'Talk to your systems' Strukturierte Daten & Tool-Integration - LLM-Anbindung einfach gemacht End-to-end architectures with LLMs Clients Services LLMs Desktop Web Mobile Service A Service B Service C API Gateway Monitoring LLM 1 LLM 2 9

'Talk to your systems' Strukturierte Daten & Tool-Integration - LLM-Anbindung einfach gemacht It’s just HTTP APIs Inference, FTW 10

'Talk to your systems' Strukturierte Daten & Tool-Integration - LLM-Anbindung einfach gemacht Your LLM OpenAI & beyond 11

§ Llama, Mistral, Qwen families show big potential § Success factors § Use case § Parameter size § Quantization § GPU processing power needed § CPU optimization on its way § Local inference runtimes with APIs § E.g. llama.cpp, ollama, llamafile, vLLM 'Talk to your systems' Strukturierte Daten & Tool-Integration - LLM-Anbindung einfach gemacht Open-source LLMs thrive § Local UIs § E.g. Open WebUI 12

'Talk to your systems' Strukturierte Daten & Tool-Integration - LLM-Anbindung einfach gemacht TALK TO YOUR SYSTEMS HOW? 13

'Talk to your systems' Strukturierte Daten & Tool-Integration - LLM-Anbindung einfach gemacht Most convenient platform for developers today to work with Gen AI 14

'Talk to your systems' Strukturierte Daten & Tool-Integration - LLM-Anbindung einfach gemacht Prompting Talk to me! 15

'Talk to your systems' Strukturierte Daten & Tool-Integration - LLM-Anbindung einfach gemacht ‘Function’ Calling Give it schema! 16

'Talk to your systems' Strukturierte Daten & Tool-Integration - LLM-Anbindung einfach gemacht Pydantic & Instructor Make it easier! 17

'Talk to your systems' Strukturierte Daten & Tool-Integration - LLM-Anbindung einfach gemacht TALK TO YOUR SYSTEMS WHAT? 18

'Talk to your systems' Strukturierte Daten & Tool-Integration - LLM-Anbindung einfach gemacht End-to-End Talking to your applications 19

'Talk to your systems' Strukturierte Daten & Tool-Integration - LLM-Anbindung einfach gemacht Talk to Thinktecture Angular PWA Speech-to-Text Internal Gateway (Python FastAPI) LLM / SLM Text-to-Speech Transcribe spoken text Transcribed text Check for experts availability with text Extract { experts, booking times } from text Structured JSON data (Tool calling) Generate response with availability Response Response with experts availability 🗣 🔉 Speech-to-text for response Response audio Internal Business API (node.js – veeeery old) Query Availability API Availability When is CL…? CL will be… 20

Filling Angular forms with human language input – TypeScript-only protected readonly formGroup = this.fb.group({ firstName: [’’], lastName: [’’], addressLine1: [’’], addressLine2: [’’], city: [’’], state: [’’], zip: [’’], country: [’’] }); 'Talk to your systems' Strukturierte Daten & Tool-Integration - LLM-Anbindung einfach gemacht Smart form filling OK, nice – so here is my address then:: Peter Schmitt, Rheinstr. 7 in Schkeuditz – postcode is 04435, BTW. Smart Form Filler (TS code & LLM) 21

'Talk to your systems' Strukturierte Daten & Tool-Integration - LLM-Anbindung einfach gemacht TALK TO YOUR SYSTEMS RECAP 22

§ Human language enables new powerful use cases & access to our software § Always use structured output § Structured output is the secret sauce for integrating LLMs into your application architectures § Consider applying the Maybe pattern § Brings more robustness § Function Calling can be flaky § Especially with smaller models § Do not use frameworks that ‘auto-magically’ map Function Calling results to local code § Always validate return data! § Instructor is a helpful library to boost LLM use cases § Implements lots of best practices § Supports any LLM / SLM § Integrates with FastAPI 'Talk to your systems' Strukturierte Daten & Tool-Integration - LLM-Anbindung einfach gemacht Recap & Recommendations 23

Thank you! Christian Weyer https://thinktecture.com/christian-weyer https://github.com/thinktecture-labs/talk-to-your-systems https://github.com/thinktecture-labs/smart-form-filler 24