‘Talk to your systems’ Integrating Gen AI into your architectures with structured LLM output Christian Weyer | Co-Founder & CTO | Thinktecture AG | [email protected]

§ Technology catalyst § AI-powered solutions § Pragmatic end-to-end architectures § Microsoft Regional Director § Microsoft MVP for AI § Google GDE for Web Technologies [email protected] @christianweyer https://www.thinktecture.com 'Talk to your systems' Integrating Gen AI into your architectures with structured LLM output Christian Weyer Co-Founder & CTO @ Thinktecture AG 2

'Talk to your systems' Integrating Gen AI into your architectures with structured LLM output 3 Talk to your systems Why? What? How?

'Talk to your systems' Integrating Gen AI into your architectures with structured LLM output TALK TO YOUR SYSTEMS WHY? 4

'Talk to your systems' Integrating Gen AI into your architectures with structured LLM output 5 Human language rocks Extending access to software

One possible UX pattern 'Talk to your systems' Integrating Gen AI into your architectures with structured LLM output A language-enabled “UI” 6

'Talk to your systems' Integrating Gen AI into your architectures with structured LLM output 7 LLMs Use wisely

§ LLMs are always part of end-to-end architectures § Client apps (Web, desktop, mobile) § Services with APIs § Databases § etc. § An LLM is ‘just’ an additional asset in your architecture § Enabling human language understanding & generation § It is not the Holy Grail for everything § Enable human language as a first-class citizen 'Talk to your systems' Integrating Gen AI into your architectures with structured LLM output End-to-end architectures with LLMs 8 Clients Services LLMs Desktop Web Mobile Service A Service B Service C API Gateway Monitoring LLM 1 LLM 2

'Talk to your systems' Integrating Gen AI into your architectures with structured LLM output It’s just HTTP APIs Inference, FTW 9

'Talk to your systems' Integrating Gen AI into your architectures with structured LLM output 10 Your LLM OpenAI & beyond

§ Llama, Mistral, Qwen families show big potential § Success factors § Use case § Parameter size § Quantization § Processing power needed § CPU optimization on its way § Local inference runtimes with APIs § E.g. llama.cpp, ollama, llamafile, vLLM 'Talk to your systems' Integrating Gen AI into your architectures with structured LLM output Open-source LLMs thrive 11 § Local UIs § E.g. Open WebUI

'Talk to your systems' Integrating Gen AI into your architectures with structured LLM output TALK TO YOUR SYSTEMS HOW? 12

'Talk to your systems' Integrating Gen AI into your architectures with structured LLM output Most convenient platform for developers today to work with Gen AI 13

'Talk to your systems' Integrating Gen AI into your architectures with structured LLM output 14 Prompting Talk to me!

'Talk to your systems' Integrating Gen AI into your architectures with structured LLM output 15 JSON Mode Give it structure!

'Talk to your systems' Integrating Gen AI into your architectures with structured LLM output 16 ‘Function’ Calling Give it schema!

'Talk to your systems' Integrating Gen AI into your architectures with structured LLM output 17 Strict Mode Make it robust!

'Talk to your systems' Integrating Gen AI into your architectures with structured LLM output 18 Pydantic & Instructor Make my life easy!

'Talk to your systems' Integrating Gen AI into your architectures with structured LLM output 19 Talking to systems No auto-magic

'Talk to your systems' Integrating Gen AI into your architectures with structured LLM output TALK TO YOUR SYSTEMS WHAT? 20

'Talk to your systems' Integrating Gen AI into your architectures with structured LLM output 21 End-to-End examples Talking to your applications

'Talk to your systems' Integrating Gen AI into your architectures with structured LLM output Talk to Thinktecture 22 Angular PWA Speech-to-Text Internal Gateway (Python FastAPI) LLM / SLM Text-to-Speech Transcribe spoken text Transcribed text Check for experts availability with text Extract { experts, booking times } from text Structured JSON data (Tool calling) Generate response with availability Response Response with experts availability 🗣 🔉 Speech-to-text for response Response audio Internal Business API (node.js – veeeery old) Query Availability API Availability When is CL…? CL will be…

Filling Angular forms with human language input protected readonly formGroup = this.fb.group({ firstName: [’’], lastName: [’’], addressLine1: [’’], addressLine2: [’’], city: [’’], state: [’’], zip: [’’], country: [’’] }); 'Talk to your systems' Integrating Gen AI into your architectures with structured LLM output Smart Form Filler OK, nice – so here is my address then:: Peter Schmitt, Rheinstr. 7 in Schkeuditz – postcode is 04435, BTW. 23 Smart Form Filler (TS code & LLM)

'Talk to your systems' Integrating Gen AI into your architectures with structured LLM output TALK TO YOUR SYSTEMS RECAP 24

§ Human language enables new powerful use cases & access to our software § Always use structured output § Structured output is the secret sauce for integrating LLMs into your application architectures § Consider applying the Maybe pattern § Brings more robustness § Function Calling can be flaky § Especially with smaller models § Do not use frameworks that ‘auto-magically’ map Function Calling results to local code § Always validate return data! § Instructor is a helpful library to boost LLM use cases § Implements lots of best practices § Supports any LLM / SLM § Integrates with FastAPI 'Talk to your systems' Integrating Gen AI into your architectures with structured LLM output Recap & Recommendations 25

Thank you! Christian Weyer https://thinktecture.com/christian-weyer 26 https://github.com/thinktecture-labs/talk-to-your-systems https://github.com/thinktecture-labs/smart-form-filler