‘Talk to your systems’ Structured data & tool integration with LLMs Christian Weyer | Co-Founder & CTO | Thinktecture AG | [email protected] 

§ Technology catalyst § AI-powered solutions § Pragmatic end-to-end architectures § Microsoft Regional Director § Microsoft MVP for AI § Google GDE for Web AI [email protected] @christianweyer https://www.thinktecture.com 'Talk to your systems' Structured data & tool integration with LLMs Christian Weyer Co-Founder & CTO @ Thinktecture AG 2 

'Talk to your systems' Structured data & tool integration with LLMs Our journey 3 The Universal Interface Integrating LLMs Talk to your Systems End-to-End scenarios AI all-the-things? 

'Talk to your systems' Structured data & tool integration with LLMs AI ALL-THE-THINGS? 4 

'Talk to your systems' Structured data & tool integration with LLMs A.I. - 1985 5 

'Talk to your systems' Structured data & tool integration with LLMs AI all-the-things? Data Science Artificial Intelligence Machine Learning Unsupervised, supervised, reinforcement learning Deep Learning ANN, CNN, RNN etc. NLP (Natural Language Processing) Generative AI GAN, VAE, Transformers etc. Image / Video Generation GAN, VAE Large Language Models Transformers 6 

'Talk to your systems' Structured data & tool integration with LLMs THE UNIVERSAL INTERFACE 7 

'Talk to your systems' Structured data & tool integration with LLMs Human language rocks Extending access to software 8 

'Talk to your systems' Structured data & tool integration with LLMs Classical applications & UIs 9 

'Talk to your systems' Structured data & tool integration with LLMs Language-enabled “UIs” – Talk-to-TT sample 10 

'Talk to your systems' Structured data & tool integration with LLMs INTEGRATING LLMS 11 

§ LLMs are always part of end-to-end architectures § Client apps (Web, desktop, mobile) § Services with APIs § Databases § etc. § An LLM is ‘just’ an additional asset in your architecture § It is not the Holy Grail for everything § Enable human language as a first-class citizen 🤯 'Talk to your systems' Structured data & tool integration with LLMs End-to-end architectures with LLMs Clients Services LLMs Desktop Web Mobile Service A Service B Service C API Gateway Monitoring LLM 1 LLM 2 12 

'Talk to your systems' Structured data & tool integration with LLMs It’s just HTTP APIs Inference, FTW 13 

'Talk to your systems' Structured data & tool integration with LLMs Most convenient platform to work with genuine Gen AI 14 

'Talk to your systems' Structured data & tool integration with LLMs TALK TO YOUR SYSTEMS 15 

'Talk to your systems' Structured data & tool integration with LLMs Prompting Talk to me! 16 

'Talk to your systems' Structured data & tool integration with LLMs ‘Function’ Calling Give it schema! 17 

§ Predefined JSON schema structure § All major libs support tool calling with abstractions § OpenAI SDKs § Langchain § Semantic Kernel § Etc. 'Talk to your systems' Structured data & tool integration with LLMs OpenAI Tool calling – plain HTTP calls curl https://api.openai.com/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "model": "gpt-4o", "messages": [ { "role": "user", "content": "What is the weather like in Boston?" } ], "tools": [ { "type": "function", "function": { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"] } }, "required": ["location"] } } } ], "tool_choice": "auto" }' https://platform.openai.com/docs/api-reference/chat/create#chat-create-tools 18 

§ External metadata, e.g. JSON description/files § .NET: Reflection § Python: Pydantic § JS / TypeScript: nothing out of the box (yet) § Typically using Zod 'Talk to your systems' Structured data & tool integration with LLMs Provide metadata about your functions / tools 19 

'Talk to your systems' Structured data & tool integration with LLMs Pydantic & Instructor Make it easier! 20 

'Talk to your systems' Structured data & tool integration with LLMs END-TO-END SCENARIOS 21 

Semantic routing 'Talk to your systems' Structured data & tool integration with LLMs "Talk to your systems" - for Availability queries 22 Web App / Watch App Speech-to-Text Internal Gateway (Python FastAPI) LLM / SLM Text-to-Speech Transcribe spoken text Transcribed text Check for experts availability with text Extract { experts, booking times } from text Structured JSON data (Function calling) Generate response with availability Response Response with experts availability 🔉 Speech-to-text for response Response audio Internal Business API (node.js – veeeery old) Query Availability API Availability When is CL…? CL will be… 

Filling Angular forms with human language input – TypeScript-only protected readonly formGroup = this.fb.group({ firstName: [’’], lastName: [’’], addressLine1: [’’], addressLine2: [’’], city: [’’], state: [’’], zip: [’’], country: [’’] }); 'Talk to your systems' Structured data & tool integration with LLMs Smart form filling OK, nice – so here is my address then: Peter Schmitt, Rheinstr. 7 in Schkeuditz – postcode is 04435, BTW. Smart Form Filler (TS code & LM) 23 

'Talk to your systems' Structured data & tool integration with LLMs One more thing … 24 

'Talk to your systems' Structured data & tool integration with LLMs AGENTS 25 ARE LOOPS OVER TOOL CALLS 

§ Give LLM more capabilities with tools § Access data and other functionality § Connect your APIs, applications, and environments 'Talk to your systems' Structured data & tool integration with LLMs Extending LLM capabilities for complex tasks “Do x!” LLM “Do x!” System prompt Tool 1 metadata Tool 2 metadata ... { “answer”: “toolcall”, “tool” : “tool1” “args”: […] } 26 

'Talk to your systems' Structured data & tool integration with LLMs ReAct: Reasoning and Acting https://arxiv.org/abs/2210.03629 27 

§ Involve an LLM making decisions through ‘thinking’ § Which actions to take (‘thought’) § Taking that action (executed via your code) § Seeing an observation § Repeating until done § Keep human in the loop for missing data, validation, approval 'Talk to your systems' Structured data & tool integration with LLMs ReAct: Reasoning and Acting 28 

'Talk to your systems' Structured data & tool integration with LLMs ReAct – illustrated LLM My code / agent Query Your API Your database Prompt Tools Final answer Answer ❓ ❓ ❗ 💡 29 § Tool calling is being standardized through MCP 

'Talk to your systems' Structured data & tool integration with LLMs RECAP 30 

'Talk to your systems' Structured data & tool integration with LLMs Talk to your systems 31 Integrating LLMs via APIs Structured output & Tool calling End-to-End scenarios ‘Lightweight’ Agents The Universal Interface 

'Talk to your systems' Structured data & tool integration with LLMs Talk to your systems 32 Integrating LLMs via APIs Structured output & Tool calling End-to-End scenarios Lightweight Agents The Universal Interface LLMs everywhere AI solutions are 10% AI. And 100% software engineering. 

Thank you! Christian Weyer https://thinktecture.com/christian-weyer https://github.com/thinktecture-labs/talk-to-your-systems https://github.com/thinktecture-labs/smart-form-filler 33