App Automation with LLMS & Generative AI

In this session, Max Marschall introduces participants to the future of web app automation.

He demonstrates how the integration of Large Language Models (LLMs) with generative AI creates new possibilities in app development.

Instead of rigid function calls, he establishes an adaptive system that allows both user interaction and self-learning automation.

The audience will learn how a flexible, AI-driven infrastructure transforms web apps and enables both users and AI to control and personalise the applications in real time.

Marshall's approach breaks with traditional patterns and offers an insight into the self-driven and user-centred future of web technology

Transcript

1. Max Marschall @MaxOSchulte App Automation with LLMs & Generative AI Session @MaxOSchulte | Thinktecture AG 1
2. Max Marschall Consultant & Developer @ Thinktecture AG @MaxOSchulte Babylon.js , Angular , Ionic & GenAi App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 2
3. DEMO App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 3
4. LLM vs. ML Generative AI "understands" us General understanding of ... Depending on training data set size App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 4
5. What Generative AI is not / can not Intelligent Making decisions Problems understanding non-linear content Drawing conclusions App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 5
6. Use Case Generate content Summarize content Automation Simplify complexity ... App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 6
7. UI / UX Avoid auto-commit (e.g save) Enable feedback Highlight probability Clear entry point into AI workflows Progress / generation indicators Suggest common actions App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 7
8. Development mindset Shit in / Shit out "Talking to fast learning & smart junior" No default assumptions Documentation is key App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 8
9. Prompting Multiple messages = prompt composition Single-Shot prompt Few/Multi-Shot prompt Roles System User Assistant App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 9
10. System Prompt Base structure Rules Configuration Intent & purpose App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 10
11. Tool Former "Function Calling" Stateless "Single-Shot" Structured & "reliable" Tool definitions besides prompt LLM defines what Tool to call with input User takes action App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 11
12. Tool Former "Function Calling" Stateless "Single-Shot" Structured & "reliable" Tool definitions besides prompt LLM defines what Tool to call with input User takes action App Automation with LLMS & Generative AI Session https://python.langchain.com/docs/use_cases/tool_use/ @MaxOSchulte | Thinktecture AG 12
13. Chains Pre-defined "Pipeline" Data-(pre-)processing Prompt building https://python.langchain.com/docs/modules/chains/ App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 13
14. Agents Tools & prompts as input Decides on a "path" Can make multiple Tool calls App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 14
15. Agents Tools & prompts as input Decides on a "path" Can make multiple Tool calls App Automation with LLMS & Generative AI Session https://python.langchain.com/docs/use_cases/tool_use/ @MaxOSchulte | Thinktecture AG 15
16. Agent vs. Chain The core idea of agents is to use a language model to choose a sequence of actions to take. In chains, a sequence of actions is hardcoded (in code). In agents, a language model is used as a reasoning engine to determine which actions to take and in which order. https://python.langchain.com/docs/modules/agents/ App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 16
17. CODE App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 17
18. Creating Tools from existing code TypeScript Langchain.js for TypeScript Zod for runtime type information App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 18
19. Creating Tools from existing code TypeScript Langchain.js for TypeScript Zod for runtime type information App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 19
20. Creating Tools from existing code TypeScript + Python LangChain \ OpenAi for Python Pydantic "Manual" AST Parsing [...] const className = node.name.text; const classDoc = getDocumentation(node, checker); const methods = node.members .filter(isMethodDeclaration) .filter(getPublic) .filter(method => tsGetDecorator(method)?.some(decorator => decorator.getText() === '@AiTool')) .map(method => parseMethod(method, sourceFile, checker)); return methods.map(method => ({ type: 'function', className, classDocumentation: classDoc ?? '', function: method, })); [...] App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 20
21. Drawbacks using generative AI Long-running tasks Remote models Possible many tokes --> high cost App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 21
22. Mitigation Chains with filter & processing Multi-Model Smarter re-selection App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 22
23. Client Side vs. Server Side Client Server Easy Integration ✅ Tooling ✅ Python (Ecosystem) ✅ TypeScript / JS ✅ ✅ Secrets / Security ✅ Schema Validation ✅ ✅ App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 23
24. Client Side vs. Server Side Client → Fast, manageable use-cases Server → Full fledged power and complexity App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 24
25. Tooling App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 25
26. Conclusion Document well Create simple APIs Vendor APIs are going to change Rapidly evolving App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 26
27. Thank You ! @MaxOSchulte https://www.thinktecture.com/thinktects/max-marschall App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 27