App Automation with LLMS & Generative AI

by Max Schulte

Slide 1

Slide 1 text

Max Marschall @MaxOSchulte App Automation with LLMs & Generative AI Session @MaxOSchulte | Thinktecture AG 1

Slide 2

Slide 2 text

Max Marschall Consultant & Developer @ Thinktecture AG @MaxOSchulte Babylon.js , Angular , Ionic & GenAi App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 2

Slide 3

Slide 3 text

DEMO App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 3

Slide 4

Slide 4 text

LLM vs. ML Generative AI "understands" us General understanding of ... Depending on training data set size App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 4

Slide 5

Slide 5 text

What Generative AI is not / can not Intelligent Making decisions Problems understanding non-linear content Drawing conclusions App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 5

Slide 6

Slide 6 text

Use Case Generate content Summarize content Automation Simplify complexity ... App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 6

Slide 7

Slide 7 text

UI / UX Avoid auto-commit (e.g save) Enable feedback Highlight probability Clear entry point into AI workflows Progress / generation indicators Suggest common actions App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 7

Slide 8

Slide 8 text

Development mindset Shit in / Shit out "Talking to fast learning & smart junior" No default assumptions Documentation is key App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 8

Slide 9

Slide 9 text

Prompting Multiple messages = prompt composition Single-Shot prompt Few/Multi-Shot prompt Roles System User Assistant App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 9

Slide 10

Slide 10 text

System Prompt Base structure Rules Configuration Intent & purpose App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 10

Slide 11

Slide 11 text

Tool Former "Function Calling" Stateless "Single-Shot" Structured & "reliable" Tool definitions besides prompt LLM defines what Tool to call with input User takes action App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 11

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Chains Pre-defined "Pipeline" Data-(pre-)processing Prompt building https://python.langchain.com/docs/modules/chains/ App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 13

Slide 14

Slide 14 text

Agents Tools & prompts as input Decides on a "path" Can make multiple Tool calls App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 14

Slide 15

Slide 15 text

Agents Tools & prompts as input Decides on a "path" Can make multiple Tool calls App Automation with LLMS & Generative AI Session https://python.langchain.com/docs/use_cases/tool_use/ @MaxOSchulte | Thinktecture AG 15

Slide 16

Slide 16 text

Agent vs. Chain The core idea of agents is to use a language model to choose a sequence of actions to take. In chains, a sequence of actions is hardcoded (in code). In agents, a language model is used as a reasoning engine to determine which actions to take and in which order. https://python.langchain.com/docs/modules/agents/ App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 16

Slide 17

Slide 17 text

CODE App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 17

Slide 18

Slide 18 text

Creating Tools from existing code TypeScript Langchain.js for TypeScript Zod for runtime type information App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 18

Slide 19

Slide 19 text

Creating Tools from existing code TypeScript Langchain.js for TypeScript Zod for runtime type information App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 19

Slide 20

Slide 20 text

Creating Tools from existing code TypeScript + Python LangChain \ OpenAi for Python Pydantic "Manual" AST Parsing [...] const className = node.name.text; const classDoc = getDocumentation(node, checker); const methods = node.members .filter(isMethodDeclaration) .filter(getPublic) .filter(method => tsGetDecorator(method)?.some(decorator => decorator.getText() === '@AiTool')) .map(method => parseMethod(method, sourceFile, checker)); return methods.map(method => ({ type: 'function', className, classDocumentation: classDoc ?? '', function: method, })); [...] App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 20

Slide 21

Slide 21 text

Drawbacks using generative AI Long-running tasks Remote models Possible many tokes --> high cost App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 21

Slide 22

Slide 22 text

Mitigation Chains with filter & processing Multi-Model Smarter re-selection App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 22

Slide 23

Slide 23 text

Client Side vs. Server Side Client Server Easy Integration ✅ Tooling ✅ Python (Ecosystem) ✅ TypeScript / JS ✅ ✅ Secrets / Security ✅ Schema Validation ✅ ✅ App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 23

Slide 24

Slide 24 text

Client Side vs. Server Side Client → Fast, manageable use-cases Server → Full fledged power and complexity App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 24

Slide 25

Slide 25 text

Tooling App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 25

Slide 26

Slide 26 text

Conclusion Document well Create simple APIs Vendor APIs are going to change Rapidly evolving App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 26

Slide 27

Slide 27 text

Thank You ! @MaxOSchulte https://www.thinktecture.com/thinktects/max-marschall App Automation with LLMS & Generative AI Session @MaxOSchulte | Thinktecture AG 27