Max Marschall @MaxOSchulte Web app automation using LLMs and generative AI MaxOSchulte Thinktecture AG 1

Max Marschall Consultant & Developer @ Thinktecture AG @MaxOSchulte Babylon.js , Angular , Ionic & GenAi Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 2

Intention Augmenting an App with an assistant, that helps the user based on displayed context. Allowing an assistant to act for me in that context. Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 3

DEMO Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 4

Intention Augmenting an App with an assistant, that helps the user based on displayed context. Allowing an assistant to act for me in that context. Problem: Integrating GenAI Metainformation Tool calls Leaking secrets "Other" system integration Solution AST parsing TS decorators Client-side "agents" Barebones Angular OpenAI Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 5

Architecture Client "Plain" Angular AST parsing Server Python / TypeScript FastAPI / Azure Function OpenAI LangFuse (LangChain) Gen AI Provider OpenAI Whisper Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 6

CODE Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 7

Meta information → →∑ Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 8

Meta information Typescript Functions /** Class Documentation [...] */ export class DialogComponent { /** * Displays a note {@link Note} for the user to save, edit or discard. * @param data Note data that is displayed to the user. */ @AiTool public showNote(data: Note): void { this.dialog.updateDialog({ role: 'Note', message: 'Note ...', data }); } } Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 9

Meta information Typescript Models export interface Note extends BaseData { /** Field is used to Filter Data for this specific type. Must be 'Note'! */ type: 'Note'; /** Title of this note */ title: string; /** List of topic of this note, comma separated */ tags: string; /** Actual text of this note, markdown preferred */ text: string; } Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 10

Provide Functionality Generated Tool definition { "type": "function", "className": "DialogComponent", "classDocumentation": "", "function": { "name": "showNote", "title": "showNote", "type": "function", "description": "Displays a note {@link Note} for the user to save, edit or discard.", "parameters": { "type": "object", "properties": { "data": { "title": "data", "description": "Note data that is displayed to the user.", "type": "object", "properties": { "type": { "description": "Field is used to Filter Data for this specific type. Must be 'Note'!", "type": "string", "const": "Note" }, "title": { "description": "Title of this note", "type": "string" }, "tags": { "description": "List of topic of this note, comma separated", "type": "string" }, "text": { "description": "Actual text of this note, markdown preferred", "type": "string" }, "id": { "description": "GUID", "type": "string" }, "date": { "description": "UTC string", "type": "string" } }, "required": [ "date", "id", "tags", "text", "title", "type" ], "$schema": "http://json-schema.org/draft-07/schema#" } } }, "method": "(data: Note) => void", "returnType": "void", "required": [ "data" ], "decorators": [] } } Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 11

Assistant Sequence Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 center MaxOSchulte Thinktecture AG 12

Provide Functionality Tools Agents Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 https://python.langchain.com/docs/use_cases/tool_use/ MaxOSchulte Thinktecture AG 13

Provide Functionality Tools Agents Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 https://python.langchain.com/docs/use_cases/tool_use/agents/ MaxOSchulte Thinktecture AG 14

Assistant Sequence Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 center MaxOSchulte Thinktecture AG 15

Client-Side vs. Server-Side Client Server Easy Integration Tooling Python (Ecosystem) TypeScript / JS Secrets / Security Schema Validation SoC ( ) Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 16

What Generative AI cannot do make decisions understand non-linear content draw conclusions GenAI is not Intelligent Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 17

Use Cases Generate content Summarize content Automation Simplify complexity Analysis ... Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 drop-sadow MaxOSchulte Thinktecture AG 18

UI / UX Avoid auto-commit Enable feedback Highlight probability Clear entry point into AI workflows Progress / generation indicators Suggest common actions Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 19

Development mindset "Talking to fast learning & smart junior" No default assumptions Documentation is key Examples are great Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 20

Lessons Learned Client-side for PoC Easy transition to server-side Write docs for dummies Unambiguous tools Unambiguous context An Agent is just a loop LLMs do not actually call functions - you do! Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 21

Thank You ! https://www.thinktecture.com/thinktects/max-marschall Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 22