Web app automation using LLMs and generative AI

Max Marschall presents an innovative approach to web app automation, highlighting the synergies between Large Language Models (LLMs) and generative AI in app development.
This talk focuses on an adaptive system extending conventional function calls, enabling dynamic user interaction and self-learning automation.
It explores how AI-driven flexibility revolutionises web apps, granting real-time control and personalisation to both users and AI.
Marschall's method extends traditional methods, providing a glimpse into the autonomous and user-oriented future of web technologies.

Transcript

1. Max Marschall @MaxOSchulte Web app automation using LLMs and generative AI MaxOSchulte Thinktecture AG 1
2. Max Marschall Consultant & Developer @ Thinktecture AG @MaxOSchulte Babylon.js , Angular , Ionic & GenAi Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 2
3. Intention Augmenting an App with an assistant, that helps the user based on displayed context. Allowing an assistant to act for me in that context. Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 3
4. DEMO Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 4
5. Intention Augmenting an App with an assistant, that helps the user based on displayed context. Allowing an assistant to act for me in that context. Problem: Integrating GenAI Metainformation Tool calls Leaking secrets "Other" system integration Solution AST parsing TS decorators Client-side "agents" Barebones Angular OpenAI Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 5
6. Architecture Client "Plain" Angular AST parsing Server Python / TypeScript FastAPI / Azure Function OpenAI LangFuse (LangChain) Gen AI Provider OpenAI Whisper Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 6
7. CODE Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 7
8. Meta information → →∑ Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 8
9. Meta information Typescript Functions /** Class Documentation [...] */ export class DialogComponent { /** * Displays a note {@link Note} for the user to save, edit or discard. * @param data Note data that is displayed to the user. */ @AiTool public showNote(data: Note): void { this.dialog.updateDialog({ role: 'Note', message: 'Note ...', data }); } } Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 9
10. Meta information Typescript Models export interface Note extends BaseData { /** Field is used to Filter Data for this specific type. Must be 'Note'! */ type: 'Note'; /** Title of this note */ title: string; /** List of topic of this note, comma separated */ tags: string; /** Actual text of this note, markdown preferred */ text: string; } Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 10
11. Provide Functionality Generated Tool definition { "type": "function", "className": "DialogComponent", "classDocumentation": "", "function": { "name": "showNote", "title": "showNote", "type": "function", "description": "Displays a note {@link Note} for the user to save, edit or discard.", "parameters": { "type": "object", "properties": { "data": { "title": "data", "description": "Note data that is displayed to the user.", "type": "object", "properties": { "type": { "description": "Field is used to Filter Data for this specific type. Must be 'Note'!", "type": "string", "const": "Note" }, "title": { "description": "Title of this note", "type": "string" }, "tags": { "description": "List of topic of this note, comma separated", "type": "string" }, "text": { "description": "Actual text of this note, markdown preferred", "type": "string" }, "id": { "description": "GUID", "type": "string" }, "date": { "description": "UTC string", "type": "string" } }, "required": [ "date", "id", "tags", "text", "title", "type" ], "$schema": "http://json-schema.org/draft-07/schema#" } } }, "method": "(data: Note) => void", "returnType": "void", "required": [ "data" ], "decorators": [] } } Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 11
12. Assistant Sequence Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 center MaxOSchulte Thinktecture AG 12
13. Provide Functionality Tools Agents Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 https://python.langchain.com/docs/use_cases/tool_use/ MaxOSchulte Thinktecture AG 13
14. Provide Functionality Tools Agents Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 https://python.langchain.com/docs/use_cases/tool_use/agents/ MaxOSchulte Thinktecture AG 14
15. Assistant Sequence Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 center MaxOSchulte Thinktecture AG 15
16. Client-Side vs. Server-Side Client Server Easy Integration Tooling Python (Ecosystem) TypeScript / JS Secrets / Security Schema Validation SoC ( ) Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 16
17. What Generative AI cannot do make decisions understand non-linear content draw conclusions GenAI is not Intelligent Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 17
18. Use Cases Generate content Summarize content Automation Simplify complexity Analysis ... Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 drop-sadow MaxOSchulte Thinktecture AG 18
19. UI / UX Avoid auto-commit Enable feedback Highlight probability Clear entry point into AI workflows Progress / generation indicators Suggest common actions Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 19
20. Development mindset "Talking to fast learning & smart junior" No default assumptions Documentation is key Examples are great Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 20
21. Lessons Learned Client-side for PoC Easy transition to server-side Write docs for dummies Unambiguous tools Unambiguous context An Agent is just a loop LLMs do not actually call functions - you do! Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 21
22. Thank You ! https://www.thinktecture.com/thinktects/max-marschall Web app automation using LLMs and generative AI WeAreDevelopers World Congress 2024 MaxOSchulte Thinktecture AG 22