GenAI and LLMs in exiting web-apps — automate everything?

Join this session to discover how to seamlessly integrate generative AI and Large Language Models (LLMs) into your existing web applications. Max Marschall will demonstrate techniques such as tool-calling and client-sided agents, providing users with new opportunities and tools within their familiar workflows.

The session will discuss an architecture required to incorporate generative AI into web applications, without requiring major changes. You’ll learn best practices for achieving a seamless integration, as well as valuable lessons from real-world implementations.

Max Marschall also explore the balance between client-side execution and server-side calculations, discussing various approaches and their pros and cons. By the end of the session, you’ll gain a deeper understanding of the technical considerations necessary for successful implementation of AI and LLMs into your own web applications.

Learning objectives
Use-Cases integrating GenAI
Pitfalls & Best-Practices
How to integrate GenAI
UI / UX benefits and drawbacks

Transcript

1. Max Marschall @MaxOSchulte Consultant / Developer GenAI and LLMs in exiting web-apps automate everything?
2. Arti fi cial Intelligence Machine Learning Data Science NLP A Deep Learning Generative AI Large Language Models Image / Video Generation
3. Bla bla bla … Intention
4. Bla bla bla … Generation of… Intention Summarisation of… Automation of… Simpli fi cation of…
5. Bla bla bla … “Augmenting an App with an assistant, that helps the user based on displayed context. Allowing an assistant to act for me in that context.” Intention
6. Intention Bla bla bla … “Augmenting an App with an assistant, that helps the user based on displayed context. Allowing an assistant to act for me in that context.” Problem: Integration GenAI Metainformation Tool calls Leaking secrets System integration Solutions AST parsing TS Decorators Client-side "agents" Barebones Angular OpenAI
7. Intention Bla bla bla … “Augmenting an App with an assistant, that helps the user based on displayed context. Allowing an assistant to act for me in that context.”
8. Tools & Functions Process Think Display Bla bla bla …
9. User Prompt • Voice input • Text input • Image input • Unstructured • Universal interface • Natural language as fi rst-class input
10. • Prompts • RAG • Sanitising • Guards • Vector-Database • Tool selection • Additional data (Processing)
11. • System prompt • Base information • Rules • Intention • Output format Processing You are an friendly helpfull assistent proficient in creating and managing forms. You will receive information on the state of an app. If you feel there a … System Prompt https://docs.anthropic.com/en/release-notes/system-prompts#oct-22nd-2024 You are a data analyst API capable of sentiment analysis that responds in JSON. The JSON schema should include { "sentiment_analysis": { "sentiment": "string (positive, negative, neutral)", "confidence_score": "number (0-1)" # Include additional fields as required } https://console.groq.com/docs/text-chat Prompts
12. • State • Additional information • Generation examples Context /** Additional Information: basic list of static pages shown in the side navigation. */ [ { title: 'Contributions', icon: 'question_answer', link: ['/contributions'] }, { title: 'Conferences', icon: 'podium', link: ['/conferences'] }, { title: 'Speakers', icon: 'person', link: ['/speakers'] }, { title: 'Collections', icon: 'category', link: ['/collections'] }, ];
13. Tools & Functions export interface NavigationPath { /** * Absolute path to navigate * e.g. {path: '/collections/:id'} * ":id" is replaced by an actual entity id. */ resolvedPath: string; } /** * Accepts a resolved sitemap entry and navigates to it. */ public navigate(navigation: NavigationPath): Promise<boolean> { return this.router.navigateByUrl('/' + navigation.resolvedPath); }
14. export interface NavigationPath { /** * Absolute path to navigate * e.g. {path: '/collections/:id'} * ":id" is replaced by an actual entity id. */ resolvedPath: string; } /** * Accepts a resolved sitemap entry and navigates to it. */ @AiTool public navigate(navigation: NavigationPath): Promise<boolean> { return this.router.navigateByUrl('/' + navigation.resolvedPath); } Tools & Functions { "type": "function", "function": { "name": "navigate", "description": "Accepts a resolved sitemap entry …, "parameters": { "type": "object", "properties": { "resolvedPath": { "title": "resolvedPath", "description": "", "type": "string" } } }, "required": ["resolvedPath"], } } AST parsing
15. Large Language Model • Transformer Neural Network • Context & relation aware • Pattern matching & recognition • “Natural” language understanding • Generating / creating content • Summarisation
16. Large Language Model • It cannot make decisions • It cannot understand non-linear content • It cannot draw conclusions
17. Large Language Model • Not deterministic • Can hallucinate • Prompt-Injection
18. { "type": "object", "properties": { "steps": { "type": "array", "items": { "type": "object", "properties": { "title": { "type": "string", "description": "Wizard Step Title." }, "description": { "type": "string", "description": "Wizard Step Short Description. }, "fields": { "type": "array", "items": { "type": "object", "properties": { "type": { "type": "string", "enum": [ "title", "subtitle", "description", "text", "textarea", "select", "radioGroup", "date", "dateRange", const WizardStepSchema = z.object({ steps: z .object({ title: z.string().describe('Wizard Step Title.'), description: z.string().describe('Wizard Step Short Description.'), fields: z .object({ type: z.nativeEnum(FormFieldType).describe('Type of form'), description: z.string() .describe('Description of the Field. What it is and… ‘), label: z.string().describe('Form field label…'), required: z.boolean().describe('Whether the field …’), key: z.string().describe('Key to store the value under.'), }) .array(), }) .array(), }); Zod Schema de fi nition Schema De fi nition & Languages Large Language Model
19. { "HeaderFormField": { "description": "Header shown in a form. The label is… "type": "object", "properties": { "type": { "description": "Type of form field.", "type": "string", "const": "title" }, … "required": { "description": "Whether the field is required… "type": "boolean" }, … }, "required": ["key", "label", "type"] }, /** Base type for all form field types. */ interface BaseFormField { /** Type of form field. */ type: FormFieldType; /** Description of the Field. What it is and what values should be there. */ description?: string; /** Form field label shown to the user. */ label: string; /** Whether the field is required. Defaults to false. */ required?: boolean; /** Key to store the value under. */ key: string; } /** Header shown in a form. The label is used as its text. */ interface HeaderFormField extends BaseFormField { type: FormFieldType.Title; } AST Parsing Large Language Model Schema De fi nition & Languages
20. Create a Wizard to collect information about a car accident. The fi rst step is to collect a clients information as well as the insurance ID. The second step is about collection information on the accident, What type of accident, which other party is involved, which fault it was. The third and fi nal step is about accepting the terms of use and a check if it is ok to process these data. steps: [ { title: 'Client Information', description: 'Please provide your personal and insurance info fields: [ { type: 'text', description: 'Enter your legal first name as it appea label: 'First Name', required: true, key: 'firstName', }, … ], }, ], [ { title: 'Accident Information', description: 'Please provide details about the accident', fields: [ { type: 'select', description: 'Select the type of accident', label: 'Type of Accident', required: true, key: ‘accidentType', options: [ { value: 'collision', label: 'Collision' }, { value: 'theft', label: 'Theft' }, { value: 'vandalism', label: 'Vandalism' }, { value: 'other', label: 'Other' }, ], Large Language Model Schema De fi nition & Languages
21. Create a Wizard to collect information about a car accident. The fi rst step is to collect a clients information as well as the insurance ID. The second step is about collection information on the accident, What type of accident, which other party is involved, which fault it was. The third and fi nal step is about accepting the terms of use and a check if it is ok to process these data. steps: [ { title: 'Client Information', description: 'Please provide your personal and insurance info fields: [ { type: 'text', description: 'Enter your legal first name as it appea label: 'First Name', required: true, key: 'firstName', }, … ], }, ], [ { title: 'Accident Information', description: 'Please provide details about the accident', fields: [ { type: 'select', description: 'Select the type of accident', label: 'Type of Accident', required: true, key: ‘accidentType', options: [ { value: 'collision', label: 'Collision' }, { value: 'theft', label: 'Theft' }, { value: 'vandalism', label: 'Vandalism' }, { value: 'other', label: 'Other' }, ], Large Language Model Schema De fi nition & Languages
22. Results
23. Result Bla bla bla …
24. Result = Bla bla bla …
25. Tools & Functions Process Work Display <…/>
26. Agents
27. Agents
28. Agents /** * Automates a process based on a user query. * @param content - The query from the user. */ async automate(content: string): Promise<void> { // get the current available tools and application state const { tools, state } = this.createToolState(); const message: ChatCompletionMessageParam = { role: 'user', content, }; // make backend request, choices that the model makes // -> messages and tool calls const choices = await this.aiBackend.automate([message], tools, state, trace); if (!choices) { … handle error } // loop and automate, handle llm choice internally for (const choice of choices) { this.handleChoice(choice, [message], tools, state, trace); } }
29. Agents /** * Automates a process based on a user query. * @param content - The query from the user. */ async automate(content: string): Promise<void>
30. /** * Automates a process based on a user query. * @param content - The query from the user. */ async automate(content: string): Promise<void> Agents /** * Handles a choice made by the AI model. * @param choice - The choice made by the model. * @param chat - The current chat history. * @param tools - Available tools for the AI. * @param state - Current application state. * @param trace - Langfuse trace client for logging. */ private async handleChoice( choice: Choice, chat: ChatCompletionMessageParam[], tools: Tool[], state?: any, trace?: LangfuseTraceClient, ): Promise<void> { // handle tool call, can be one or more if (choice.finish_reason === 'tool_calls') { // display information to the user this.createToolMessage(choice).forEach(message => this.message$.next(message)); // make the actual function call and create a message from it const toolResult = this.handleToolCalls(choice); // respond to the API with made calls if (toolResult) { chat.push(choice.message, ...toolResult); (await this.aiBackend.automate(chat, tools, state, trace)).map(choice => this.handleChoice(choice, chat, tools, state, trace), ); } } else { this.message$.next({ role: choice.message.role, message: choice.message.content ?? 'NO MESSAGE', });
31. /** * Automates a process based on a user query. * @param content - The query from the user. */ async automate(content: string): Promise<void> Agents /** * Handles a choice made by the AI model. * @param choice - The choice made by the model. * @param chat - The current chat history. * @param tools - Available tools for the AI. * @param state - Current application state. * @param trace - Langfuse trace client for logging. */ private async handleChoice( choice: Choice, chat: ChatCompletionMessageParam[], tools: Tool[], state?: any, trace?: LangfuseTraceClient, ): Promise<void> { // handle tool call, can be one or more if (choice.finish_reason === 'tool_calls') { // display information to the user this.createToolMessage(choice).forEach(message => this.message$.next(message)); // make the actual function call and create a message from it const toolResult = this.handleToolCalls(choice); // respond to the API with made calls if (toolResult) { chat.push(choice.message, ...toolResult); (await this.aiBackend.automate(chat, tools, state, trace)).map(choice => this.handleChoice(choice, chat, tools, state, trace), ); } } else { this.message$.next({ role: choice.message.role, message: choice.message.content ?? 'NO MESSAGE', }); /** * Handles tool calls made by the AI model. * @param param0 - Object containing the message with tool calls. * @returns An array of tool message params. */ private handleToolCalls({ message }: Pick<Choice, 'message'>): ChatCompletionToolMessageParam[] { // iterate through tool calls in choices. return message.tool_calls!.map(call => { // find the matching instance for this tool const { name, guid } = this.splitFunctionName(call.function.name); const toolInstance = this.findToolInstance(name, guid); let content = ''; // call the instance function with params const instance = toolInstance?.instance as any; if (instance && name in instance) { const toolFunction = instance[name]; content = JSON.stringify( toolFunction.call(instance, ...Object.values(JSON.parse(call.function.arguments))) ?? '', ); // message user about tool call results … } return { content, tool_call_id: call.id, role: 'tool', name: call.function.name, }; }); }
32. Agents /** * Automates a process based on a user query. * @param content - The query from the user. */ async automate(content: string): Promise<void> /** * Handles a choice made by the AI model. * @param choice - The choice made by the model. * @param chat - The current chat history. * @param tools - Available tools for the AI. * @param state - Current application state. * @param trace - Langfuse trace client for logging. */ private async handleChoice( choice: Choice, chat: ChatCompletionMessageParam[], tools: Tool[], state?: any, trace?: LangfuseTraceClient, ): Promise<void> { // handle tool call, can be one or more if (choice.finish_reason === 'tool_calls') { // display information to the user this.createToolMessage(choice).forEach(message => this.message$.next(message)); // make the actual function call and create a message from it const toolResult = this.handleToolCalls(choice); // respond to the API with made calls if (toolResult) { chat.push(choice.message, ...toolResult); (await this.aiBackend.automate(chat, tools, state, trace)).map(choice => this.handleChoice(choice, chat, tools, state, trace), ); } } else { this.message$.next({ role: choice.message.role, message: choice.message.content ?? 'NO MESSAGE', }); /** * Handles tool calls made by the AI model. * @param param0 - Object containing the message with tool calls. * @returns An array of tool message params. */ private handleToolCalls({ message }: Pick<Choice, 'message'>): ChatCompletionToolMessageParam[] { // iterate through tool calls in choices. return message.tool_calls!.map(call => { // find the matching instance for this tool const { name, guid } = this.splitFunctionName(call.function.name); const toolInstance = this.findToolInstance(name, guid); let content = ''; // call the instance function with params const instance = toolInstance?.instance as any; if (instance && name in instance) { const toolFunction = instance[name]; content = JSON.stringify( toolFunction.call(instance, ...Object.values(JSON.parse(call.function.arguments))) ?? '', ); // message user about tool call results … } return { content, tool_call_id: call.id, role: 'tool', name: call.function.name, }; }); }
33. Agents /** * Automates a process based on a user query. * @param content - The query from the user. */ async automate(content: string): Promise<void> /** * Handles a choice made by the AI model. * @param choice - The choice made by the model. * @param chat - The current chat history. * @param tools - Available tools for the AI. * @param state - Current application state. * @param trace - Langfuse trace client for logging. */ private async handleChoice( choice: Choice, chat: ChatCompletionMessageParam[], tools: Tool[], state?: any, trace?: LangfuseTraceClient, ): Promise<void> { // handle tool call, can be one or more if (choice.finish_reason === 'tool_calls') { // display information to the user this.createToolMessage(choice).forEach(message => this.message$.next(message)); // make the actual function call and create a message from it const toolResult = this.handleToolCalls(choice); // respond to the API with made calls if (toolResult) { chat.push(choice.message, ...toolResult); (await this.aiBackend.automate(chat, tools, state, trace)).map(choice => this.handleChoice(choice, chat, tools, state, trace), ); } } else { this.message$.next({ role: choice.message.role, message: choice.message.content ?? 'NO MESSAGE', }); /** * Handles tool calls made by the AI model. * @param param0 - Object containing the message with tool calls. * @returns An array of tool message params. */ private handleToolCalls({ message }: Pick<Choice, 'message'>): ChatCompletionToolMessageParam[] { // iterate through tool calls in choices. return message.tool_calls!.map(call => { // find the matching instance for this tool const { name, guid } = this.splitFunctionName(call.function.name); const toolInstance = this.findToolInstance(name, guid); let content = ''; // call the instance function with params const instance = toolInstance?.instance as any; if (instance && name in instance) { const toolFunction = instance[name]; content = JSON.stringify( toolFunction.call(instance, ...Object.values(JSON.parse(call.function.arguments))) ?? '', ); // message user about tool call results … } return { content, tool_call_id: call.id, role: 'tool', name: call.function.name, }; }); }
34. /** * Automates a process based on a user query. * @param content - The query from the user. */ async automate(content: string): Promise<void> Agents /** * Handles a choice made by the AI model. * @param choice - The choice made by the model. * @param chat - The current chat history. * @param tools - Available tools for the AI. * @param state - Current application state. * @param trace - Langfuse trace client for logging. */ private async handleChoice( choice: Choice, chat: ChatCompletionMessageParam[], tools: Tool[], state?: any, trace?: LangfuseTraceClient, ): Promise<void> { // handle tool call, can be one or more if (choice.finish_reason === 'tool_calls') { // display information to the user this.createToolMessage(choice).forEach(message => this.message$.next(message)); // make the actual function call and create a message from it const toolResult = this.handleToolCalls(choice); // respond to the API with made calls if (toolResult) { chat.push(choice.message, ...toolResult); (await this.aiBackend.automate(chat, tools, state, trace)).map(choice => this.handleChoice(choice, chat, tools, state, trace), ); } } else { this.message$.next({ role: choice.message.role, message: choice.message.content ?? 'NO MESSAGE', });
35. Agents private async handleChoice( … ): Promise<void> { if (choice.finish_reason === 'tool_calls') { // display information to the user this.createToolMessage(choice).forEach(message => this.message$.next(message)); // make the actual function call and create a message from it const toolResult = this.handleToolCalls(choice); // respond to the API with made calls if (toolResult) { chat.push(choice.message, ...toolResult); (await this.aiBackend.automate(chat, tools, state, trace)).map(choice => this.handleChoice(choice, chat, tools, state, trace), ); } } else { … show result message to user } } } /** * Automates a process based on a user query. * @param content - The query from the user. */ async automate(content: string): Promise<void>
36. Agents private async handleChoice( … ): Promise<void> { if (choice.finish_reason === 'tool_calls') { // display information to the user this.createToolMessage(choice).forEach(message => this.message$.next(message)); // make the actual function call and create a message from it const toolResult = this.handleToolCalls(choice); // respond to the API with made calls if (toolResult) { chat.push(choice.message, ...toolResult); (await this.aiBackend.automate(chat, tools, state, trace)).map(choice => this.handleChoice(choice, chat, tools, state, trace), ); } } else { … show result message to user } } } /** * Automates a process based on a user query. * @param content - The query from the user. */ async automate(content: string): Promise<void>
37. Agents /** * Handles a choice made by the AI model. */ private async handleChoice(…): Promise<void> /** * Automates a process based on a user query. * @param content - The query from the user. */ async automate(content: string): Promise<void> /** * Handles tool calls made by the AI model. * @param param0 - Object containing the message with tool calls. * @returns An array of tool message params. */ private handleToolCalls({ message }: Pick<Choice, 'message'>): ChatCompletionToolMessageParam[] {
38. Agents
39. Agents
40. Tools from Codebase
41. Tools from Codebase /** Base type for all form field types. */ interface BaseFormField { /** Type of form field. */ type: FormFieldType; /** Description of the Field. What it is and what values should be there. */ description?: string; /** Form field label shown to the user. */ label: string; /** Whether the field is required. Defaults to false. */ required?: boolean; /** Key to store the value under. */ key: string; } /** Header shown in a form. The label is used as its text. */ interface HeaderFormField extends BaseFormField { type: FormFieldType.Title; } { "HeaderFormField": { "description": "Header shown in a form. The label is… "type": "object", "properties": { "type": { "description": "Type of form field.", "type": "string", "const": "title" }, … "required": { "description": "Whether the field is required… "type": "boolean" }, … }, "required": ["key", "label", "type"] }, AST Parsing
42. Tools from Codebase /** * Retrieve a list of entries for a specific type. */ @AiTool public getCollection(collectionType: CollectionType): object[] { switch (collectionType.type) { case 'contributions': return this.entryStore.contributions(); case 'speakers': return this.entryStore.speakers(); case 'collections': return []; case 'conferences': return this.entryStore.conferences(); default: return []; } } { "type": "function", "classDocumentation": "", "function": { "title": "getCollection", "description": "Retrieve a list of entries for …”, "parameters": { "title": "collectionType", "description": "", "type": "object", "properties": { "type": { "description": "Type of different data sets…” "enum": [ “collections", "conferences", "contributions", "speakers" ], "type": "string" } }, "required": [ "type" AST Parsing
43. @Directive({ selector: '[aiClick]', standalone: true, }) export class AiClickDirective { private element = inject(ElementRef); metaInfo = input.required<string>({ alias: 'aiClick' }); guid = input<string>(crypto.randomUUID()); constructor() { registerInstance(() => ({ guid: this.guid(), instance: this, state: this.metaInfo(), className: 'AiClickDirective', metaInfo: this.metaInfo(), })); } @AiTool click(): void { this.element.nativeElement.click(); } } Tools from Codebase
44. export class AiClickDirective { … constructor() { registerInstance(() => ({ guid: this.guid(), instance: this, state: this.metaInfo(), className: 'AiClickDirective', metaInfo: this.metaInfo(), })); } @AiTool click(): void { this.element.nativeElement.click(); } } Tools from Codebase
45. export class AiClickDirective { … constructor() { registerInstance(() => …); } … } Tools from Codebase
46. export class AiClickDirective { … constructor() { registerInstance(() => …); } … } export function registerInstance(info: () => RegisterInfo) { const aiService = inject(AiService); const destroyRef = inject(DestroyRef); aiService.registerInstance(info); destroyRef.onDestroy(() => aiService.removeInstance(info)); } Tools from Codebase
47. export class AiClickDirective { … constructor() { registerInstance(() => …); } … } export function registerInstance(info: () => RegisterInfo) { const aiService = inject(AiService); const destroyRef = inject(DestroyRef); aiService.registerInstance(info); destroyRef.onDestroy(() => aiService.removeInstance(info)); } private handleToolCalls({ message }: Pick<Choice, 'message'>): ChatCompletionToolMessageParam[] { // iterate through tool calls in choices. return message.tool_calls!.map(call => { // find the matching instance for this tool const { name, guid } = this.splitFunctionName(call.function.name); const toolInstance = this.findToolInstance(name, guid); let content = ''; // call the instance function with params const instance = toolInstance?.instance as any; if (instance && name in instance) { const toolFunction = instance[name]; content = JSON.stringify( toolFunction.call(instance, ...Object.values(JSON.parse(call.function.arguments))) ?? '', ); } … }); } Tools from Codebase
48. export class AiClickDirective { … constructor() { registerInstance(() => …); } … } export function registerInstance(info: () => RegisterInfo) { const aiService = inject(AiService); const destroyRef = inject(DestroyRef); aiService.registerInstance(info); destroyRef.onDestroy(() => aiService.removeInstance(info)); } /** * Finds a tool instance based on the function name and optional GUID. * … */ private findToolInstance(name: string, guid?: string): RegisterInfo | undefined { const className = this.tools.find(tool => tool.function.name === name)?.className; return this.registeredInstances.find(info => { const { className: infoClassName, guid: infoGuid } = info(); return className === infoClassName && guid === infoGuid; })?.(); } Tools from Codebase
49. Overview
50. • “Talking to fast learning & smart junior" • No default assumptions • Documentation is key • Examples are great Development Mindset
51. • Avoid auto-commit • Enable feedback • Highlight probability • Clear entry point into AI work fl ows • Progress / generation indicators • Suggest common actions UI & UX
52. • Client-side for PoC • Easy transition to server-side • Write docs for dummies • Unambiguous tools • Unambiguous context • An Agent is just a loop LLMs do not actually call functions - you do! Lessons Learned
53. It's a wrap! • @MaxOSchulte • max.marschall@thinktecture.com • www.thinktecture.com/max-marschall