Max Marschall @MaxOSchulte Consultant / Developer GenAI and LLMs in exiting web-apps automate everything?

Arti fi cial Intelligence Machine Learning Data Science NLP A Deep Learning Generative AI Large Language Models Image / Video Generation

Bla bla bla … Intention

Bla bla bla … Generation of… Intention Summarisation of… Automation of… Simpli fi cation of…

Bla bla bla … “Augmenting an App with an assistant, that helps the user based on displayed context. Allowing an assistant to act for me in that context.” Intention

Intention Bla bla bla … “Augmenting an App with an assistant, that helps the user based on displayed context. Allowing an assistant to act for me in that context.” Problem: Integration GenAI Metainformation Tool calls Leaking secrets System integration Solutions AST parsing TS Decorators Client-side "agents" Barebones Angular OpenAI

Intention Bla bla bla … “Augmenting an App with an assistant, that helps the user based on displayed context. Allowing an assistant to act for me in that context.”

Tools & Functions Process Think Display Bla bla bla …

User Prompt • Voice input • Text input • Image input • Unstructured • Universal interface • Natural language as fi rst-class input

• Prompts • RAG • Sanitising • Guards • Vector-Database • Tool selection • Additional data (Processing)

• System prompt • Base information • Rules • Intention • Output format Processing You are an friendly helpfull assistent proficient in creating and managing forms. You will receive information on the state of an app. If you feel there a … System Prompt https://docs.anthropic.com/en/release-notes/system-prompts#oct-22nd-2024 You are a data analyst API capable of sentiment analysis that responds in JSON. The JSON schema should include { "sentiment_analysis": { "sentiment": "string (positive, negative, neutral)", "confidence_score": "number (0-1)" # Include additional fields as required } https://console.groq.com/docs/text-chat Prompts

• State • Additional information • Generation examples Context /** Additional Information: basic list of static pages shown in the side navigation. */ [ { title: 'Contributions', icon: 'question_answer', link: ['/contributions'] }, { title: 'Conferences', icon: 'podium', link: ['/conferences'] }, { title: 'Speakers', icon: 'person', link: ['/speakers'] }, { title: 'Collections', icon: 'category', link: ['/collections'] }, ];

Tools & Functions export interface NavigationPath { /** * Absolute path to navigate * e.g. {path: '/collections/:id'} * ":id" is replaced by an actual entity id. */ resolvedPath: string; } /** * Accepts a resolved sitemap entry and navigates to it. */ public navigate(navigation: NavigationPath): Promise { return this.router.navigateByUrl('/' + navigation.resolvedPath); }

export interface NavigationPath { /** * Absolute path to navigate * e.g. {path: '/collections/:id'} * ":id" is replaced by an actual entity id. */ resolvedPath: string; } /** * Accepts a resolved sitemap entry and navigates to it. */ @AiTool public navigate(navigation: NavigationPath): Promise { return this.router.navigateByUrl('/' + navigation.resolvedPath); } Tools & Functions { "type": "function", "function": { "name": "navigate", "description": "Accepts a resolved sitemap entry …, "parameters": { "type": "object", "properties": { "resolvedPath": { "title": "resolvedPath", "description": "", "type": "string" } } }, "required": ["resolvedPath"], } } AST parsing

Large Language Model • Transformer Neural Network • Context & relation aware • Pattern matching & recognition • “Natural” language understanding • Generating / creating content • Summarisation

Large Language Model • It cannot make decisions • It cannot understand non-linear content • It cannot draw conclusions

Large Language Model • Not deterministic • Can hallucinate • Prompt-Injection

{ "type": "object", "properties": { "steps": { "type": "array", "items": { "type": "object", "properties": { "title": { "type": "string", "description": "Wizard Step Title." }, "description": { "type": "string", "description": "Wizard Step Short Description. }, "fields": { "type": "array", "items": { "type": "object", "properties": { "type": { "type": "string", "enum": [ "title", "subtitle", "description", "text", "textarea", "select", "radioGroup", "date", "dateRange", const WizardStepSchema = z.object({ steps: z .object({ title: z.string().describe('Wizard Step Title.'), description: z.string().describe('Wizard Step Short Description.'), fields: z .object({ type: z.nativeEnum(FormFieldType).describe('Type of form'), description: z.string() .describe('Description of the Field. What it is and… ‘), label: z.string().describe('Form field label…'), required: z.boolean().describe('Whether the field …’), key: z.string().describe('Key to store the value under.'), }) .array(), }) .array(), }); Zod Schema de fi nition Schema De fi nition & Languages Large Language Model

{ "HeaderFormField": { "description": "Header shown in a form. The label is… "type": "object", "properties": { "type": { "description": "Type of form field.", "type": "string", "const": "title" }, … "required": { "description": "Whether the field is required… "type": "boolean" }, … }, "required": ["key", "label", "type"] }, /** Base type for all form field types. */ interface BaseFormField { /** Type of form field. */ type: FormFieldType; /** Description of the Field. What it is and what values should be there. */ description?: string; /** Form field label shown to the user. */ label: string; /** Whether the field is required. Defaults to false. */ required?: boolean; /** Key to store the value under. */ key: string; } /** Header shown in a form. The label is used as its text. */ interface HeaderFormField extends BaseFormField { type: FormFieldType.Title; } AST Parsing Large Language Model Schema De fi nition & Languages

Create a Wizard to collect information about a car accident. The fi rst step is to collect a clients information as well as the insurance ID. The second step is about collection information on the accident, What type of accident, which other party is involved, which fault it was. The third and fi nal step is about accepting the terms of use and a check if it is ok to process these data. steps: [ { title: 'Client Information', description: 'Please provide your personal and insurance info fields: [ { type: 'text', description: 'Enter your legal first name as it appea label: 'First Name', required: true, key: 'firstName', }, … ], }, ], [ { title: 'Accident Information', description: 'Please provide details about the accident', fields: [ { type: 'select', description: 'Select the type of accident', label: 'Type of Accident', required: true, key: ‘accidentType', options: [ { value: 'collision', label: 'Collision' }, { value: 'theft', label: 'Theft' }, { value: 'vandalism', label: 'Vandalism' }, { value: 'other', label: 'Other' }, ], Large Language Model Schema De fi nition & Languages

Create a Wizard to collect information about a car accident. The fi rst step is to collect a clients information as well as the insurance ID. The second step is about collection information on the accident, What type of accident, which other party is involved, which fault it was. The third and fi nal step is about accepting the terms of use and a check if it is ok to process these data. steps: [ { title: 'Client Information', description: 'Please provide your personal and insurance info fields: [ { type: 'text', description: 'Enter your legal first name as it appea label: 'First Name', required: true, key: 'firstName', }, … ], }, ], [ { title: 'Accident Information', description: 'Please provide details about the accident', fields: [ { type: 'select', description: 'Select the type of accident', label: 'Type of Accident', required: true, key: ‘accidentType', options: [ { value: 'collision', label: 'Collision' }, { value: 'theft', label: 'Theft' }, { value: 'vandalism', label: 'Vandalism' }, { value: 'other', label: 'Other' }, ], Large Language Model Schema De fi nition & Languages

Results

Result Bla bla bla …

Result = Bla bla bla …

Tools & Functions Process Work Display <…/>

Agents

Agents

Agents /** * Automates a process based on a user query. * @param content - The query from the user. */ async automate(content: string): Promise { // get the current available tools and application state const { tools, state } = this.createToolState(); const message: ChatCompletionMessageParam = { role: 'user', content, }; // make backend request, choices that the model makes // -> messages and tool calls const choices = await this.aiBackend.automate([message], tools, state, trace); if (!choices) { … handle error } // loop and automate, handle llm choice internally for (const choice of choices) { this.handleChoice(choice, [message], tools, state, trace); } }

Agents /** * Automates a process based on a user query. * @param content - The query from the user. */ async automate(content: string): Promise

/** * Automates a process based on a user query. * @param content - The query from the user. */ async automate(content: string): Promise Agents /** * Handles a choice made by the AI model. * @param choice - The choice made by the model. * @param chat - The current chat history. * @param tools - Available tools for the AI. * @param state - Current application state. * @param trace - Langfuse trace client for logging. */ private async handleChoice( choice: Choice, chat: ChatCompletionMessageParam[], tools: Tool[], state?: any, trace?: LangfuseTraceClient, ): Promise { // handle tool call, can be one or more if (choice.finish_reason === 'tool_calls') { // display information to the user this.createToolMessage(choice).forEach(message => this.message$.next(message)); // make the actual function call and create a message from it const toolResult = this.handleToolCalls(choice); // respond to the API with made calls if (toolResult) { chat.push(choice.message, ...toolResult); (await this.aiBackend.automate(chat, tools, state, trace)).map(choice => this.handleChoice(choice, chat, tools, state, trace), ); } } else { this.message$.next({ role: choice.message.role, message: choice.message.content ?? 'NO MESSAGE', });

/** * Automates a process based on a user query. * @param content - The query from the user. */ async automate(content: string): Promise Agents /** * Handles a choice made by the AI model. * @param choice - The choice made by the model. * @param chat - The current chat history. * @param tools - Available tools for the AI. * @param state - Current application state. * @param trace - Langfuse trace client for logging. */ private async handleChoice( choice: Choice, chat: ChatCompletionMessageParam[], tools: Tool[], state?: any, trace?: LangfuseTraceClient, ): Promise { // handle tool call, can be one or more if (choice.finish_reason === 'tool_calls') { // display information to the user this.createToolMessage(choice).forEach(message => this.message$.next(message)); // make the actual function call and create a message from it const toolResult = this.handleToolCalls(choice); // respond to the API with made calls if (toolResult) { chat.push(choice.message, ...toolResult); (await this.aiBackend.automate(chat, tools, state, trace)).map(choice => this.handleChoice(choice, chat, tools, state, trace), ); } } else { this.message$.next({ role: choice.message.role, message: choice.message.content ?? 'NO MESSAGE', }); /** * Handles tool calls made by the AI model. * @param param0 - Object containing the message with tool calls. * @returns An array of tool message params. */ private handleToolCalls({ message }: Pick): ChatCompletionToolMessageParam[] { // iterate through tool calls in choices. return message.tool_calls!.map(call => { // find the matching instance for this tool const { name, guid } = this.splitFunctionName(call.function.name); const toolInstance = this.findToolInstance(name, guid); let content = ''; // call the instance function with params const instance = toolInstance?.instance as any; if (instance && name in instance) { const toolFunction = instance[name]; content = JSON.stringify( toolFunction.call(instance, ...Object.values(JSON.parse(call.function.arguments))) ?? '', ); // message user about tool call results … } return { content, tool_call_id: call.id, role: 'tool', name: call.function.name, }; }); }

Agents /** * Automates a process based on a user query. * @param content - The query from the user. */ async automate(content: string): Promise /** * Handles a choice made by the AI model. * @param choice - The choice made by the model. * @param chat - The current chat history. * @param tools - Available tools for the AI. * @param state - Current application state. * @param trace - Langfuse trace client for logging. */ private async handleChoice( choice: Choice, chat: ChatCompletionMessageParam[], tools: Tool[], state?: any, trace?: LangfuseTraceClient, ): Promise { // handle tool call, can be one or more if (choice.finish_reason === 'tool_calls') { // display information to the user this.createToolMessage(choice).forEach(message => this.message$.next(message)); // make the actual function call and create a message from it const toolResult = this.handleToolCalls(choice); // respond to the API with made calls if (toolResult) { chat.push(choice.message, ...toolResult); (await this.aiBackend.automate(chat, tools, state, trace)).map(choice => this.handleChoice(choice, chat, tools, state, trace), ); } } else { this.message$.next({ role: choice.message.role, message: choice.message.content ?? 'NO MESSAGE', }); /** * Handles tool calls made by the AI model. * @param param0 - Object containing the message with tool calls. * @returns An array of tool message params. */ private handleToolCalls({ message }: Pick): ChatCompletionToolMessageParam[] { // iterate through tool calls in choices. return message.tool_calls!.map(call => { // find the matching instance for this tool const { name, guid } = this.splitFunctionName(call.function.name); const toolInstance = this.findToolInstance(name, guid); let content = ''; // call the instance function with params const instance = toolInstance?.instance as any; if (instance && name in instance) { const toolFunction = instance[name]; content = JSON.stringify( toolFunction.call(instance, ...Object.values(JSON.parse(call.function.arguments))) ?? '', ); // message user about tool call results … } return { content, tool_call_id: call.id, role: 'tool', name: call.function.name, }; }); }

Agents /** * Automates a process based on a user query. * @param content - The query from the user. */ async automate(content: string): Promise /** * Handles a choice made by the AI model. * @param choice - The choice made by the model. * @param chat - The current chat history. * @param tools - Available tools for the AI. * @param state - Current application state. * @param trace - Langfuse trace client for logging. */ private async handleChoice( choice: Choice, chat: ChatCompletionMessageParam[], tools: Tool[], state?: any, trace?: LangfuseTraceClient, ): Promise { // handle tool call, can be one or more if (choice.finish_reason === 'tool_calls') { // display information to the user this.createToolMessage(choice).forEach(message => this.message$.next(message)); // make the actual function call and create a message from it const toolResult = this.handleToolCalls(choice); // respond to the API with made calls if (toolResult) { chat.push(choice.message, ...toolResult); (await this.aiBackend.automate(chat, tools, state, trace)).map(choice => this.handleChoice(choice, chat, tools, state, trace), ); } } else { this.message$.next({ role: choice.message.role, message: choice.message.content ?? 'NO MESSAGE', }); /** * Handles tool calls made by the AI model. * @param param0 - Object containing the message with tool calls. * @returns An array of tool message params. */ private handleToolCalls({ message }: Pick): ChatCompletionToolMessageParam[] { // iterate through tool calls in choices. return message.tool_calls!.map(call => { // find the matching instance for this tool const { name, guid } = this.splitFunctionName(call.function.name); const toolInstance = this.findToolInstance(name, guid); let content = ''; // call the instance function with params const instance = toolInstance?.instance as any; if (instance && name in instance) { const toolFunction = instance[name]; content = JSON.stringify( toolFunction.call(instance, ...Object.values(JSON.parse(call.function.arguments))) ?? '', ); // message user about tool call results … } return { content, tool_call_id: call.id, role: 'tool', name: call.function.name, }; }); }

/** * Automates a process based on a user query. * @param content - The query from the user. */ async automate(content: string): Promise Agents /** * Handles a choice made by the AI model. * @param choice - The choice made by the model. * @param chat - The current chat history. * @param tools - Available tools for the AI. * @param state - Current application state. * @param trace - Langfuse trace client for logging. */ private async handleChoice( choice: Choice, chat: ChatCompletionMessageParam[], tools: Tool[], state?: any, trace?: LangfuseTraceClient, ): Promise { // handle tool call, can be one or more if (choice.finish_reason === 'tool_calls') { // display information to the user this.createToolMessage(choice).forEach(message => this.message$.next(message)); // make the actual function call and create a message from it const toolResult = this.handleToolCalls(choice); // respond to the API with made calls if (toolResult) { chat.push(choice.message, ...toolResult); (await this.aiBackend.automate(chat, tools, state, trace)).map(choice => this.handleChoice(choice, chat, tools, state, trace), ); } } else { this.message$.next({ role: choice.message.role, message: choice.message.content ?? 'NO MESSAGE', });

Agents private async handleChoice( … ): Promise { if (choice.finish_reason === 'tool_calls') { // display information to the user this.createToolMessage(choice).forEach(message => this.message$.next(message)); // make the actual function call and create a message from it const toolResult = this.handleToolCalls(choice); // respond to the API with made calls if (toolResult) { chat.push(choice.message, ...toolResult); (await this.aiBackend.automate(chat, tools, state, trace)).map(choice => this.handleChoice(choice, chat, tools, state, trace), ); } } else { … show result message to user } } } /** * Automates a process based on a user query. * @param content - The query from the user. */ async automate(content: string): Promise

Agents private async handleChoice( … ): Promise { if (choice.finish_reason === 'tool_calls') { // display information to the user this.createToolMessage(choice).forEach(message => this.message$.next(message)); // make the actual function call and create a message from it const toolResult = this.handleToolCalls(choice); // respond to the API with made calls if (toolResult) { chat.push(choice.message, ...toolResult); (await this.aiBackend.automate(chat, tools, state, trace)).map(choice => this.handleChoice(choice, chat, tools, state, trace), ); } } else { … show result message to user } } } /** * Automates a process based on a user query. * @param content - The query from the user. */ async automate(content: string): Promise

Agents /** * Handles a choice made by the AI model. */ private async handleChoice(…): Promise /** * Automates a process based on a user query. * @param content - The query from the user. */ async automate(content: string): Promise /** * Handles tool calls made by the AI model. * @param param0 - Object containing the message with tool calls. * @returns An array of tool message params. */ private handleToolCalls({ message }: Pick): ChatCompletionToolMessageParam[] {

Agents

Agents

Tools from Codebase

Tools from Codebase /** Base type for all form field types. */ interface BaseFormField { /** Type of form field. */ type: FormFieldType; /** Description of the Field. What it is and what values should be there. */ description?: string; /** Form field label shown to the user. */ label: string; /** Whether the field is required. Defaults to false. */ required?: boolean; /** Key to store the value under. */ key: string; } /** Header shown in a form. The label is used as its text. */ interface HeaderFormField extends BaseFormField { type: FormFieldType.Title; } { "HeaderFormField": { "description": "Header shown in a form. The label is… "type": "object", "properties": { "type": { "description": "Type of form field.", "type": "string", "const": "title" }, … "required": { "description": "Whether the field is required… "type": "boolean" }, … }, "required": ["key", "label", "type"] }, AST Parsing

Tools from Codebase /** * Retrieve a list of entries for a specific type. */ @AiTool public getCollection(collectionType: CollectionType): object[] { switch (collectionType.type) { case 'contributions': return this.entryStore.contributions(); case 'speakers': return this.entryStore.speakers(); case 'collections': return []; case 'conferences': return this.entryStore.conferences(); default: return []; } } { "type": "function", "classDocumentation": "", "function": { "title": "getCollection", "description": "Retrieve a list of entries for …”, "parameters": { "title": "collectionType", "description": "", "type": "object", "properties": { "type": { "description": "Type of different data sets…” "enum": [ “collections", "conferences", "contributions", "speakers" ], "type": "string" } }, "required": [ "type" AST Parsing

@Directive({ selector: '[aiClick]', standalone: true, }) export class AiClickDirective { private element = inject(ElementRef); metaInfo = input.required({ alias: 'aiClick' }); guid = input(crypto.randomUUID()); constructor() { registerInstance(() => ({ guid: this.guid(), instance: this, state: this.metaInfo(), className: 'AiClickDirective', metaInfo: this.metaInfo(), })); } @AiTool click(): void { this.element.nativeElement.click(); } } Tools from Codebase

export class AiClickDirective { … constructor() { registerInstance(() => ({ guid: this.guid(), instance: this, state: this.metaInfo(), className: 'AiClickDirective', metaInfo: this.metaInfo(), })); } @AiTool click(): void { this.element.nativeElement.click(); } } Tools from Codebase

export class AiClickDirective { … constructor() { registerInstance(() => …); } … } Tools from Codebase

export class AiClickDirective { … constructor() { registerInstance(() => …); } … } export function registerInstance(info: () => RegisterInfo) { const aiService = inject(AiService); const destroyRef = inject(DestroyRef); aiService.registerInstance(info); destroyRef.onDestroy(() => aiService.removeInstance(info)); } Tools from Codebase

export class AiClickDirective { … constructor() { registerInstance(() => …); } … } export function registerInstance(info: () => RegisterInfo) { const aiService = inject(AiService); const destroyRef = inject(DestroyRef); aiService.registerInstance(info); destroyRef.onDestroy(() => aiService.removeInstance(info)); } private handleToolCalls({ message }: Pick): ChatCompletionToolMessageParam[] { // iterate through tool calls in choices. return message.tool_calls!.map(call => { // find the matching instance for this tool const { name, guid } = this.splitFunctionName(call.function.name); const toolInstance = this.findToolInstance(name, guid); let content = ''; // call the instance function with params const instance = toolInstance?.instance as any; if (instance && name in instance) { const toolFunction = instance[name]; content = JSON.stringify( toolFunction.call(instance, ...Object.values(JSON.parse(call.function.arguments))) ?? '', ); } … }); } Tools from Codebase

export class AiClickDirective { … constructor() { registerInstance(() => …); } … } export function registerInstance(info: () => RegisterInfo) { const aiService = inject(AiService); const destroyRef = inject(DestroyRef); aiService.registerInstance(info); destroyRef.onDestroy(() => aiService.removeInstance(info)); } /** * Finds a tool instance based on the function name and optional GUID. * … */ private findToolInstance(name: string, guid?: string): RegisterInfo | undefined { const className = this.tools.find(tool => tool.function.name === name)?.className; return this.registeredInstances.find(info => { const { className: infoClassName, guid: infoGuid } = info(); return className === infoClassName && guid === infoGuid; })?.(); } Tools from Codebase

Overview

• “Talking to fast learning & smart junior" • No default assumptions • Documentation is key • Examples are great Development Mindset

• Avoid auto-commit • Enable feedback • Highlight probability • Clear entry point into AI work fl ows • Progress / generation indicators • Suggest common actions UI & UX

• Client-side for PoC • Easy transition to server-side • Write docs for dummies • Unambiguous tools • Unambiguous context • An Agent is just a loop LLMs do not actually call functions - you do! Lessons Learned

It's a wrap! • @MaxOSchulte • [email protected] • www.thinktecture.com/max-marschall