Next gen(eration) forms

Slide 1

Slide 1 text

Next gen(eration) forms Max Marschall @MaxOSchulte Consultant / Developer

Slide 2

Slide 2 text

Bla bla bla … Intention

Slide 3

Slide 3 text

Bla bla bla … Generation of… Intention Summarisation of… Automation of… Simpli fi cation of…

Slide 4

Slide 4 text

Bla bla bla … “Augmenting an App with an assistant, that helps the user based on displayed context. Allowing an assistant to act for me in that context.” Intention

Slide 5

Slide 5 text

Intention Bla bla bla … “Augmenting an App with an assistant, that helps the user based on displayed context. Allowing an assistant to act for me in that context.” Problem: Integration GenAI Metainformation Tool calls Leaking secrets System integration Solutions AST parsing TS Decorators Client-side "agents" Barebones Angular OpenAI

Slide 6

Slide 6 text

Intention Bla bla bla … “Augmenting an App with an assistant, that helps the user based on displayed context. Allowing an assistant to act for me in that context.”

Slide 7

Slide 7 text

Tools & Functions Process Think Display Bla bla bla …

Slide 8

Slide 8 text

User Prompt • Voice input • Text input • Image input • Unstructured • Universal interface • Natural language as fi rst-class input

Slide 9

Slide 9 text

• Prompts • RAG • Sanitising • Guards • Vector-Database • Tool selection • Additional data (Processing)

Slide 10

Slide 10 text

• System prompt • Base information • Rules • Intention • Output format Processing You are an friendly helpfull assistent proficient in creating and managing forms. You will receive information on the state of an app. If you feel there a … System Prompt https://docs.anthropic.com/en/release-notes/system-prompts#oct-22nd-2024 You are a data analyst API capable of sentiment analysis that responds in JSON. The JSON schema should include { "sentiment_analysis": { "sentiment": "string (positive, negative, neutral)", "confidence_score": "number (0-1)" # Include additional fields as required } https://console.groq.com/docs/text-chat Prompts

Slide 11

Slide 11 text

• State • Additional information • Generation examples Context /** Additional Information: basic list of static pages shown in the side navigation. */ [ { title: 'Contributions', icon: 'question_answer', link: ['/contributions'] }, { title: 'Conferences', icon: 'podium', link: ['/conferences'] }, { title: 'Speakers', icon: 'person', link: ['/speakers'] }, { title: 'Collections', icon: 'category', link: ['/collections'] }, ];

Slide 12

Slide 12 text

Tools & Functions /** * Accepts a resolved sitemap entry and navigates to it. * * e.g. path: /collections/:id * ":id" is replaced by an actual entity id. */ public navigate(resolvedPath: string): void { this.router.navigateByUrl('/' + resolvedPath); } { "type": "function", "function": { "name": "navigate", "description": "Accepts a resolved sitemap entry …, "parameters": { "type": "object", "properties": { "resolvedPath": { "title": "resolvedPath", "description": "", "type": "string" } } }, "required": ["resolvedPath"], } }

Slide 13

Slide 13 text

Large Language Model • Transformer Neural Network • Context & relation aware • Pattern matching & recognition • “Natural” language understanding • Generating / creating content • Summarisation

Slide 14

Slide 14 text

Large Language Model • It cannot make decisions • It cannot understand non-linear content • It cannot draw conclusions

Slide 15

Slide 15 text

Large Language Model • Not deterministic • Can hallucinate • Prompt-Injection

Slide 16

Slide 16 text

Result Bla bla bla …

Slide 17

Slide 17 text

Result = Bla bla bla …

Slide 18

Slide 18 text

Tools & Functions Process Work Display

Slide 19

Slide 19 text

• Pattern matching • Context extraction • Language understanding • Transformation “language to language” Creating Components Used LLM capabilities

Slide 20

Slide 20 text

Creating Components Create a Wizard to collect information about a car accident. The fi rst step is to collect a clients information as well as the insurance ID. The second step is about collection information on the accident, What type of accident, which other party is involved, which fault it was. The third and fi nal step is about accepting the terms of use and a check if it is ok to process these data. steps: [ { title: 'Client Information', description: 'Please provide your personal and insurance info fields: [ { type: 'text', description: 'Enter your legal first name as it appea label: 'First Name', required: true, key: 'firstName', }, … ], }, ], [ { title: 'Accident Information', description: 'Please provide details about the accident', fields: [ { type: 'select', description: 'Select the type of accident', label: 'Type of Accident', required: true, key: ‘accidentType', options: [ { value: 'collision', label: 'Collision' }, { value: 'theft', label: 'Theft' }, { value: 'vandalism', label: 'Vandalism' }, { value: 'other', label: 'Other' }, ],

Slide 21

Slide 21 text

Slide 22

Slide 22 text

Slide 23

Slide 23 text

• “Contract” • Output Language • Interpreted in Code • Guiding Rails • Output Validation Creating Components Schema De fi nition

Slide 24

Slide 24 text

Creating Components { "type": "object", "properties": { "steps": { "type": "array", "items": { "type": "object", "properties": { "title": { "type": "string", "description": "Wizard Step Title." }, "description": { "type": "string", "description": "Wizard Step Short Description. }, "fields": { "type": "array", "items": { "type": "object", "properties": { "type": { "type": "string", "enum": [ "title", "subtitle", "description", "text", "textarea", "select", "radioGroup", "date", "dateRange", const WizardStepSchema = z.object({ steps: z .object({ title: z.string().describe('Wizard Step Title.'), description: z.string().describe('Wizard Step Short Description.'), fields: z .object({ type: z.nativeEnum(FormFieldType).describe('Type of form'), description: z.string() .describe('Description of the Field. What it is and… ‘), label: z.string().describe('Form field label…'), required: z.boolean().describe('Whether the field …’), key: z.string().describe('Key to store the value under.'), }) .array(), }) .array(), }); Zod Schema de fi nition Schema De fi nition

Slide 25

Slide 25 text

Creating Components { "HeaderFormField": { "description": "Header shown in a form. The label is… "type": "object", "properties": { "type": { "description": "Type of form field.", "type": "string", "const": "title" }, … "required": { "description": "Whether the field is required… "type": "boolean" }, … }, "required": ["key", "label", "type"] }, /** Base type for all form field types. */ interface BaseFormField { /** Type of form field. */ type: FormFieldType; /** Description of the Field. What it is and what values should be there. */ description?: string; /** Form field label shown to the user. */ label: string; /** Whether the field is required. Defaults to false. */ required?: boolean; /** Key to store the value under. */ key: string; } /** Header shown in a form. The label is used as its text. */ interface HeaderFormField extends BaseFormField { type: FormFieldType.Title; } AST Parsing Schema De fi nition

Slide 26

Slide 26 text

Schemas Process Work Display

Slide 27

Slide 27 text

Process Work Schemas • Output Schema in Prompt • Structured Outputs • Function / Tool calling

Slide 28

Slide 28 text

Structured Output (OpenAI) Process Work Structured Outputs is a feature that ensures the model will always generate responses that adhere to your supplied JSON Schema, so you don't need to worry about the model omitting a required key, or hallucinating an invalid enum value. https://platform.openai.com/docs/guides/structured-outputs?context=without_parse

Slide 29

Slide 29 text

Process Work const completion = await this.client.chat.completions.create({ tool_choice: 'auto', model: 'gpt-4o-2024-08-06', temperature: 0, messages, tools, }); const completion = await this.client.chat.completions.create({ model: ‘gpt-4o-2024-08-06', messages, structured_output, }); "fields": [ { "type": "title", "label": "Event Information", "description": "Form to collect information about an upcoming event.", "key": "eventInformationTitle" }, … ] tool_calls": [ { "id": "call_wqxp1eYfhDTZrkuXfPqFP7uc", "type": "function", "function": { "name": "createFormFields", "arguments": { "fields": [ { "type": "title", "label": "Event Information", "description": "Form to collect information about an upcoming eve "key": "eventInformationTitle" … Structured Output (OpenAI)

Slide 30

Slide 30 text

Structured Output (OpenAI) Process Work • Providing only an output schema, no function • Some limitation • “Use for presenting to the user” • Reliable type-safety • Explicit refusals

Slide 31

Slide 31 text

Process Work • String: minLength, maxLength, pattern, format • Numbers: minimum, maximum, multipleOf • Objects: patternProperties, unevaluatedProperties, propertyNames, minProperties, maxProperties • Arrays: unevaluatedItems, contains, minContains, maxContains, minItems, maxItems, uniqueItems Limitations on strict Structured Output (OpenAI)

Slide 32

Slide 32 text

Process Work Tools & Functions { "HeaderFormField": { "description": "Header shown in a form. The label is… "type": "object", "properties": { "type": { "description": "Type of form field.", "type": "string", "const": "title" }, … "required": { "description": "Whether the field is required… "type": "boolean" }, … }, "required": ["key", "label", "type"] }, /** Base type for all form field types. */ interface BaseFormField { /** Type of form field. */ type: FormFieldType; /** Description of the Field. What it is and what values should be there. */ description?: string; /** Form field label shown to the user. */ label: string; /** Whether the field is required. Defaults to false. */ required?: boolean; /** Key to store the value under. */ key: string; } /** Header shown in a form. The label is used as its text. */ interface HeaderFormField extends BaseFormField { type: FormFieldType.Title; }

Slide 33

Slide 33 text

Tools & Functions Process Work /** * Creates form fields for a provided list of formField definition. */ createFormFields(fields: FormField[]): FormField[] { … } /** Union of all supported form field types. */ export type FormField = | HeaderFormField | SubheaderFormFiel | …

Slide 34

Slide 34 text

/** * Creates form fields for a provided list of formField definition. */ @AiTool createFormFields(fields: FormField[]): FormField[] { … } /** Union of all supported form field types. */ export type FormField = | HeaderFormField | SubheaderFormFiel | … Tools & Functions Process Work { "type": "function", "className": "FormBuilderService", "classDocumentation": "", "function": { "name": "createFormFields", "title": "createFormFields", "type": "function", "description": "Creates form fields for a provided list… "parameters": { "type": "object", "properties": { "fields": { "title": "fields", "description": "Union of all supported form… "anyOf": [ { "$ref": "#/definitions/HeaderFormField" }, …

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Process Work Tools & Functions /** * Creates form fields for a provided list of formField definition. */ @AiTool createFormFields(fields: FormField[]): FormField[] { … } /** Union of all supported form field types. */ export type FormField = | HeaderFormField | SubheaderFormFiel | … { "type": "function", "className": "FormBuilderService", "classDocumentation": "", "function": { "name": "createFormFields", "title": "createFormFields", "type": "function", "description": "Creates form fields for a provided list… "parameters": { "type": "object", "properties": { "fields": { "title": "fields", "description": "Union of all supported form… "anyOf": [ { "$ref": "#/definitions/HeaderFormField" }, … tool_calls": [ { "id": "call_wqxp1eYfhDTZrkuXfPqFP7uc", "type": "function", "function": { "name": "createFormFields", "arguments": { "fields": [ { "type": "title", "label": "Event Information", "description": "Form to collect information about an upcoming event.", "key": "eventInformationTitle" }, { "type": "text", "label": "Event Name", "description": "Enter the name of the event.", "key": "eventName" }, { "type": "select", "label": "Speaker", "description": "Select an existing speaker for the event.", "options": { "labelKey": "fullName", "collectionId": "speakers" }, "key": "speaker" } ] } …

Slide 37

Slide 37 text

• “Talking to fast learning & smart junior" • No default assumptions • Documentation is key • Examples are great • Limited possibilities deliver better results Development Mindset

Slide 38

Slide 38 text

• Avoid auto-commit • Enable feedback • Highlight probability • Clear entry point into AI work fl ows • Progress / generation indicators • Suggest common actions UI & UX

Slide 39

Slide 39 text

• Client-side for PoC • Easy transition to server-side • Write docs for dummies • Unambiguous tools • Unambiguous context • An Agent is just a loop LLMs do not actually call functions - you do! Lessons Learned

Slide 40

Slide 40 text

It's a wrap! • Slides: www.thinktecture.com/max-marschall • @MaxOSchulte, max.marschall@thinktecture.com