Next gen(eration) forms

Why create forms ourselves when we can leverage LLMs for repetitive tasks? In this session, Max Marschall provides an escape, making forms over data fun again. You will learn how to utilize Large Language Models to generate complex forms, with data access, model management, relations, and references in your web apps - lifting the workload from developers. However, developers are not the only beneficiaries of such an approach. Join this session to see how you can augment your web app with Generative AI to simplify the daily business of everyone in the loop, including the users.

Transcript

1. Next gen(eration) forms Max Marschall @MaxOSchulte Consultant / Developer
2. Bla bla bla … Intention
3. Bla bla bla … Generation of… Intention Summarisation of… Automation of… Simpli fi cation of…
4. Bla bla bla … “Augmenting an App with an assistant, that helps the user based on displayed context. Allowing an assistant to act for me in that context.” Intention
5. Intention Bla bla bla … “Augmenting an App with an assistant, that helps the user based on displayed context. Allowing an assistant to act for me in that context.” Problem: Integration GenAI Metainformation Tool calls Leaking secrets System integration Solutions AST parsing TS Decorators Client-side "agents" Barebones Angular OpenAI
6. Intention Bla bla bla … “Augmenting an App with an assistant, that helps the user based on displayed context. Allowing an assistant to act for me in that context.”
7. Tools & Functions Process Think Display Bla bla bla …
8. User Prompt • Voice input • Text input • Image input • Unstructured • Universal interface • Natural language as fi rst-class input
9. • Prompts • RAG • Sanitising • Guards • Vector-Database • Tool selection • Additional data (Processing)
10. • System prompt • Base information • Rules • Intention • Output format Processing You are an friendly helpfull assistent proficient in creating and managing forms. You will receive information on the state of an app. If you feel there a … System Prompt https://docs.anthropic.com/en/release-notes/system-prompts#oct-22nd-2024 You are a data analyst API capable of sentiment analysis that responds in JSON. The JSON schema should include { "sentiment_analysis": { "sentiment": "string (positive, negative, neutral)", "confidence_score": "number (0-1)" # Include additional fields as required } https://console.groq.com/docs/text-chat Prompts
11. • State • Additional information • Generation examples Context /** Additional Information: basic list of static pages shown in the side navigation. */ [ { title: 'Contributions', icon: 'question_answer', link: ['/contributions'] }, { title: 'Conferences', icon: 'podium', link: ['/conferences'] }, { title: 'Speakers', icon: 'person', link: ['/speakers'] }, { title: 'Collections', icon: 'category', link: ['/collections'] }, ];
12. Tools & Functions /** * Accepts a resolved sitemap entry and navigates to it. * * e.g. path: /collections/:id * ":id" is replaced by an actual entity id. */ public navigate(resolvedPath: string): void { this.router.navigateByUrl('/' + resolvedPath); } { "type": "function", "function": { "name": "navigate", "description": "Accepts a resolved sitemap entry …, "parameters": { "type": "object", "properties": { "resolvedPath": { "title": "resolvedPath", "description": "", "type": "string" } } }, "required": ["resolvedPath"], } }
13. Large Language Model • Transformer Neural Network • Context & relation aware • Pattern matching & recognition • “Natural” language understanding • Generating / creating content • Summarisation
14. Large Language Model • It cannot make decisions • It cannot understand non-linear content • It cannot draw conclusions
15. Large Language Model • Not deterministic • Can hallucinate • Prompt-Injection
16. Result Bla bla bla …
17. Result = Bla bla bla …
18. Tools & Functions Process Work Display
19. • Pattern matching • Context extraction • Language understanding • Transformation “language to language” Creating Components Used LLM capabilities
20. Creating Components Create a Wizard to collect information about a car accident. The fi rst step is to collect a clients information as well as the insurance ID. The second step is about collection information on the accident, What type of accident, which other party is involved, which fault it was. The third and fi nal step is about accepting the terms of use and a check if it is ok to process these data. steps: [ { title: 'Client Information', description: 'Please provide your personal and insurance info fields: [ { type: 'text', description: 'Enter your legal first name as it appea label: 'First Name', required: true, key: 'firstName', }, … ], }, ], [ { title: 'Accident Information', description: 'Please provide details about the accident', fields: [ { type: 'select', description: 'Select the type of accident', label: 'Type of Accident', required: true, key: ‘accidentType', options: [ { value: 'collision', label: 'Collision' }, { value: 'theft', label: 'Theft' }, { value: 'vandalism', label: 'Vandalism' }, { value: 'other', label: 'Other' }, ],
21. Creating Components Create a Wizard to collect information about a car accident. The fi rst step is to collect a clients information as well as the insurance ID. The second step is about collection information on the accident, What type of accident, which other party is involved, which fault it was. The third and fi nal step is about accepting the terms of use and a check if it is ok to process these data. steps: [ { title: 'Client Information', description: 'Please provide your personal and insurance info fields: [ { type: 'text', description: 'Enter your legal first name as it appea label: 'First Name', required: true, key: 'firstName', }, … ], }, ], [ { title: 'Accident Information', description: 'Please provide details about the accident', fields: [ { type: 'select', description: 'Select the type of accident', label: 'Type of Accident', required: true, key: ‘accidentType', options: [ { value: 'collision', label: 'Collision' }, { value: 'theft', label: 'Theft' }, { value: 'vandalism', label: 'Vandalism' }, { value: 'other', label: 'Other' }, ],
22. Creating Components Create a Wizard to collect information about a car accident. The fi rst step is to collect a clients information as well as the insurance ID. The second step is about collection information on the accident, What type of accident, which other party is involved, which fault it was. The third and fi nal step is about accepting the terms of use and a check if it is ok to process these data. steps: [ { title: 'Client Information', description: 'Please provide your personal and insurance info fields: [ { type: 'text', description: 'Enter your legal first name as it appea label: 'First Name', required: true, key: 'firstName', }, … ], }, ], [ { title: 'Accident Information', description: 'Please provide details about the accident', fields: [ { type: 'select', description: 'Select the type of accident', label: 'Type of Accident', required: true, key: ‘accidentType', options: [ { value: 'collision', label: 'Collision' }, { value: 'theft', label: 'Theft' }, { value: 'vandalism', label: 'Vandalism' }, { value: 'other', label: 'Other' }, ],
23. • “Contract” • Output Language • Interpreted in Code • Guiding Rails • Output Validation Creating Components Schema De fi nition
24. Creating Components { "type": "object", "properties": { "steps": { "type": "array", "items": { "type": "object", "properties": { "title": { "type": "string", "description": "Wizard Step Title." }, "description": { "type": "string", "description": "Wizard Step Short Description. }, "fields": { "type": "array", "items": { "type": "object", "properties": { "type": { "type": "string", "enum": [ "title", "subtitle", "description", "text", "textarea", "select", "radioGroup", "date", "dateRange", const WizardStepSchema = z.object({ steps: z .object({ title: z.string().describe('Wizard Step Title.'), description: z.string().describe('Wizard Step Short Description.'), fields: z .object({ type: z.nativeEnum(FormFieldType).describe('Type of form'), description: z.string() .describe('Description of the Field. What it is and… ‘), label: z.string().describe('Form field label…'), required: z.boolean().describe('Whether the field …’), key: z.string().describe('Key to store the value under.'), }) .array(), }) .array(), }); Zod Schema de fi nition Schema De fi nition
25. Creating Components { "HeaderFormField": { "description": "Header shown in a form. The label is… "type": "object", "properties": { "type": { "description": "Type of form field.", "type": "string", "const": "title" }, … "required": { "description": "Whether the field is required… "type": "boolean" }, … }, "required": ["key", "label", "type"] }, /** Base type for all form field types. */ interface BaseFormField { /** Type of form field. */ type: FormFieldType; /** Description of the Field. What it is and what values should be there. */ description?: string; /** Form field label shown to the user. */ label: string; /** Whether the field is required. Defaults to false. */ required?: boolean; /** Key to store the value under. */ key: string; } /** Header shown in a form. The label is used as its text. */ interface HeaderFormField extends BaseFormField { type: FormFieldType.Title; } AST Parsing Schema De fi nition
26. Schemas Process Work Display
27. Process Work Schemas • Output Schema in Prompt • Structured Outputs • Function / Tool calling
28. Structured Output (OpenAI) Process Work Structured Outputs is a feature that ensures the model will always generate responses that adhere to your supplied JSON Schema, so you don't need to worry about the model omitting a required key, or hallucinating an invalid enum value. https://platform.openai.com/docs/guides/structured-outputs?context=without_parse
29. Process Work const completion = await this.client.chat.completions.create({ tool_choice: 'auto', model: 'gpt-4o-2024-08-06', temperature: 0, messages, tools, }); const completion = await this.client.chat.completions.create({ model: ‘gpt-4o-2024-08-06', messages, structured_output, }); "fields": [ { "type": "title", "label": "Event Information", "description": "Form to collect information about an upcoming event.", "key": "eventInformationTitle" }, … ] tool_calls": [ { "id": "call_wqxp1eYfhDTZrkuXfPqFP7uc", "type": "function", "function": { "name": "createFormFields", "arguments": { "fields": [ { "type": "title", "label": "Event Information", "description": "Form to collect information about an upcoming eve "key": "eventInformationTitle" … Structured Output (OpenAI)
30. Structured Output (OpenAI) Process Work • Providing only an output schema, no function • Some limitation • “Use for presenting to the user” • Reliable type-safety • Explicit refusals
31. Process Work • String: minLength, maxLength, pattern, format • Numbers: minimum, maximum, multipleOf • Objects: patternProperties, unevaluatedProperties, propertyNames, minProperties, maxProperties • Arrays: unevaluatedItems, contains, minContains, maxContains, minItems, maxItems, uniqueItems Limitations on strict Structured Output (OpenAI)
32. Process Work Tools & Functions { "HeaderFormField": { "description": "Header shown in a form. The label is… "type": "object", "properties": { "type": { "description": "Type of form field.", "type": "string", "const": "title" }, … "required": { "description": "Whether the field is required… "type": "boolean" }, … }, "required": ["key", "label", "type"] }, /** Base type for all form field types. */ interface BaseFormField { /** Type of form field. */ type: FormFieldType; /** Description of the Field. What it is and what values should be there. */ description?: string; /** Form field label shown to the user. */ label: string; /** Whether the field is required. Defaults to false. */ required?: boolean; /** Key to store the value under. */ key: string; } /** Header shown in a form. The label is used as its text. */ interface HeaderFormField extends BaseFormField { type: FormFieldType.Title; }
33. Tools & Functions Process Work /** * Creates form fields for a provided list of formField definition. */ createFormFields(fields: FormField[]): FormField[] { … } /** Union of all supported form field types. */ export type FormField = | HeaderFormField | SubheaderFormFiel | …
34. /** * Creates form fields for a provided list of formField definition. */ @AiTool createFormFields(fields: FormField[]): FormField[] { … } /** Union of all supported form field types. */ export type FormField = | HeaderFormField | SubheaderFormFiel | … Tools & Functions Process Work { "type": "function", "className": "FormBuilderService", "classDocumentation": "", "function": { "name": "createFormFields", "title": "createFormFields", "type": "function", "description": "Creates form fields for a provided list… "parameters": { "type": "object", "properties": { "fields": { "title": "fields", "description": "Union of all supported form… "anyOf": [ { "$ref": "#/definitions/HeaderFormField" }, …
35. Process Work Tools & Functions /** * Creates form fields for a provided list of formField definition. */ @AiTool createFormFields(fields: FormField[]): FormField[] { … } /** Union of all supported form field types. */ export type FormField = | HeaderFormField | SubheaderFormFiel | … { "type": "function", "className": "FormBuilderService", "classDocumentation": "", "function": { "name": "createFormFields", "title": "createFormFields", "type": "function", "description": "Creates form fields for a provided list… "parameters": { "type": "object", "properties": { "fields": { "title": "fields", "description": "Union of all supported form… "anyOf": [ { "$ref": "#/definitions/HeaderFormField" }, …
36. Process Work Tools & Functions /** * Creates form fields for a provided list of formField definition. */ @AiTool createFormFields(fields: FormField[]): FormField[] { … } /** Union of all supported form field types. */ export type FormField = | HeaderFormField | SubheaderFormFiel | … { "type": "function", "className": "FormBuilderService", "classDocumentation": "", "function": { "name": "createFormFields", "title": "createFormFields", "type": "function", "description": "Creates form fields for a provided list… "parameters": { "type": "object", "properties": { "fields": { "title": "fields", "description": "Union of all supported form… "anyOf": [ { "$ref": "#/definitions/HeaderFormField" }, … tool_calls": [ { "id": "call_wqxp1eYfhDTZrkuXfPqFP7uc", "type": "function", "function": { "name": "createFormFields", "arguments": { "fields": [ { "type": "title", "label": "Event Information", "description": "Form to collect information about an upcoming event.", "key": "eventInformationTitle" }, { "type": "text", "label": "Event Name", "description": "Enter the name of the event.", "key": "eventName" }, { "type": "select", "label": "Speaker", "description": "Select an existing speaker for the event.", "options": { "labelKey": "fullName", "collectionId": "speakers" }, "key": "speaker" } ] } …
37. • “Talking to fast learning & smart junior" • No default assumptions • Documentation is key • Examples are great • Limited possibilities deliver better results Development Mindset
38. • Avoid auto-commit • Enable feedback • Highlight probability • Clear entry point into AI work fl ows • Progress / generation indicators • Suggest common actions UI & UX
39. • Client-side for PoC • Easy transition to server-side • Write docs for dummies • Unambiguous tools • Unambiguous context • An Agent is just a loop LLMs do not actually call functions - you do! Lessons Learned
40. It's a wrap! • Slides: www.thinktecture.com/max-marschall • @MaxOSchulte, max.marschall@thinktecture.com