Slide 1
Slide 1 text
Forms++
Generating forms that match actual requirement
Max Marschall
@MaxOSchulte
Consultant / Developer
Slide 2
Slide 2 text
Arti
fi
cial Intelligence
Machine Learning
Data
Science
NLP
A
Deep Learning
Generative AI
Large Language Models
Image / Video Generation
Slide 3
Slide 3 text
Bla bla bla …
Intention
Slide 4
Slide 4 text
Bla bla bla …
Generation of…
Intention
Summarisation of… Automation of…
Simpli
fi
cation of…
Slide 5
Slide 5 text
Bla bla bla …
“Augmenting an App with an assistant, that helps the user based on displayed context.
Allowing an assistant to act for me in that context.”
Intention
Slide 6
Slide 6 text
Intention
Bla bla bla …
“Augmenting an App with an assistant, that helps the user based on displayed context.
Allowing an assistant to act for me in that context.”
Problem: Integration GenAI
Metainformation
Tool calls
Leaking secrets
System integration
Solutions
AST parsing
TS Decorators
Client-side "agents"
Barebones Angular
OpenAI
Slide 7
Slide 7 text
Intention
Bla bla bla …
“Augmenting an App with an assistant, that helps the user based on displayed context.
Allowing an assistant to act for me in that context.”
Slide 8
Slide 8 text
Tools & Functions
Process Think Display
Bla bla bla …
Slide 9
Slide 9 text
User Prompt
• Voice input
• Text input
• Image input
• Unstructured
• Universal interface
• Natural language as
fi
rst-class input
Slide 10
Slide 10 text
• Prompts
• RAG
• Sanitising
• Guards
• Vector-Database
• Tool selection
• Additional data
(Processing)
Slide 11
Slide 11 text
• System prompt
• Base information
• Rules
• Intention
• Output format
Processing
You are an friendly helpfull assistent proficient in creating and managing forms.
You will receive information on the state of an app.
If you feel there a …
System Prompt
https://docs.anthropic.com/en/release-notes/system-prompts#oct-22nd-2024
You are a data analyst API capable of sentiment analysis that responds in JSON. The JSON
schema should include
{
"sentiment_analysis": {
"sentiment": "string (positive, negative, neutral)",
"confidence_score": "number (0-1)"
# Include additional fields as required
}
https://console.groq.com/docs/text-chat
Prompts
Slide 12
Slide 12 text
• State
• Additional information
• Generation examples
Context
/** Additional Information: basic list of static pages shown in the side navigation. */
[
{ title: 'Contributions', icon: 'question_answer', link: ['/contributions'] },
{ title: 'Conferences', icon: 'podium', link: ['/conferences'] },
{ title: 'Speakers', icon: 'person', link: ['/speakers'] },
{ title: 'Collections', icon: 'category', link: ['/collections'] },
];
Slide 13
Slide 13 text
Tools & Functions
/**
* Accepts a resolved sitemap entry and navigates to it.
*
* e.g. path: /collections/:id
* ":id" is replaced by an actual entity id.
*/
public navigate(resolvedPath: string): void {
this.router.navigateByUrl('/' + resolvedPath);
}
{
"type": "function",
"function": {
"name": "navigate",
"description": "Accepts a resolved sitemap entry …,
"parameters": {
"type": "object",
"properties": {
"resolvedPath": {
"title": "resolvedPath",
"description": "",
"type": "string"
}
}
},
"required": ["resolvedPath"],
}
}
Slide 14
Slide 14 text
Large Language Model
• Transformer Neural Network
• Context & relation aware
• Pattern matching & recognition
• “Natural” language understanding
• Generating / creating content
• Summarisation
Slide 15
Slide 15 text
Large Language Model
• It cannot make decisions
• It cannot understand non-linear content
• It cannot draw conclusions
Slide 16
Slide 16 text
Large Language Model
• Not deterministic
• Can hallucinate
• Prompt-Injection
Slide 17
Slide 17 text
Result
Bla bla bla …
Slide 18
Slide 18 text
Result
=
Bla bla bla …
Slide 19
Slide 19 text
Tools & Functions
Process Work Display
Slide 20
Slide 20 text
• Pattern matching
• Context extraction
• Language understanding
• Transformation “language to language”
Creating Components
Used LLM capabilities
Slide 21
Slide 21 text
• “Contract”
• Output Language
• Interpreted in Code
• Guiding Rails
• Output Validation
Creating Components
Schema De
fi
nition
Slide 22
Slide 22 text
Creating Components
{
"type": "object",
"properties": {
"steps": {
"type": "array",
"items": {
"type": "object",
"properties": {
"title": {
"type": "string",
"description": "Wizard Step Title."
},
"description": {
"type": "string",
"description": "Wizard Step Short Description.
},
"fields": {
"type": "array",
"items": {
"type": "object",
"properties": {
"type": {
"type": "string",
"enum": [
"title",
"subtitle",
"description",
"text",
"textarea",
"select",
"radioGroup",
"date",
"dateRange",
const WizardStepSchema = z.object({
steps: z
.object({
title: z.string().describe('Wizard Step Title.'),
description: z.string().describe('Wizard Step Short Description.'),
fields: z
.object({
type: z.nativeEnum(FormFieldType).describe('Type of form'),
description: z.string()
.describe('Description of the Field. What it is and… ‘),
label: z.string().describe('Form field label…'),
required: z.boolean().describe('Whether the field …’),
key: z.string().describe('Key to store the value under.'),
})
.array(),
})
.array(),
});
Zod Schema de
fi
nition
Schema De
fi
nition
Slide 23
Slide 23 text
Creating Components
{
"HeaderFormField": {
"description": "Header shown in a form. The label is…
"type": "object",
"properties": {
"type": {
"description": "Type of form field.",
"type": "string",
"const": "title"
},
…
"required": {
"description": "Whether the field is required…
"type": "boolean"
},
…
},
"required": ["key", "label", "type"]
},
/** Base type for all form field types. */
interface BaseFormField {
/** Type of form field. */
type: FormFieldType;
/** Description of the Field. What it is and what values should be there. */
description?: string;
/** Form field label shown to the user. */
label: string;
/** Whether the field is required. Defaults to false. */
required?: boolean;
/** Key to store the value under. */
key: string;
}
/** Header shown in a form. The label is used as its text. */
interface HeaderFormField extends BaseFormField {
type: FormFieldType.Title;
}
AST Parsing
Schema De
fi
nition
Slide 24
Slide 24 text
Creating Components
Create a Wizard to collect information
about a car accident. The
fi
rst step is to
collect a clients information as well as the
insurance ID. The second step is about
collection information on the accident,
What type of accident, which other party
is involved, which fault it was. The third
and
fi
nal step is about accepting the
terms of use and a check if it is ok to
process these data.
steps: [
{
title: 'Client Information',
description: 'Please provide your personal and insurance info
fields: [
{
type: 'text',
description: 'Enter your legal first name as it appea
label: 'First Name',
required: true,
key: 'firstName',
},
…
],
},
],
[
{
title: 'Accident Information',
description: 'Please provide details about the accident',
fields: [
{
type: 'select',
description: 'Select the type of accident',
label: 'Type of Accident',
required: true,
key: ‘accidentType',
options: [
{ value: 'collision', label: 'Collision' },
{ value: 'theft', label: 'Theft' },
{ value: 'vandalism', label: 'Vandalism' },
{ value: 'other', label: 'Other' },
],
Slide 25
Slide 25 text
Creating Components
Create a Wizard to collect information
about a car accident. The
fi
rst step is to
collect a clients information as well as the
insurance ID. The second step is about
collection information on the accident,
What type of accident, which other party
is involved, which fault it was. The third
and
fi
nal step is about accepting the
terms of use and a check if it is ok to
process these data.
steps: [
{
title: 'Client Information',
description: 'Please provide your personal and insurance info
fields: [
{
type: 'text',
description: 'Enter your legal first name as it appea
label: 'First Name',
required: true,
key: 'firstName',
},
…
],
},
],
[
{
title: 'Accident Information',
description: 'Please provide details about the accident',
fields: [
{
type: 'select',
description: 'Select the type of accident',
label: 'Type of Accident',
required: true,
key: ‘accidentType',
options: [
{ value: 'collision', label: 'Collision' },
{ value: 'theft', label: 'Theft' },
{ value: 'vandalism', label: 'Vandalism' },
{ value: 'other', label: 'Other' },
],
Slide 26
Slide 26 text
Creating Components
Create a Wizard to collect information
about a car accident. The
fi
rst step is to
collect a clients information as well as the
insurance ID. The second step is about
collection information on the accident,
What type of accident, which other party
is involved, which fault it was. The third
and
fi
nal step is about accepting the
terms of use and a check if it is ok to
process these data.
steps: [
{
title: 'Client Information',
description: 'Please provide your personal and insurance info
fields: [
{
type: 'text',
description: 'Enter your legal first name as it appea
label: 'First Name',
required: true,
key: 'firstName',
},
…
],
},
],
[
{
title: 'Accident Information',
description: 'Please provide details about the accident',
fields: [
{
type: 'select',
description: 'Select the type of accident',
label: 'Type of Accident',
required: true,
key: ‘accidentType',
options: [
{ value: 'collision', label: 'Collision' },
{ value: 'theft', label: 'Theft' },
{ value: 'vandalism', label: 'Vandalism' },
{ value: 'other', label: 'Other' },
],
Slide 27
Slide 27 text
Tools & Functions
Process Work Display
Slide 28
Slide 28 text
Process Work
Tools & Functions
Slide 29
Slide 29 text
Process Work
Tools & Functions
{
"HeaderFormField": {
"description": "Header shown in a form. The label is…
"type": "object",
"properties": {
"type": {
"description": "Type of form field.",
"type": "string",
"const": "title"
},
…
"required": {
"description": "Whether the field is required…
"type": "boolean"
},
…
},
"required": ["key", "label", "type"]
},
/** Base type for all form field types. */
interface BaseFormField {
/** Type of form field. */
type: FormFieldType;
/** Description of the Field. What it is and what values should be there. */
description?: string;
/** Form field label shown to the user. */
label: string;
/** Whether the field is required. Defaults to false. */
required?: boolean;
/** Key to store the value under. */
key: string;
}
/** Header shown in a form. The label is used as its text. */
interface HeaderFormField extends BaseFormField {
type: FormFieldType.Title;
}
Slide 30
Slide 30 text
Tools & Functions
Process Work
/**
* Creates form fields for a provided list of formField definition.
*/
createFormFields(fields: FormField[]): FormField[] {
…
}
/** Union of all supported form field types. */
export type FormField =
| HeaderFormField
| SubheaderFormFiel
| …
Slide 31
Slide 31 text
/**
* Creates form fields for a provided list of formField definition.
*/
@AiTool
createFormFields(fields: FormField[]): FormField[] {
…
}
/** Union of all supported form field types. */
export type FormField =
| HeaderFormField
| SubheaderFormFiel
| …
Tools & Functions
Process Work
{
"type": "function",
"className": "FormBuilderService",
"classDocumentation": "",
"function": {
"name": "createFormFields",
"title": "createFormFields",
"type": "function",
"description": "Creates form fields for a provided list…
"parameters": {
"type": "object",
"properties": {
"fields": {
"title": "fields",
"description": "Union of all supported form…
"anyOf": [
{
"$ref": "#/definitions/HeaderFormField"
},
…
Slide 32
Slide 32 text
Process Work
Tools & Functions
/**
* Creates form fields for a provided list of formField definition.
*/
@AiTool
createFormFields(fields: FormField[]): FormField[] {
…
}
/** Union of all supported form field types. */
export type FormField =
| HeaderFormField
| SubheaderFormFiel
| …
{
"type": "function",
"className": "FormBuilderService",
"classDocumentation": "",
"function": {
"name": "createFormFields",
"title": "createFormFields",
"type": "function",
"description": "Creates form fields for a provided list…
"parameters": {
"type": "object",
"properties": {
"fields": {
"title": "fields",
"description": "Union of all supported form…
"anyOf": [
{
"$ref": "#/definitions/HeaderFormField"
},
…
Slide 33
Slide 33 text
Process Work
Tools & Functions
/**
* Creates form fields for a provided list of formField definition.
*/
@AiTool
createFormFields(fields: FormField[]): FormField[] {
…
}
/** Union of all supported form field types. */
export type FormField =
| HeaderFormField
| SubheaderFormFiel
| …
{
"type": "function",
"className": "FormBuilderService",
"classDocumentation": "",
"function": {
"name": "createFormFields",
"title": "createFormFields",
"type": "function",
"description": "Creates form fields for a provided list…
"parameters": {
"type": "object",
"properties": {
"fields": {
"title": "fields",
"description": "Union of all supported form…
"anyOf": [
{
"$ref": "#/definitions/HeaderFormField"
},
…
tool_calls": [
{
"id": "call_wqxp1eYfhDTZrkuXfPqFP7uc",
"type": "function",
"function": {
"name": "createFormFields",
"arguments": {
"fields": [
{
"type": "title",
"label": "Event Information",
"description": "Form to collect information about an upcoming event.",
"key": "eventInformationTitle"
},
{
"type": "text",
"label": "Event Name",
"description": "Enter the name of the event.",
"key": "eventName"
},
{
"type": "select",
"label": "Speaker",
"description": "Select an existing speaker for the event.",
"options": {
"labelKey": "fullName",
"collectionId": "speakers"
},
"key": "speaker"
}
]
}
…
Slide 34
Slide 34 text
Structured Output
Process Work
Structured Outputs is a feature that
ensures the model will always generate
responses that adhere to your
supplied JSON Schema, so you don't need
to worry about the model omitting a
required key, or hallucinating an invalid
enum value.
https://platform.openai.com/docs/guides/structured-outputs?context=without_parse
Slide 35
Slide 35 text
Structured Output
Process Work
const completion = await this.client.chat.completions.create({
tool_choice: 'auto',
model: 'gpt-4o-2024-08-06',
temperature: 0,
messages,
tools,
});
const completion = await this.client.chat.completions.create({
model: ‘gpt-4o-2024-08-06',
messages,
structured_output,
});
"fields": [
{
"type": "title",
"label": "Event Information",
"description": "Form to collect information about an upcoming event.",
"key": "eventInformationTitle"
},
…
]
tool_calls": [
{
"id": "call_wqxp1eYfhDTZrkuXfPqFP7uc",
"type": "function",
"function": {
"name": "createFormFields",
"arguments": {
"fields": [
{
"type": "title",
"label": "Event Information",
"description": "Form to collect information about an upcoming eve
"key": "eventInformationTitle"
…
Slide 36
Slide 36 text
Structured Output
Process Work
• Providing only an output schema, no function
• Some limitation
• “Use for presenting to the user”
• Reliable type-safety
• Explicit refusals
Slide 37
Slide 37 text
Structured Output
Process Work
• String: minLength, maxLength, pattern, format
• Numbers: minimum, maximum, multipleOf
• Objects: patternProperties, unevaluatedProperties,
propertyNames, minProperties, maxProperties
• Arrays: unevaluatedItems, contains, minContains, maxContains, minItems,
maxItems, uniqueItems
Limitations on strict
Slide 38
Slide 38 text
• “Talking to fast learning & smart junior"
• No default assumptions
• Documentation is key
• Examples are great
Development Mindset
Slide 39
Slide 39 text
• Avoid auto-commit
• Enable feedback
• Highlight probability
• Clear entry point into AI work
fl
ows
• Progress / generation indicators
• Suggest common actions
UI & UX
Slide 40
Slide 40 text
• Client-side for PoC
• Easy transition to server-side
• Write docs for dummies
• Unambiguous tools
• Unambiguous context
• An Agent is just a loop LLMs do not actually call functions - you do!
Lessons Learned
Slide 41
Slide 41 text
It's a wrap!
• Slides: www.thinktecture.com/max-marschall
• @MaxOSchulte, [email protected]