Intelligente Formulare Christian Liebel @christianliebel Consultant Angular-Forms mit AI & LLMs aufpeppen

Hello, it’s me. Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Christian Liebel X: @christianliebel Bluesky: @christianliebel.com Email: christian.liebel @thinktecture.com Angular, PWA & Generative AI Slides: thinktecture.com /christian-liebel

Overview Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Generative AI Text OpenAI GPT Mistral … Audio/Music Musico Soundraw … Images DALL·E Firefly … Video Sora Runway … Speech Whisper tortoise-tts …

Overview Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Generative AI Text OpenAI GPT Mistral … Audio/Music Musico Soundraw … Images DALL·E Firefly … Video Sora Runway … Speech Whisper tortoise-tts …

Large: Trained on lots of data Language: Process and generate text in natural language Models: Programs/neural networks Examples: – GPT (ChatGPT, Bing Chat, …) – Gemini, Gemma (Google) – LLaMa (Meta AI) Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Large Language Models

Prompts serve as the universal interface Unstructured text conveying specific semantics Paradigm shift in software architecture Natural language becomes a first-class citizen Caveats Non-determinism and hallucination, prompt injections Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Large Language Models

Use cases beyond chat bots…? Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Large Language Models

Just transfer the 17.34 euros to me, my IBAN is DE02200505501015871393. I am with Hamburger Sparkasse (HASPDEHH). Data Extraction Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Use Case Nice, here is my address: Peter Müller, Rheinstr. 7, 04435 Schkeuditz

Just transfer the 17.34 euros to me, my IBAN is DE02200505501015871393. I am with Hamburger Sparkasse (HASPDEHH). Data Extraction Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Use Case Nice, here is my address: Peter Müller, Rheinstr. 7, 04435 Schkeuditz

protected readonly formGroup = this.fb.group({ firstName: [''], lastName: [''], addressLine1: [''], addressLine2: [''], city: [''], state: [''], zip: [''], country: [''], }); Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Idea Nice, here is my address: Peter Müller, Rheinstr. 7, 04435 Schkeuditz Smart Form Filler (LLM)

Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Idea DEMO

Form Field Prompt Generator Model Backend Response Parser Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Architecture

Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Form Field “The insurance number starts with INS, followed by six characters.” “Try to determine the post code based on the input.”

Async Clipboard API Allows reading from/writing to the clipboard in an asynchronous manner Reading from the clipboard requires user consent first (privacy!) Supported by Chrome, Edge and Safari and Firefox Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Prompt Generator

Message types System message • The form has the following setup: { "firstName": "", "lastName": "" } User message • Passt, dann hier schonmal meine Adresse: Peter Müller Assistant message • { "firstName": "Peter", "lastName": "Müller" } Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Prompt Generator

Message types Some models add additional output or may hallucinate properties, so the response cannot be parsed back to JSON: Sure! I’m glad to help. Here’s your JSON: { "firstName": "Peter", "lastName": "Müller" } There are two alternatives: 1. JSON mode: Special output mode that guarantees valid JSON. 2. Tool/function calling: Well-defined API for the model to call a tool. Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Prompt Generator

Tool calling { "name": "fillForm", "description": "Fills in the form.", "parameters": { "type": "object", "properties": { "firstName": { "type": "string", "description": "The user's first name." }, }, } Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Prompt Generator Natural language descriptions evaluated by the LLM JSON schema that defines the tool’s API shape

Tool messages System message • You are a helpful assistant that helps filling in forms. • You can call these tools if needed: User message • Passt, dann hier schonmal meine Adresse: Peter Müller Tool message • { "tool_name": "fillForm", "tool_args": { "firstName": "Peter", "lastName": "Müller" } } Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Prompt Generator { "name": "fillForm", "description": "Fills in the form.", "parameters": { "type": "object", "properties": { "firstName": { "type": "string", "description": "The user's first name." }, }, }

Tool messages Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Prompt Generator DEMO

Multimodal models Recent LLMs are capable of processing other content besides just text. Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Prompt Generator

Multimodal models System message • You are a helpful assistant that helps filling in forms. • You can call these tools if needed: User message Tool message • { "tool_name": "fillForm", "tool_args": { "firstName": "Peter", "lastName": "Müller" } } Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Prompt Generator { "name": "fillForm", "description": "Fills in the form.", "parameters": { "type": "object", "properties": { "firstName": { "type": "string", "description": "The user's first name." }, }, }

Multimodal models Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Prompt Generator DEMO

Realtime models Recent models are capable of having realtime conversations. Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Prompt Generator

Realtime models Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Prompt Generator DEMO

Cloud backends Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Model Backend

Local backend (WebLLM) Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Model Backend

https://webllm.mlc.ai/ Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen WebLLM

Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen

Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Prompt API Operating System Website HTML/JS Browser Internet Apple Intelligence Gemini Nano

Part of Chrome’s Built-In AI initiative – Exploratory API for local experiments and use case determination – Downloads Gemini Nano into Google Chrome – Model can be shared across origins – Multimodality in development – Fine-tuning API might follow in the future Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Prompt API https://developer.chrome.com/docs/ai/built-in

Comparison 45 33 1200 0 200 400 600 800 1000 1200 1400 WebLLM (Llama3-8b, M4) Azure OpenAI (gpt-4o-mini) Groq (Llama3-8b) Tokens/sec Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Performance WebLLM/Groq: Own tests (14.11.2024), OpenAI/Azure OpenAI: https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/provisioned-throughput (18.07.2024)

Built-in AI Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Prompt API DEMO

Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Prompt Parser Assistant message • { "firstName": "Peter", "lastName": "Müller" }

Assistant message Parsing the assistant message as text/JSON/… Tool calling Specifying a well-defined interface via a JSON schema called by the LLM (safer, growing support) Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Prompt Parser

Form Field Prompt Generator Model Backend Response Parser Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Architecture

– Super accurate – Versatile thanks to layered architecture – Runs in the browser, locally, or in the cloud – Very fast execution is possible via Groq – Multimodal models can process content types other than text – Realtime models allow conversations Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen Summary

Q&A Intelligente Formulare Angular-Forms mit AI & LLMs aufpeppen

Thank you for your kind attention! Christian Liebel @christianliebel [email protected]