Angular-Apps smarter machen mit Generativer KI: Lokal und offlinefähig - Hands-on!

Angular-Apps smarter machen mit Generativer KI: Lokal und offlinefähig Hands-on!
Christian Liebel @christianliebel Consultant

Christian Liebel Hello, it’s me. Angular, PWA & Generative AI Slides: thinktecture.com /christian-liebel X: @christianliebel Bluesky: @christianliebel.com Email: christian.liebel @thinktecture.com

Timetable (Original) 09:00–10:30 Block 1 10:30–11:00 Coffee Break 11:00–12:30 Block 2 12:30–13:30 Lunch Break 13:30–15:00 Block 3 15:00–15:30 Coffee Break 15:30–16:30 Block 4

Timetable (Proposal) 09:00–10:30 Block 1 10:30–10:55 Coffee Break 10:55–12:30 Block 2 12:30–13:25 Lunch Break 13:25–15:00 Block 3 15:00–15:25 Coffee Break 15:20–16:15 Block 4

Expectations What to expect Focus on web app development Focus on Generative AI Up-to-date insights: the ML/AI field is evolving fast Live demos on real hardware 18 hands-on labs What not to expect Deep dive into AI specifics, RAG, model finetuning or training Stable libraries or specifications Hu ak! e r b y a m s g in h T ents! m e ir u q e r h ig H ! ge downloads

Workshop Slides

DEMO

Demo Use Case (Workshop Edition) DEMO

Setup Setup complete? (Node.js, Google Chrome, Editor, Git, macOS/Windows, 20 GB free disk space, 6 GB VRAM)

Setup git clone https://github.com/thinktecture/basta2026-genai.git cd basta-2026-genai npm i LAB #0

Generative AI everywhere Source: https://www.apple.com/chde/apple-intelligence/

Single-Page Applications Run locally on the user’s system

Progressive Web Apps Make SPAs offline-capable fetch Website HTML/JS Internet Cache Service Worker

Generative AI Overview Text Images Video Audio/Music Speech OpenAI GPT Claude Opus … Nano 🍌 Firefly … Omni Sora … Musico Soundraw … Whisper tortoise-tts …

WebNN, Built-in AI, WebMCP What’s New in Web AI? Generative
AI Cloud Providers Examples

Generative AI Cloud Providers Drawbacks Require a (stable) internet connection Subject to network latency and server availability Data is transferred to the cloud service Require a subscription

Can we run GenAI models locally?

Large Language Models Large: Trained on lots of data Language: Process and generate text Models: Programs/neural networks Examples: – Sonnet, Opus, Faible (Claude) – Gemini, Gemma (Google) – LFM (Liquid AI)

Large Language Models Token A meaningful unit of text (e.g., a word, a part of a word, a character). Context Window The maximum amount of tokens the model can process. Parameters/weights Internal variables learned during training, used to make predictions.

Large Language Models Prompts serve as the universal interface Unstructured text conveying specific semantics Paradigm shift in software architecture Natural language becomes a first-class citizen Caveats Non-determinism and hallucination, prompt injections

npm i @huggingface/transformers npm start -- -o LAB #1

Transformers.js @huggingface/transformers – JavaScript library by Hugging Face 🤗 – Functionally equivalent to Hugging Face’s transformers python lib – Supports various ML/AI use cases (LLMs, computer vision, audio, …) – Models are executed on-device (100% local, offline-capable) – Uses ONNX Runtime (model inference runtime) internally

Transformers.js vs. WebLLM Source: https://www.npmcharts.com/compare/@huggingface/transformers,@mlc-ai/web-llm?log=false

WebLLM Alternative

Model Selection Size Comparison Model:Parameters lfm2.5-thinking:1.2b lfm2:2.6b ministral-3:3b gemma4:12b gpt-oss:20b devstral-2:123b Size 0.7 GB 1.6 GB 3.0 GB 8.1 GB 14 GB 75 GB

Model Selection Liquid Foundation Models (LFM) Highly performant on-device LLMs by LiquidAI https://www.liquid.ai/models For this workshop, we are going to use the 2.6B model. Model sheet: https://huggingface.co/onnxcommunity/LFM2-2.6B-ONNX Source: https://www.liquid.ai/blog/introducing-lfm2-2-6b-redefining-efficiency-in-language-models

Downloading a model (1/3) 1. Go to webgpureport.org 2. Does your GPU support the feature “shader-f16”? LAB #2

Downloading a model (2/3) LAB #2 3. In src/app/todo/todo.ts (ngOnInit()), add the following line: await this.llmService.loadModel('2.6B'); 4. If your GPU does not support shader-f16, add this parameter: await this.llmService.loadModel('2.6B', 'q4');

Downloading a model (3/3) LAB #2 5. In todo.html, change the following lines: @if(!llmService.isReady()) { <mat-progress-bar mode="determinate" [value]="llmService.progress()"></mat-progress-bar> } … <button mat-raised-button (click)="runPrompt(prompt.value, langModel.value)" [disabled]="!llmService.isReady()"> The progress bar should begin to move.

Cache API Storing model files locally Website HTML/JS Cache with model files Internet Hugging Face Note: Due to the Same-Origin Policy, models cannot be shared across origins.

Cache API Parameter cache

WebAssembly (Wasm) – Bytecode for the web – Compile target for arbitrary languages – Can be faster than JavaScript – WebLLM uses a modelspecific Wasm library to accelerate model computations

WebGPU – Grants low-level access to the Graphics Processing Unit (GPU) – Near native performance for machine learning applications – Supported by Chromium-based browsers on Windows and macOS from version 113, Safari 26, and Firefox 141 on Windows

WebNN – Grants web apps access to the device’s CPU, GPU and Neural Processing Unit (NPU) – In specification by the WebML Working Group at W3C – Origin Trial in Chrome 146 – Potentially even better performance compared to WebGPU Source: https://webmachinelearning.github.io/webnn-intro/

WebNN DEMO https://huggingface.co/webnn/spaces

WebNN: near-native inference performance Source: Intel. Browser: Chrome Canary 118.0.5943.0, DUT: Dell/Linux/i7-1260P, single p-core, Workloads: MediaPipe solution models (FP32, batch=1)

WebNN Drawbacks Models can’t be shared across origins Inference is fast, but doesn’t reach full native speed

Cross-Origin Storage https://github.com/wicg/cross-origin-storage

Model inference (1/4) 1. In todo.ts, add the following line at the LAB #3 marker: protected readonly reply = signal(''); LAB #3

Model inference (2/4) LAB #3 2. In the inferTransformersJs() method, add the following code: this.llmService.clearPastKeyValues(); const messages = [ { role: "user", content: userPrompt }, ]; return this.llmService.generateResponse(messages, []);

Model inference (3/4) 3. In the runPrompt() method, add the following code: this.reply.set('…'); const chunks = inferenceEngine === 'transformers-js' ? this.inferTransformersJs(userPrompt) : this.inferPromptApi(userPrompt); let reply = ''; for await (const chunk of chunks) { reply += chunk; this.reply.set(reply); } LAB #3

Model inference (4/4) LAB #3 4. In todo.html, change the following line: <pre>{{ reply() }}</pre> You should now be able to send prompts to the model and see the responses in the template. ⚠ Note: Browsers support better options for streaming LLM responses: https://developer.chrome.com/docs/ai/render-llm-responses

Todo management (1/2) In todo.ts, add the following signal at the top: protected readonly todos = signal<TodoDto[]>([]); Add the following line to the addTodo() method: text ??= prompt() ?? ''; this.todos.update(todos => [...todos, { done: false, text }]); LAB #4

Todo management (2/2) LAB #4 In todo.html, add the following lines to add todos from the UI: @for (todo of todos(); track $index) { <mat-list-option>{{ todo.text }}</mat-list-option> }

Todo management (extended) LAB #5 @for (todo of todos(); track $index) { <mat-list-option [(selected)]="todo.done"> {{ todo.text }} </mat-list-option> } ⚠ Boo! This pattern is not recommended. Instead, you should set the changed values on the signal. But this messes up with Angular Material…

Chat with data Concept and limitations The todo data has to be serialized (e.g., as JSON or Markdown). For the sake of simplicity, we will add all TODOs to the prompt. Remember: LLMs have a context window (LFM2-2.6B: 32K). If you need to chat with larger sets of text, refer to Retrieval Augmented Generation (RAG). These are the todos: * Wash clothes * Pet the dog * Take out the trash

Chat with data System prompt Metaprompt that defines… – character – capabilities/limitations – output format – behavior – grounding data You are a helpful assistant. Answer user questions on todos. Generate a valid JSON object. Avoid negative content. These are the user’s todos: … Hallucinations and prompt injections cannot be eliminated.

Chat with data Flow System message • The user has these todos: 1. … 2. … 3. … User message • How many todos do I have? Assistant message • You have three todos.

Chat with data Using a system & user prompt LAB #6 Adjust the code in inferTransformerJs() to include the system prompt: const systemPrompt = `Here's the user's todo list: ${JSON.stringify(this.todos())}`; const messages = [ { role: "system", content: systemPrompt }, { role: "user", content: userPrompt }, ];

Prompt Engineering Techniques – Providing examples (single shot, few shot, …) – Priming outputs – Specify output structure – Repeating instructions – Chain of thought – … Success also depends on the model. https://learn.microsoft.com/en-us/azure/ai-foundry/openai/concepts/prompt-engineering

Prompt Engineering LAB #7 const systemPrompt = `You are a helpful assistant. The user will ask questions about their todo list. Briefly answer the questions. Don't try to make up an answer if you don't know it. Here's the user's todo list: ${JSON.stringify(this.todos())}`;

Prompt Engineering Alternatives Prompt Engineering Retrieval Augmented Generation Fine-tuning Custom model Effort

Performance LAB #8 Adjust todo.ts as follows: return this.llmService.generateResponse(messages, [], { measurePerformance: true, }); Ask a new question and check your console for performance statistics.

Performance Workshop Participants Device Tokens/s (Decode)

Performance Comparison 1400 1200 1200 1000 800 600 400 200 45 33 WebLLM (Llama3-8b, M4) Azure OpenAI (gpt-4o-mini) 0 Groq (Llama3-8b) Tokens/sec WebLLM/Groq: Own tests (14.11.2024), OpenAI/Azure OpenAI: https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/provisioned-throughput (18.07.2024)

Tool Calling Tool Calling allows an LLM to execute “real-world” actions. A tool usually has… – a name – a natural-language description – an interface definition usually in JSON Schema The LLM “calls” the tool. However, the developer has to take care of actually executing the code and to feed back the result into the conversation. add_todo get_weather search_web read_file

Tool Calling LAB #9 1. Add the TODO_TOOL to the tools array in inferTransformersJs(): return this.llmService.generateResponse(messages, [TODO_TOOL], { measurePerformance: true }); 2. Add this line to the end of the runPrompt() method: this.llmService.executeToolCalls(reply, { addTodo: (args: { text: string }) => this.addTodo(args.text), });

Web AI Landscape Bring Your Own AI (BYOAI) – Libraries – WebLLM – Frameworks – Transformers.js – ONNX Runtime – TensorFlow.js – APIs – WebGPU, WebNN – Cross-Origin Storage NEW! Built-in AI (BIAI) – Writing Assistance APIs – Summarizer API – Writer API – Rewriter API – Proofreader API – Translator & Language Detector APIs – Embedding API NEW! – Prompt API NEW! Tool use – WebMCP

LAB #10 Prompt API If you are using Edge, go to edge://flags and set Prompt API for Phi mini to Enabled. Open Dev Tools (F12) and enter: await LanguageModel.create(); Go to about://components and verify the Optimization Guide On Device Model component is downloading. Refer to about://on-device-internals for additional details.

Prompt API Website HTML/JS Browser Operating System Apple Intelligence Gemini Nano Internet

Prompt API Part of Chrome’s Built-In AI initiative – Initiative by Google Chrome – Exploratory APIs for local experiments and use case determination – Downloads AI models into Google Chrome – Models are shared across origins – Uses native APIs directly (full performance) https://developer.chrome.com/docs/ai/built-in

Prompt API Standard Positions https://github.com/WebKit/standards-positions/issues/495 https://github.com/mozilla/standards-positions/issues/1213

Prompt API Polyfill Supports two backends: – Cloud backend: needs an API key (no longer local/offline-capable) – Local backend: uses Transformers.js (via WebGPU or Wasm) https://developer.chrome.com/docs/ai/prompt-api-polyfill

Built-in AI & WebMCP Die AI-Revolution direkt im Browser Prompt
API Easy Language Model https://www.npmjs.com/package/easy-language-model

Built-in AI & WebMCP Die AI-Revolution direkt im Browser Built-in
AI APIs TypeScript Definitions

Prompt API npm i -D @types/dom-chromium-ai add "dom-chromium-ai" to the types in tsconfig.app.json LAB #11

Local AI Models Add the following lines to inferPromptApi(): LAB #12 const systemPrompt = ` The user will ask questions about their todo list. Here's the user's todo list: ${JSON.stringify(this.todos())}`; const languageModel = await LanguageModel.create({ initialPrompts: [{ role: "system", content: systemPrompt }] }); const chunks = languageModel.promptStreaming(userPrompt); for await (const chunk of chunks) { yield chunk; }

Built-in AI & WebMCP Die AI-Revolution direkt im Browser Prompt
API Tool Use NEW! – Interface between the unstructured (natural language) and structured (programmatic) parts of the application – Backed by a JavaScript function – API shape described by a JSON Schema tools: [{ name: "getWeather", description: "Get the weather in a location.", inputSchema: { type: "object", properties: { location: { type: "string", description: "The city to check for the weather condition.", }, }, required: ["location"], }, async execute({ location }) { // do something }, }],

Local AI Models Alternatives: Ollama – Local runner for AI models – Offers a local server a website can connect to à allows sharing models across origins – Supported on Windows, Linux, macOS https://ollama.ai/

Local RAG Demo https://webml-demo.vercel.app/ https://github.com/jacoblee93/fully-local-pdf-chatbot/

Local AI Models Alternatives – Pre-trained, specialized, significantly smaller models beyond GenAI – JavaScript library to run Hugging Face transformers in the browser https://huggingface.co/docs/transformers.js

Realtime Models On-Device DEMO https://huggingface.co/spaces/webml-community/conversational-webgpu

Realtime Models On-device pipeline Whisper (STT) Silero (VAD) SmolLM21.7B (LLM) Kokoro (TTS)

Use Case Data Extraction Just transfer the 17.34 euros to me, my IBAN is DE02200505501015871393. I am with Hamburger Sparkasse (HASPDEHH). Nice, here is my address: Peter Müller, Rheinstr. 7, 04435 Schkeuditz

Idea protected readonly formGroup = this.fb.group({ firstName: [''], lastName: [''], Smart addressLine1: [''], Form addressLine2: [''], Filler city: [''], state: [''], (LLM) zip: [''], country: [''], }); Nice, here is my address: Peter Müller, Rheinstr. 7, 04435 Schkeuditz

Form Field “Try to determine the country based on the input.” “Insurance numbers always start with INS.”

Form Field (1/2) Add the following code to form.ts: protected readonly model = signal({ name: '', city: '', }); protected readonly form = form(this.model); async fillForm(value: string) {} LAB #13

Form Field (2/2) LAB #13 Add the following code to form.html: <input type="text" #fillInput> <button (click)="fillForm(fillInput.value)">Fill form</button> <form> <input placeholder="Name" [formField]="form.name"> <input placeholder="City" [formField]="form.city"> </form>

Prompt Generator Async Clipboard API Allows reading from/writing to the clipboard in an asynchronous manner Reading from the clipboard requires user consent first (privacy!) Supported by Chrome, Edge and Safari and Firefox

Async Clipboard API (1/2) LAB #14 Add the following code to form.ts: async paste() { const content = await navigator.clipboard.readText(); await this.fillForm(content); }

Async Clipboard API (2/2) Add the following code to form.html (after the “Fill form” button): <button (click)="paste()">Paste</button> LAB #14

Prompt Generator System message • The form has the following setup: { "name": "", "city": "" } User message Assistant message • I am Peter from Berlin • { "name": "Peter", "city": "Berlin" }

Prompt Generator Add the following code to the fillForm() method: LAB #15 const languageModel = await LanguageModel.create({ initialPrompts: [{ role: 'system', content: `Extract the information to a JSON object of this shape: ${JSON.stringify(this.model())}`, }], }); const result = await languageModel.prompt(value); console.log(result);

Prompt Generator (Structured Output) LAB #16 Add the following code to form.ts (fillForm() method) and remove the model from the system prompt: const result = await languageModel.prompt(value, { responseConstraint: { type: 'object', properties: { name: { type: 'string' }, city: { type: 'string' }, }, }, });

Prompt Parser Assistant message • { "name": "Peter", "city": "Berlin" }

Prompt Parser Add the following code to form.ts (fillForm() method): this.model.set(JSON.parse(result)); LAB #17

Prompt Parser Assistant message Parsing the assistant message as text/JSON/… Tool calling Specifying a well-defined interface via a JSON schema called by the LLM (safer, growing support) JSON Mode Structured Output

WebNN, Built-in AI, WebMCP What’s New in Web AI? Agentic
Browsers

WebMCP – Allows websites to register tools for use by browser or external AI agents – Joint effort by Microsoft and Google – In specification by the WebML CG – In Developer Trial since Chromium 146, expected Origin Trial in Chromium 149–156, expected Shipping in Chromium 157 (https://chromestatus.com/feature/5117755 740913664) https://github.com/webmachinelearning/webmcp

WebMCP API Imperative Declarative document.modelContext.registerTool({ "name": "start_game", "description": "Start a new game.", "inputSchema": {}, "execute": () => {}, }); <form id="reservationForm" toolname="book_table_le_petit_bistro" tooldescription=...> <input name="name" toolparamdescription="Customer's full name (min 2 chars)" /> </form>

WebMCP Only tested with Chrome Canary! LAB #18 1. Install WebMCP typings: npm i -D @mcp-b/webmcp-types 2. Add "@mcp-b/webmcp-types" to the types array in tsconfig.app.json 3. Go to about://flags, set the flag WebMCP for Testing to Enabled, and relaunch Chrome. 4. Download the WebMCP Model Context Tool Inspector extension from https://bit.ly/mcpext 5. Get a free Gemini API key from https://aistudio.google.com/ 6. Open the WebMCP Model Context Tool Inspector extension, click Set Gemini API key, and click OK.

WebMCP 7. Add this code to form.ts: LAB #18 constructor() { document.modelContext.registerTool({ name: 'fill-form', description: 'Fills in the form.', inputSchema: { type: 'object', properties: { name: { type: 'string', description: 'The name of the person' }, city: { type: 'string', description: 'The city of the person' }, }, }, execute: (result) => this.formGroup.setValue(result as any), }); }

WebMCP & Angular (22+) https://next.angular.dev/ai/webmcp

Summary Pros & Cons + Data does not leave the browser (privacy) + High availability (offline support) + Low latency + Stability (no external API changes) + Low cost – Lower quality – High system (RAM, GPU) and bandwidth requirements – Large model size, models cannot always be shared – Model initialization and inference are relatively slow – APIs are experimental

Summary – Cloud-based models remain the most powerful models – Due to their size and high system requirements, local generative AI models are currently rather interesting for very special scenarios (e.g., high privacy demands, offline availability) – Small, specialized models are an interesting alternative (if available) – Large language models are becoming more compact and efficient – Vendors are shipping AI models with their devices – Devices are becoming more powerful for running AI workloads – Experiment with the AI APIs and make your Angular App smarter!

Thank you for your kind attention! Christian Liebel @christianliebel [email protected]

Angular-Apps smarter machen mit Generativer KI:...

Angular-Apps smarter machen mit Generativer KI: Lokal und offlinefähig - Hands-on!

More Decks by Christian Liebel

Featured

Transcript