Making Angular Apps Smarter with Generative AI: Local and Offline-capable (Hands-on)

Generative AI is on everyone's lips: Adobe Photoshop allows objects in images to be exchanged by simply entering a prompt, and Microsoft Copilot has come to Office and Windows. With WebLLM and WebSD, we can now bring Generative AI to your Angular app: locally and offline-capable. We generate images from text input and add a chatbot to a to-do application. If you want to code along, please bring a powerful device with Windows or macOS, a current version of Chrome, Node.js, an editor of your choice, and at least 20 GB of free hard disk space. Ideally, this should be your private device, as group policies or company proxies tend to throw a spanner in the works.

Transcript

1. Christian Liebel @christianliebel Consultant Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on)
2. Hello, it’s me. Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Christian Liebel X: @christianliebel Bluesky: @christianliebel.com Email: christian.liebel @thinktecture.com Angular, PWA & Generative AI Slides: thinktecture.com /christian-liebel
3. 09:00–10:30 Block 1 10:30–11:00 Coffee Break 11:00–12:30 Block 2 12:30–13:30 Lunch Break 13:30–15:00 Block 3 15:00–15:30 Coffee Break 15:30–16:30 Block 4 Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Timetable
4. What to expect Focus on web app development Focus on Generative AI Up-to-date insights: the ML/AI field is evolving fast Live demos on real hardware Hands-on labs What not to expect Deep dive into AI specifics, RAG, model finetuning or training Stable libraries or specifications WebSD in Angular Polished workshop Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Expectations Huge downloads! High requirements! Things may break!
5. (Workshop Edition) Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Demo Use Case DEMO
6. Setup complete? (Node.js, Google Chrome, Editor, Git, macOS/Windows, 20 GB free disk space, 6 GB VRAM) Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Setup
7. https://webgpureport.org/ Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Setup
8. ng new genai-app ng add @angular/material Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on)
9. git clone https://github.com/thinktecture/ijs-muc- 2024-genai.git cd ijs-muc-2024-genai npm i npm start -- --open Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Setup LAB
10. Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Generative AI everywhere Source: https://www.apple.com/chde/apple-intelligence/
11. Run locally on the user’s system Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Single-Page Applications Server- Logik Web API Push Service Web API DBs HTML, JS, CSS, Assets Webserver Webbrowser SPA Client- Logik View HTML/CSS View HTML/CSS View HTML/CSS HTTPS WebSockets HTTPS HTTPS
12. Make SPAs offline-capable Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Progressive Web Apps Service Worker Internet Website HTML/JS Cache fetch
13. Overview Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Generative AI Text OpenAI GPT Mistral … Speech OpenAI Whisper tortoise-tts … Images DALL·E Stable Diffusion … Audio/Music Musico Soundraw …
14. Overview Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Generative AI Text OpenAI GPT Mistral … Speech OpenAI Whisper tortoise-tts … Images DALL·E Stable Diffusion … Audio/Music Musico Soundraw …
15. Examples Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Generative AI Cloud Providers
16. Drawbacks Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Generative AI Cloud Providers Require a (stable) internet connection Subject to network latency and server availability Data is transferred to the cloud service Require a subscription
17. Can we run GenAI models locally? Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on)
18. Large: Trained on lots of data Language: Process and generate text Models: Programs/neural networks Examples: – GPT (ChatGPT, Microsoft Copilot, …) – Gemini, Gemma (Google) – LLaMa (Meta AI) Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Large Language Models
19. Token A meaningful unit of text (e.g., a word, a part of a word, a character). Context Window The maximum amount of tokens the model can process. Parameters/weights Internal variables learned during training, used to make predictions. Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Large Language Models
20. Prompts serve as the universal interface Unstructured text conveying specific semantics Paradigm shift in software architecture Natural language becomes a first-class citizen Caveats Non-determinism and hallucination, prompt injections Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Large Language Models
21. Size Comparison Model:Parameters Size phi3:3b 2.2 GB mistral:7b 4.1 GB llama3:8b 4.7 GB gemma2:9b 5.4 GB gemma2:27b 16 GB llama3:70b 40 GB Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Large Language Models
22. https://webllm.mlc.ai/ Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) WebLLM DEMO
23. On NPM Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) WebLLM
24. npm i @mlc-ai/web-llm Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) LAB
25. (1/3) In app.component.ts, add the following lines: protected readonly progress = signal(0); protected readonly ready = signal(false); protected engine?: MLCEngine; Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Downloading a model LAB
26. (2/3) In app.component.ts (ngOnInit()), add the following lines: const model = 'Llama-3.2-3B-Instruct-q4f16_1-MLC'; this.engine = await CreateMLCEngine(model, { initProgressCallback: ({ progress }) => this.progress.set(progress) }); this.ready.set(true); Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Downloading a model LAB
27. (3/3) In app.component.html, add the following lines: <div><progress [value]="progress()"></progress></div> <input type="text" #prompt> <button (click)="runPrompt(prompt.value)" [disabled]="!ready()"> Ask </button> Launch the app via npm start. The progress bar should begin to move. Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Downloading a model LAB
28. Storing model files locally Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Cache API Internet Website HTML/JS Cache with model files Hugging Face Note: Due to the Same-Origin Policy, models cannot be shared across origins.
29. Parameter cache Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Cache API
30. Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) WebAssembly (Wasm) – Bytecode for the web – Compile target for arbitrary languages – Can be faster than JavaScript – WebLLM uses a model- specific Wasm library to accelerate model computations
31. Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) WebGPU – Grants low-level access to the Graphics Processing Unit (GPU) – Near native performance for machine learning applications – Supported by Chromium-based browsers on Windows and macOS from version 113
32. – Grants web apps access to the device’s CPU, GPU and Neural Processing Unit (NPU) – In specification by the WebML Working Group at W3C – Implementation in progress in Chromium (behind a flag) – Even better performance compared to WebGPU Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) WebNN Source: https://webmachinelearning.github.io/webnn-intro/ DEMO
33. Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) WebNN: near-native inference performance Source: Intel. Browser: Chrome Canary 118.0.5943.0, DUT: Dell/Linux/i7-1260P, single p-core, Workloads: MediaPipe solution models (FP32, batch=1)
34. (1/3) In app.component.ts, add the following lines at the top of the class: protected readonly reply = signal(''); Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Model inference LAB
35. (2/3) In the runPrompt() method, add the following code: await this.engine!.resetChat(); this.reply.set('…'); const messages: ChatCompletionMessageParam[] = [ { role: "user", content: value } ]; const reply = await this.engine!.chat.completions.create({ messages }); this.reply.set(reply.choices[0].message.content ?? ''); Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Model inference LAB
36. (3/3) In app.component.html, add the following line: <pre>{{ reply() }}</pre> You should now be able to send prompts to the model and see the responses in the template. Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Model inference LAB
37. npm run build Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) LAB
38. 1. In angular.json, increase the bundle size for the Angular project (property architect.build.configurations.production.budgets[0] .maximumError) to at least 5MB. 2. Then, run npm run build again. This time, the build should succeed. 3. If you stopped the development server, don’t forget to bring it back up again (npm start). Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Build issues LAB
39. (1/2) In app.component.ts, add the following signal at the top: protected readonly todos = signal<Todo[]>([]); Add the following method: addTodo(text: string) { this.todos.update(todos => [...todos, { done: false, text }]); } Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Todo management LAB
40. (2/2) In app.component.html, add the following lines to add todos from the UI: <input type="text" #input> <button (click)="addTodo(input.value)">Add</button> <ul> @for(todo of todos(); track $index) { <li>{{ todo.text }}</li> } </ul> Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Todo management LAB
41. In app.component.ts, add the following function toggleTodo(index: number) { this.todos.update(todos => todos.map((todo, todoIndex) => todoIndex === index ? { ...todo, done: !todo.done } : todo)); } In app.component.html, add the following content to the <li> node: <input type="checkbox" [checked]="todo.done" (change)="toggleTodo($index)"> You should now be able to toggle the checkboxes. Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Todo management (extended) LAB
42. Concept and limitations The todo data has to be converted into natural language. For the sake of simplicity, we will add all TODOs to the prompt. Remember: LLMs have a context window (Mistral-7B: 8K). If you need to chat with larger sets of text, refer to Retrieval Augmented Generation (RAG). These are the todos: * Wash clothes * Pet the dog * Take out the trash Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Chat with data
43. System prompt Metaprompt that defines… – character – capabilities/limitations – output format – behavior – grounding data Hallucinations and prompt injections cannot be eliminated. You are a helpful assistant. Answer user questions on todos. Generate a valid JSON object. Avoid negative content. These are the user’s todos: … Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Chat with data
44. Flow System message • The user has these todos: 1. … 2. … 3. … User message • How many todos do I have? Assistant message • You have three todos. Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Chat with data
45. Using a system & user prompt Adjust the implementation in runPrompt() to include the system prompt: const systemPrompt = `Here's the user's todo list: ${this.todos().map(todo => `* ${todo.text} (${todo.done ? 'done' : 'not done'})`).join('\n')}`; const messages: ChatCompletionMessageParam[] = [ { role: "system", content: systemPrompt }, { role: "user", content: value } ]; Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Chat with data LAB
46. Techniques – Providing examples (single shot, few shot, …) – Priming outputs – Specify output structure – Repeating instructions – Chain of thought – … Success also depends on the model. Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Prompt Engineering https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/advanced-prompt-engineering
47. const systemPrompt = `You are a helpful assistant. The user will ask questions about their todo list. Briefly answer the questions. Don't try to make up an answer if you don't know it. Here's the user's todo list: ${this.todos().map(todo => `* ${todo.text} (this todo is ${todo.done ? 'done' : 'not done'})`).join('\n')} ${this.todos().length === 0 ? 'The list is empty, there are no todos.' : ''}`; Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Prompt Engineering LAB
48. Alternatives Prompt Engineering Retrieval Augmented Generation Fine-tuning Custom model Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Prompt Engineering Effort
49. Add the following line to the runPrompt() method: console.log(reply.usage); Ask a new question and check your console for performance statistics. Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Performance LAB
50. Workshop Participants Device Tokens/s (Decode) MacBook i7 (2021) 2.70 MacBook M1 (1) 15.35 MacBook M1 (2) 14.52 MacBook M4 19.70 DELL i7 + Iris (1) 1.43 DELL i7 + Iris (2) 3.15 Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Performance
51. Comparison 45 33 1200 0 200 400 600 800 1000 1200 1400 WebLLM (Llama3-8b, M4) Azure OpenAI (gpt-4o-mini) Groq (Llama3-8b) Tokens/sec Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Performance WebLLM/Groq: Own tests (14.11.2024), OpenAI/Azure OpenAI: https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/provisioned-throughput (18.07.2024)
52. Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) DEMO
53. Just transfer the 17.34 euros to me, my IBAN is DE02200505501015871393. I am with Hamburger Sparkasse (HASPDEHH). Data Extraction Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Use Case Nice, here is my address: Peter Müller, Rheinstr. 7, 04435 Schkeuditz
54. Just transfer the 17.34 euros to me, my IBAN is DE02200505501015871393. I am with Hamburger Sparkasse (HASPDEHH). Data Extraction Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Use Case Nice, here is my address: Peter Müller, Rheinstr. 7, 04435 Schkeuditz
55. protected readonly formGroup = this.fb.group({ firstName: [''], lastName: [''], addressLine1: [''], addressLine2: [''], city: [''], state: [''], zip: [''], country: [''], }); Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Idea Nice, here is my address: Peter Müller, Rheinstr. 7, 04435 Schkeuditz Smart Form Filler (LLM)
56. Form Field Prompt Generator Model Backend Response Parser Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Architecture
57. Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Form Field “Try to determine the country based on the input.” “If present, put the department in this field.”
58. (1/2) Add the following code to app.component.ts: private fb = inject(NonNullableFormBuilder); protected formGroup = this.fb.group({ name: '', city: '', }); async fillForm(value: string) {} Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Form Field LAB
59. (2/2) Add the following code to app.component.html: <input type="text" #form> <button (click)="fillForm(form.value)" [disabled]="!ready()">Fill form</button> <form [formGroup]="formGroup"> <input placeholder="Name" formControlName="name"> <input placeholder="City" formControlName="city"> </form> Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Form Field LAB
60. Async Clipboard API Allows reading from/writing to the clipboard in an asynchronous manner Reading from the clipboard requires user consent first (privacy!) Supported by Chrome, Edge and Safari and Firefox Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Prompt Generator
61. (1/2) Add the following code to app.component.ts: async paste() { const content = await navigator.clipboard.readText(); await this.fillForm(content); } Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Async Clipboard API LAB
62. (2/2) Add the following code to app.component.html (after the “Fill form” button): <button (click)="paste()" [disabled]="!ready()">Paste</button> Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Async Clipboard API LAB
63. System message • The form has the following setup: { "name": "", "city": "" } User message • I am Peter from Berlin Assistant message • { "name": "Peter", "city": "Berlin" } Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Prompt Generator
64. Add the following code to app.component.ts (fillForm() method): const messages: ChatCompletionMessageParam[] = [{ role: "system", content: `Extract the information to a JSON object of this shape: ${JSON.stringify(this.formGroup.value)} Do not add any other text.` }, { role: "user", content: value }]; Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Prompt Generator LAB
65. Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Model Backend – Cloud – WebLLM – Prompt API – Ollama (on-premise) – …
66. Add the following code to app.component.ts (fillForm() method): const reply = await this.engine!.chat.completions.create({ messages }); Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Model Generator LAB
67. Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Prompt Parser Assistant message • { "name": "Peter", "city": "Berlin" }
68. Add the following code to app.component.ts (fillForm() method): this.formGroup.setValue(JSON.parse(reply.choices[0] .message.content ?? '')); Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Prompt Parser LAB
69. Assistant message Parsing the assistant message as text/JSON/… Tool calling Specifying a well-defined interface via a JSON schema called by the LLM (safer, growing support) Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Prompt Parser
70. https://www.google.com/chrome/canary/ about://flags Enables optimization guide on device à EnabledBypassPerfRequirement Prompt API for Gemini Nano à Enabled await ai.languageModel.create(); about://components Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Prompt API LAB
71. Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Prompt API Operating System Website HTML/JS Browser Internet Apple Intelligence Gemini Nano
72. Part of Chrome’s Built-In AI initiative – Exploratory API for local experiments and use case determination – Downloads Gemini Nano into Google Chrome – Model can be shared across origins – Uses native APIs directly – Fine-tuning API might follow in the future Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Prompt API https://developer.chrome.com/docs/ai/built-in
73. npm i -D @types/dom-chromium-ai add @types/dom-chromium-ai to the types in tsconfig.app.json Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Prompt API LAB
74. Adjust the implementations of runPrompt()/fillForm(): const session = await window.ai.languageModel.create({ systemPrompt }); const reply = await session.prompt(value); // runPrompt(): this.reply.set(reply); // fillForm(): this.formGroup.setValue(JSON.parse(reply)); Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Prompt API LAB
75. Alternatives: Ollama – Local runner for AI models – Offers a local server a website can connect to à allows sharing models across origins – Supported on macOS and Linux (Windows in Preview) https://webml-demo.vercel.app/ https://ollama.ai/ Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Local AI Models
76. Alternatives: Hugging Face Transformers Pre-trained, specialized, significantly smaller models beyond GenAI Examples: – Text generation – Image classification – Translation – Speech recognition – Image-to-text Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Local AI Models
77. Alternatives: Transformers.js – Pre-trained, specialized, significantly smaller models beyond GenAI – JavaScript library to run Hugging Face transformers in the browser – Supports most of the models https://huggingface.co/docs/transformers.js Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Local AI Models
78. Pros & Cons + Data does not leave the browser (privacy) + High availability (offline support) + Low latency + Stability (no external API changes) + Low cost – Lower quality – High system (RAM, GPU) and bandwidth requirements – Large model size, models cannot always be shared – Model initialization and inference are relatively slow – APIs are experimental Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Local AI Models
79. – Cloud-based models remain the most powerful models – Due to their size and high system requirements, local generative AI models are currently rather interesting for very special scenarios (e.g., high privacy demands, offline availability) – Small, specialized models are an interesting alternative (if available) – Large language models are becoming more compact and efficient – Vendors start shipping AI models with their devices – Devices are becoming more powerful for running AI tasks – Experiment with the AI APIs and make your Angular App smarter! Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Summary
80. Thank you for your kind attention! Christian Liebel @christianliebel christian.liebel@thinktecture.com