Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

Making Angular Apps Smarter with Generative AI:...

Making Angular Apps Smarter with Generative AI: Local and Offline-capable (Hands-on)

Generative AI is on everyone's lips: Adobe Photoshop allows objects in images to be exchanged by simply entering a prompt, and Microsoft Copilot has come to Office and Windows. With WebLLM and WebSD, we can now bring Generative AI to your Angular app: locally and offline-capable. We generate images from text input and add a chatbot to a to-do application. If you want to code along, please bring a powerful device with Windows or macOS, a current version of Chrome, Node.js, an editor of your choice, and at least 20 GB of free hard disk space. Ideally, this should be your private device, as group policies or company proxies tend to throw a spanner in the works.

Christian Liebel

November 15, 2024
Tweet

More Decks by Christian Liebel

Other Decks in Programming

Transcript

  1. Hello, it’s me. Making Angular Apps Smarter with Generative AI

    Local and Offline-capable (Hands-on) Christian Liebel X: @christianliebel Bluesky: @christianliebel.com Email: christian.liebel @thinktecture.com Angular, PWA & Generative AI Slides: thinktecture.com /christian-liebel
  2. 09:00–10:30 Block 1 10:30–11:00 Coffee Break 11:00–12:30 Block 2 12:30–13:30

    Lunch Break 13:30–15:00 Block 3 15:00–15:30 Coffee Break 15:30–16:30 Block 4 Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Timetable
  3. What to expect Focus on web app development Focus on

    Generative AI Up-to-date insights: the ML/AI field is evolving fast Live demos on real hardware Hands-on labs What not to expect Deep dive into AI specifics, RAG, model finetuning or training Stable libraries or specifications WebSD in Angular Polished workshop Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Expectations Huge downloads! High requirements! Things may break!
  4. (Workshop Edition) Making Angular Apps Smarter with Generative AI Local

    and Offline-capable (Hands-on) Demo Use Case DEMO
  5. Setup complete? (Node.js, Google Chrome, Editor, Git, macOS/Windows, 20 GB

    free disk space, 6 GB VRAM) Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Setup
  6. ng new genai-app ng add @angular/material Making Angular Apps Smarter

    with Generative AI Local and Offline-capable (Hands-on)
  7. git clone https://github.com/thinktecture/ijs-muc- 2024-genai.git cd ijs-muc-2024-genai npm i npm start

    -- --open Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Setup LAB
  8. Making Angular Apps Smarter with Generative AI Local and Offline-capable

    (Hands-on) Generative AI everywhere Source: https://www.apple.com/chde/apple-intelligence/
  9. Run locally on the user’s system Making Angular Apps Smarter

    with Generative AI Local and Offline-capable (Hands-on) Single-Page Applications Server- Logik Web API Push Service Web API DBs HTML, JS, CSS, Assets Webserver Webbrowser SPA Client- Logik View HTML/CSS View HTML/CSS View HTML/CSS HTTPS WebSockets HTTPS HTTPS
  10. Make SPAs offline-capable Making Angular Apps Smarter with Generative AI

    Local and Offline-capable (Hands-on) Progressive Web Apps Service Worker Internet Website HTML/JS Cache fetch
  11. Overview Making Angular Apps Smarter with Generative AI Local and

    Offline-capable (Hands-on) Generative AI Text OpenAI GPT Mistral … Speech OpenAI Whisper tortoise-tts … Images DALL·E Stable Diffusion … Audio/Music Musico Soundraw …
  12. Overview Making Angular Apps Smarter with Generative AI Local and

    Offline-capable (Hands-on) Generative AI Text OpenAI GPT Mistral … Speech OpenAI Whisper tortoise-tts … Images DALL·E Stable Diffusion … Audio/Music Musico Soundraw …
  13. Examples Making Angular Apps Smarter with Generative AI Local and

    Offline-capable (Hands-on) Generative AI Cloud Providers
  14. Drawbacks Making Angular Apps Smarter with Generative AI Local and

    Offline-capable (Hands-on) Generative AI Cloud Providers Require a (stable) internet connection Subject to network latency and server availability Data is transferred to the cloud service Require a subscription
  15. Can we run GenAI models locally? Making Angular Apps Smarter

    with Generative AI Local and Offline-capable (Hands-on)
  16. Large: Trained on lots of data Language: Process and generate

    text Models: Programs/neural networks Examples: – GPT (ChatGPT, Microsoft Copilot, …) – Gemini, Gemma (Google) – LLaMa (Meta AI) Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Large Language Models
  17. Token A meaningful unit of text (e.g., a word, a

    part of a word, a character). Context Window The maximum amount of tokens the model can process. Parameters/weights Internal variables learned during training, used to make predictions. Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Large Language Models
  18. Prompts serve as the universal interface Unstructured text conveying specific

    semantics Paradigm shift in software architecture Natural language becomes a first-class citizen Caveats Non-determinism and hallucination, prompt injections Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Large Language Models
  19. Size Comparison Model:Parameters Size phi3:3b 2.2 GB mistral:7b 4.1 GB

    llama3:8b 4.7 GB gemma2:9b 5.4 GB gemma2:27b 16 GB llama3:70b 40 GB Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Large Language Models
  20. On NPM Making Angular Apps Smarter with Generative AI Local

    and Offline-capable (Hands-on) WebLLM
  21. (1/3) In app.component.ts, add the following lines: protected readonly progress

    = signal(0); protected readonly ready = signal(false); protected engine?: MLCEngine; Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Downloading a model LAB
  22. (2/3) In app.component.ts (ngOnInit()), add the following lines: const model

    = 'Llama-3.2-3B-Instruct-q4f16_1-MLC'; this.engine = await CreateMLCEngine(model, { initProgressCallback: ({ progress }) => this.progress.set(progress) }); this.ready.set(true); Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Downloading a model LAB
  23. (3/3) In app.component.html, add the following lines: <div><progress [value]="progress()"></progress></div> <input

    type="text" #prompt> <button (click)="runPrompt(prompt.value)" [disabled]="!ready()"> Ask </button> Launch the app via npm start. The progress bar should begin to move. Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Downloading a model LAB
  24. Storing model files locally Making Angular Apps Smarter with Generative

    AI Local and Offline-capable (Hands-on) Cache API Internet Website HTML/JS Cache with model files Hugging Face Note: Due to the Same-Origin Policy, models cannot be shared across origins.
  25. Making Angular Apps Smarter with Generative AI Local and Offline-capable

    (Hands-on) WebAssembly (Wasm) – Bytecode for the web – Compile target for arbitrary languages – Can be faster than JavaScript – WebLLM uses a model- specific Wasm library to accelerate model computations
  26. Making Angular Apps Smarter with Generative AI Local and Offline-capable

    (Hands-on) WebGPU – Grants low-level access to the Graphics Processing Unit (GPU) – Near native performance for machine learning applications – Supported by Chromium-based browsers on Windows and macOS from version 113
  27. – Grants web apps access to the device’s CPU, GPU

    and Neural Processing Unit (NPU) – In specification by the WebML Working Group at W3C – Implementation in progress in Chromium (behind a flag) – Even better performance compared to WebGPU Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) WebNN Source: https://webmachinelearning.github.io/webnn-intro/ DEMO
  28. Making Angular Apps Smarter with Generative AI Local and Offline-capable

    (Hands-on) WebNN: near-native inference performance Source: Intel. Browser: Chrome Canary 118.0.5943.0, DUT: Dell/Linux/i7-1260P, single p-core, Workloads: MediaPipe solution models (FP32, batch=1)
  29. (1/3) In app.component.ts, add the following lines at the top

    of the class: protected readonly reply = signal(''); Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Model inference LAB
  30. (2/3) In the runPrompt() method, add the following code: await

    this.engine!.resetChat(); this.reply.set('…'); const messages: ChatCompletionMessageParam[] = [ { role: "user", content: value } ]; const reply = await this.engine!.chat.completions.create({ messages }); this.reply.set(reply.choices[0].message.content ?? ''); Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Model inference LAB
  31. (3/3) In app.component.html, add the following line: <pre>{{ reply() }}</pre>

    You should now be able to send prompts to the model and see the responses in the template. Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Model inference LAB
  32. npm run build Making Angular Apps Smarter with Generative AI

    Local and Offline-capable (Hands-on) LAB
  33. 1. In angular.json, increase the bundle size for the Angular

    project (property architect.build.configurations.production.budgets[0] .maximumError) to at least 5MB. 2. Then, run npm run build again. This time, the build should succeed. 3. If you stopped the development server, don’t forget to bring it back up again (npm start). Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Build issues LAB
  34. (1/2) In app.component.ts, add the following signal at the top:

    protected readonly todos = signal<Todo[]>([]); Add the following method: addTodo(text: string) { this.todos.update(todos => [...todos, { done: false, text }]); } Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Todo management LAB
  35. (2/2) In app.component.html, add the following lines to add todos

    from the UI: <input type="text" #input> <button (click)="addTodo(input.value)">Add</button> <ul> @for(todo of todos(); track $index) { <li>{{ todo.text }}</li> } </ul> Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Todo management LAB
  36. In app.component.ts, add the following function toggleTodo(index: number) { this.todos.update(todos

    => todos.map((todo, todoIndex) => todoIndex === index ? { ...todo, done: !todo.done } : todo)); } In app.component.html, add the following content to the <li> node: <input type="checkbox" [checked]="todo.done" (change)="toggleTodo($index)"> You should now be able to toggle the checkboxes. Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Todo management (extended) LAB
  37. Concept and limitations The todo data has to be converted

    into natural language. For the sake of simplicity, we will add all TODOs to the prompt. Remember: LLMs have a context window (Mistral-7B: 8K). If you need to chat with larger sets of text, refer to Retrieval Augmented Generation (RAG). These are the todos: * Wash clothes * Pet the dog * Take out the trash Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Chat with data
  38. System prompt Metaprompt that defines… – character – capabilities/limitations –

    output format – behavior – grounding data Hallucinations and prompt injections cannot be eliminated. You are a helpful assistant. Answer user questions on todos. Generate a valid JSON object. Avoid negative content. These are the user’s todos: … Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Chat with data
  39. Flow System message • The user has these todos: 1.

    … 2. … 3. … User message • How many todos do I have? Assistant message • You have three todos. Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Chat with data
  40. Using a system & user prompt Adjust the implementation in

    runPrompt() to include the system prompt: const systemPrompt = `Here's the user's todo list: ${this.todos().map(todo => `* ${todo.text} (${todo.done ? 'done' : 'not done'})`).join('\n')}`; const messages: ChatCompletionMessageParam[] = [ { role: "system", content: systemPrompt }, { role: "user", content: value } ]; Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Chat with data LAB
  41. Techniques – Providing examples (single shot, few shot, …) –

    Priming outputs – Specify output structure – Repeating instructions – Chain of thought – … Success also depends on the model. Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Prompt Engineering https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/advanced-prompt-engineering
  42. const systemPrompt = `You are a helpful assistant. The user

    will ask questions about their todo list. Briefly answer the questions. Don't try to make up an answer if you don't know it. Here's the user's todo list: ${this.todos().map(todo => `* ${todo.text} (this todo is ${todo.done ? 'done' : 'not done'})`).join('\n')} ${this.todos().length === 0 ? 'The list is empty, there are no todos.' : ''}`; Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Prompt Engineering LAB
  43. Alternatives Prompt Engineering Retrieval Augmented Generation Fine-tuning Custom model Making

    Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Prompt Engineering Effort
  44. Add the following line to the runPrompt() method: console.log(reply.usage); Ask

    a new question and check your console for performance statistics. Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Performance LAB
  45. Workshop Participants Device Tokens/s (Decode) MacBook i7 (2021) 2.70 MacBook

    M1 (1) 15.35 MacBook M1 (2) 14.52 MacBook M4 19.70 DELL i7 + Iris (1) 1.43 DELL i7 + Iris (2) 3.15 Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Performance
  46. Comparison 45 33 1200 0 200 400 600 800 1000

    1200 1400 WebLLM (Llama3-8b, M4) Azure OpenAI (gpt-4o-mini) Groq (Llama3-8b) Tokens/sec Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Performance WebLLM/Groq: Own tests (14.11.2024), OpenAI/Azure OpenAI: https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/provisioned-throughput (18.07.2024)
  47. Just transfer the 17.34 euros to me, my IBAN is

    DE02200505501015871393. I am with Hamburger Sparkasse (HASPDEHH). Data Extraction Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Use Case Nice, here is my address: Peter Müller, Rheinstr. 7, 04435 Schkeuditz
  48. Just transfer the 17.34 euros to me, my IBAN is

    DE02200505501015871393. I am with Hamburger Sparkasse (HASPDEHH). Data Extraction Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Use Case Nice, here is my address: Peter Müller, Rheinstr. 7, 04435 Schkeuditz
  49. protected readonly formGroup = this.fb.group({ firstName: [''], lastName: [''], addressLine1:

    [''], addressLine2: [''], city: [''], state: [''], zip: [''], country: [''], }); Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Idea Nice, here is my address: Peter Müller, Rheinstr. 7, 04435 Schkeuditz Smart Form Filler (LLM)
  50. Form Field Prompt Generator Model Backend Response Parser Making Angular

    Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Architecture
  51. Making Angular Apps Smarter with Generative AI Local and Offline-capable

    (Hands-on) Form Field “Try to determine the country based on the input.” “If present, put the department in this field.”
  52. (1/2) Add the following code to app.component.ts: private fb =

    inject(NonNullableFormBuilder); protected formGroup = this.fb.group({ name: '', city: '', }); async fillForm(value: string) {} Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Form Field LAB
  53. (2/2) Add the following code to app.component.html: <input type="text" #form>

    <button (click)="fillForm(form.value)" [disabled]="!ready()">Fill form</button> <form [formGroup]="formGroup"> <input placeholder="Name" formControlName="name"> <input placeholder="City" formControlName="city"> </form> Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Form Field LAB
  54. Async Clipboard API Allows reading from/writing to the clipboard in

    an asynchronous manner Reading from the clipboard requires user consent first (privacy!) Supported by Chrome, Edge and Safari and Firefox Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Prompt Generator
  55. (1/2) Add the following code to app.component.ts: async paste() {

    const content = await navigator.clipboard.readText(); await this.fillForm(content); } Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Async Clipboard API LAB
  56. (2/2) Add the following code to app.component.html (after the “Fill

    form” button): <button (click)="paste()" [disabled]="!ready()">Paste</button> Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Async Clipboard API LAB
  57. System message • The form has the following setup: {

    "name": "", "city": "" } User message • I am Peter from Berlin Assistant message • { "name": "Peter", "city": "Berlin" } Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Prompt Generator
  58. Add the following code to app.component.ts (fillForm() method): const messages:

    ChatCompletionMessageParam[] = [{ role: "system", content: `Extract the information to a JSON object of this shape: ${JSON.stringify(this.formGroup.value)} Do not add any other text.` }, { role: "user", content: value }]; Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Prompt Generator LAB
  59. Making Angular Apps Smarter with Generative AI Local and Offline-capable

    (Hands-on) Model Backend – Cloud – WebLLM – Prompt API – Ollama (on-premise) – …
  60. Add the following code to app.component.ts (fillForm() method): const reply

    = await this.engine!.chat.completions.create({ messages }); Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Model Generator LAB
  61. Making Angular Apps Smarter with Generative AI Local and Offline-capable

    (Hands-on) Prompt Parser Assistant message • { "name": "Peter", "city": "Berlin" }
  62. Add the following code to app.component.ts (fillForm() method): this.formGroup.setValue(JSON.parse(reply.choices[0] .message.content

    ?? '')); Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Prompt Parser LAB
  63. Assistant message Parsing the assistant message as text/JSON/… Tool calling

    Specifying a well-defined interface via a JSON schema called by the LLM (safer, growing support) Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Prompt Parser
  64. https://www.google.com/chrome/canary/ about://flags Enables optimization guide on device à EnabledBypassPerfRequirement Prompt

    API for Gemini Nano à Enabled await ai.languageModel.create(); about://components Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Prompt API LAB
  65. Making Angular Apps Smarter with Generative AI Local and Offline-capable

    (Hands-on) Prompt API Operating System Website HTML/JS Browser Internet Apple Intelligence Gemini Nano
  66. Part of Chrome’s Built-In AI initiative – Exploratory API for

    local experiments and use case determination – Downloads Gemini Nano into Google Chrome – Model can be shared across origins – Uses native APIs directly – Fine-tuning API might follow in the future Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Prompt API https://developer.chrome.com/docs/ai/built-in
  67. npm i -D @types/dom-chromium-ai add @types/dom-chromium-ai to the types in

    tsconfig.app.json Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Prompt API LAB
  68. Adjust the implementations of runPrompt()/fillForm(): const session = await window.ai.languageModel.create({

    systemPrompt }); const reply = await session.prompt(value); // runPrompt(): this.reply.set(reply); // fillForm(): this.formGroup.setValue(JSON.parse(reply)); Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Prompt API LAB
  69. Alternatives: Ollama – Local runner for AI models – Offers

    a local server a website can connect to à allows sharing models across origins – Supported on macOS and Linux (Windows in Preview) https://webml-demo.vercel.app/ https://ollama.ai/ Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Local AI Models
  70. Alternatives: Hugging Face Transformers Pre-trained, specialized, significantly smaller models beyond

    GenAI Examples: – Text generation – Image classification – Translation – Speech recognition – Image-to-text Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Local AI Models
  71. Alternatives: Transformers.js – Pre-trained, specialized, significantly smaller models beyond GenAI

    – JavaScript library to run Hugging Face transformers in the browser – Supports most of the models https://huggingface.co/docs/transformers.js Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Local AI Models
  72. Pros & Cons + Data does not leave the browser

    (privacy) + High availability (offline support) + Low latency + Stability (no external API changes) + Low cost – Lower quality – High system (RAM, GPU) and bandwidth requirements – Large model size, models cannot always be shared – Model initialization and inference are relatively slow – APIs are experimental Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Local AI Models
  73. – Cloud-based models remain the most powerful models – Due

    to their size and high system requirements, local generative AI models are currently rather interesting for very special scenarios (e.g., high privacy demands, offline availability) – Small, specialized models are an interesting alternative (if available) – Large language models are becoming more compact and efficient – Vendors start shipping AI models with their devices – Devices are becoming more powerful for running AI tasks – Experiment with the AI APIs and make your Angular App smarter! Making Angular Apps Smarter with Generative AI Local and Offline-capable (Hands-on) Summary