The AI Revolution in the Browser? Making Single-Page Apps Smarter

Slide 1

Slide 1 text

The AI Revolution in the Browser? Making Single-Page Apps Smarter Christian Liebel @christianliebel Consultant

Slide 2

Slide 2 text

Hello, it’s me. Intelligent Forms Spice up your Angular Forms with AI & LLMs Christian Liebel X: @christianliebel Bluesky: @christianliebel.com Email: christian.liebel @thinktecture.com Angular, PWA & Generative AI Slides: thinktecture.com /christian-liebel

Slide 3

Slide 3 text

The AI Revolution in the Browser? Making Single-Page Apps Smarter Generative AI everywhere Source: https://www.apple.com/chde/apple-intelligence/

Slide 4

Slide 4 text

Overview The AI Revolution in the Browser? Making Single-Page Apps Smarter Generative AI Text OpenAI GPT Mistral … Speech OpenAI Whisper tortoise-tts … Images DALL·E Stable Diffusion … Audio/Music Musico Soundraw …

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Examples The AI Revolution in the Browser? Making Single-Page Apps Smarter Generative AI Cloud Providers

Slide 7

Slide 7 text

Drawbacks The AI Revolution in the Browser? Making Single-Page Apps Smarter Generative AI Cloud Providers Require a (stable) internet connection Subject to network latency and server availability Data is transferred to the cloud service Require a subscription

Slide 8

Slide 8 text

Can we run GenAI models locally? The AI Revolution in the Browser? Making Single-Page Apps Smarter

Slide 9

Slide 9 text

Size Comparison Model:Parameters Size phi3:3b 2.2 GB mistral:7b 4.1 GB llama3:8b 4.7 GB gemma2:9b 5.4 GB gemma2:27b 16 GB llama3:70b 40 GB The AI Revolution in the Browser? Making Single-Page Apps Smarter Large Language Models

Slide 10

Slide 10 text

https://webllm.mlc.ai/ The AI Revolution in the Browser? Making Single-Page Apps Smarter WebLLM DEMO

Slide 11

Slide 11 text

On NPM The AI Revolution in the Browser? Making Single-Page Apps Smarter WebLLM

Slide 12

Slide 12 text

Impact on Software Architecture The AI Revolution in the Browser? Making Single-Page Apps Smarter Large Language Models Prompts serve as the universal interface for users and developers Paradigm shift Natural language becomes a first-class citizen Caveats Non-determinism, hallucinations, prompt injection

Slide 13

Slide 13 text

Demo The AI Revolution in the Browser? Making Single-Page Apps Smarter WebLLM DEMO

Slide 14

Slide 14 text

Storing model files locally The AI Revolution in the Browser? Making Single-Page Apps Smarter Cache API Internet Website HTML/JS Cache with model files Hugging Face Note: Due to the Same-Origin Policy, models cannot be shared across origins.

Slide 15

Slide 15 text

The AI Revolution in the Browser? Making Single-Page Apps Smarter WebAssembly (Wasm) – Bytecode for the web – Compile target for arbitrary languages – Can be faster than JavaScript – WebLLM uses a model- specific Wasm library to accelerate model computations

Slide 16

Slide 16 text

The AI Revolution in the Browser? Making Single-Page Apps Smarter WebGPU – Grants low-level access to the Graphics Processing Unit (GPU) – Near native performance for machine learning applications – Supported by Chromium-based browsers on Windows and macOS from version 113

Slide 17

Slide 17 text

– Grants web apps access to the device’s CPU, GPU and Neural Processing Unit (NPU) – In specification by the WebML Working Group at W3C – Implementation in progress in Chromium (behind a flag) – Even better performance compared to WebGPU The AI Revolution in the Browser? Making Single-Page Apps Smarter WebNN Source: https://webmachinelearning.github.io/webnn-intro/ DEMO

Slide 18

Slide 18 text

The AI Revolution in the Browser? Making Single-Page Apps Smarter WebNN: Near-native inference performance Source: Intel. Browser: Chrome Canary 118.0.5943.0, DUT: Dell/Linux/i7-1260P, single p-core, Workloads: MediaPipe solution models (FP32, batch=1)

Slide 19

Slide 19 text

The AI Revolution in the Browser? Making Single-Page Apps Smarter Prompt API Operating System Website HTML/JS Browser Internet Apple Intelligence Gemini Nano

Slide 20

Slide 20 text

Part of Chrome’s Built-In AI initiative – Exploratory API for local experiments and use case determination – Downloads Gemini Nano into Google Chrome – Model is shared across origins – Uses native APIs directly – Related APIs: Translation API, Writing Assistance APIs The AI Revolution in the Browser? Making Single-Page Apps Smarter Prompt API https://developer.chrome.com/docs/ai/built-in

Slide 21

Slide 21 text

Demo The AI Revolution in the Browser? Making Single-Page Apps Smarter Prompt API DEMO

Slide 22

Slide 22 text

Comparison 45 33 1200 0 200 400 600 800 1000 1200 1400 WebLLM (Llama3-8b, M4) Azure OpenAI (gpt-4o-mini) Groq (Llama3-8b) Tokens/sec The AI Revolution in the Browser? Making Single-Page Apps Smarter Performance WebLLM/Groq: Own tests (14.11.2024), OpenAI/Azure OpenAI: https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/provisioned-throughput (18.07.2024)

Slide 23

Slide 23 text

– Open-source text-to-image model – Generates 512x512px images from a prompt – WebSD: special version of Stable Diffusion for the web (2 GB in size) – No npm package this time The AI Revolution in the Browser? Making Single-Page Apps Smarter Stable Diffusion Prompt: A guinea pig eating a watermelon

Slide 24

Slide 24 text

https://websd.mlc.ai/ The AI Revolution in the Browser? Making Single-Page Apps Smarter Web Stable Diffusion DEMO

Slide 25

Slide 25 text

Pros & Cons + Data does not leave the browser (privacy) + High availability (offline support) + Low latency + Stability (no external API changes) + Low cost – Lower quality – High system (RAM, GPU) and bandwidth requirements – Large model size, models cannot always be shared – Model initialization and inference are relatively slow – APIs are experimental The AI Revolution in the Browser? Making Single-Page Apps Smarter Local AI Models

Slide 26

Slide 26 text

Transformers.js – Pre-trained, specialized, significantly smaller models beyond GenAI – JavaScript library to run Hugging Face transformers in the browser – Supports most of the models https://xenova.github.io/transformers.js/ The AI Revolution in the Browser? Making Single-Page Apps Smarter Alternatives

Slide 27

Slide 27 text

– Cloud-based models remain the most powerful models – Due to their size and high system requirements, local generative AI models are currently rather interesting for very special scenarios (e.g., high privacy demands, offline availability) – Small, specialized models are an interesting alternative (if available) – Large language models are becoming more compact and efficient – Vendors start shipping AI models with their devices – Devices are becoming more powerful for running AI tasks – Experiment with the AI APIs and make your Angular App smarter! The AI Revolution in the Browser? Making Single-Page Apps Smarter Summary

Slide 28

Slide 28 text

Thank you for your kind attention! Christian Liebel @christianliebel christian.liebel@thinktecture.com