Lock in $30 Savings on PRO—Offer Ends Soon! ⏳

Generative AI power on the web: making web apps...

Generative AI power on the web: making web apps smarter with WebGPU and WebNN

The surge of AI is evident, with tools like Generative Fill in Adobe Photoshop and Microsoft Copilot seamlessly integrated into Office and Windows. Developers are now exploring on-device AI execution, challenging the prevailing cloud-based approach. This paradigm shift offers advantages such as offline accessibility, data privacy, and cost-free execution, while considering device capabilities and initial download times. Large Language Models and Stable Diffusion models already run in web browsers thanks to the WebGPU API. The upcoming Web Neural Network API (WebNN) by W3C will boost the performance of AI model execution even more, providing web apps access to OS machine learning APIs. In this session with Christian Liebel, Thinktecture's W3C representative, explore diverse approaches, use cases, and the future of AI integration in web development.

Christian Liebel

July 19, 2024
Tweet

More Decks by Christian Liebel

Other Decks in Programming

Transcript

  1. Generative AI power on the web Making web apps smarter

    with WebGPU and WebNN Christian Liebel @christianliebel Consultant
  2. Generative AI power on the web Making web apps smarter

    with WebGPU and WebNN Generative AI everywhere
  3. Speech OpenAI Whisper tortoise-tts … Overview Generative AI power on

    the web Making web apps smarter with WebGPU and WebNN Generative AI Images Midjourney DALL·E Stable Diffusion … Audio/Music Musico Soundraw … Text OpenAI GPT LLaMa Vicuna …
  4. Speech OpenAI Whisper tortoise-tts … Overview Generative AI power on

    the web Making web apps smarter with WebGPU and WebNN Generative AI Images Midjourney DALL·E Stable Diffusion … Audio/Music Musico Soundraw … Text OpenAI GPT LLaMa Vicuna …
  5. Examples Generative AI power on the web Making web apps

    smarter with WebGPU and WebNN Generative AI Cloud Providers
  6. Drawbacks – Require an active internet connection – Affected by

    network latency and server availability – Data is transferred to the cloud service – Require a subscription à Can we run models locally? Generative AI power on the web Making web apps smarter with WebGPU and WebNN Generative AI Cloud Providers
  7. Size Comparison Model:Parameters Size phi3:3b 2.2 GB mistral:7b 4.1 GB

    llama3:8b 4.7 GB gemma2:9b 5.4 GB gemma2:27b 16 GB llama3:70b 40 GB Generative AI power on the web Making web apps smarter with WebGPU and WebNN Large Language Models
  8. On NPM Generative AI power on the web Making web

    apps smarter with WebGPU and WebNN WebLLM
  9. Storing model files locally Generative AI power on the web

    Making web apps smarter with WebGPU and WebNN Cache API Internet Website HTML/JS Cache with model files Hugging Face
  10. Parameter cache Generative AI power on the web Making web

    apps smarter with WebGPU and WebNN Cache API
  11. Generative AI power on the web Making web apps smarter

    with WebGPU and WebNN WebAssembly (Wasm) Bytecode for the web Compile target for arbitrary languages Can be faster than JavaScript WebLLM needs the model and a Wasm library to accelerate model computations
  12. Generative AI power on the web Making web apps smarter

    with WebGPU and WebNN WebGPU Grants low-level access to the Graphics Processing Unit (GPU) Near native performance for machine learning applications Supported by Chromium-based browsers on Windows and macOS from version 113
  13. Grants web applications access to the Neural Processing Unit (NPU)

    of the system via platform-specific machine learning services (e.g., ML Compute on macOS/iOS, DirectML on Windows, …) Even better performance compared to WebGPU Currently in specification by the WebML Working Group at W3C Implementation in progress for Chromium-based browsers https://webmachinelearning.github.io/webnn-intro/ Generative AI power on the web Making web apps smarter with WebGPU and WebNN Outlook: WebNN
  14. Generative AI power on the web Making web apps smarter

    with WebGPU and WebNN WebNN: near-native inference performance Source: Intel. Browser: Chrome Canary 118.0.5943.0, DUT: Dell/Linux/i7-1260P, single p-core, Workloads: MediaPipe solution models (FP32, batch=1)
  15. Caveats – Due to the Same-Origin Policy, models can’t be

    shared across origins (i.e., https://example.org cannot access https://test.example.org). – Downloading LLMs multiple times leads to very high storage consumption. Generative AI power on the web Making web apps smarter with WebGPU and WebNN WebLLM
  16. Generative AI power on the web Making web apps smarter

    with WebGPU and WebNN Prompt API Operating System Website HTML/JS Browser Internet Apple Intelligence Gemini Nano
  17. Part of Chrome’s Built-In AI initiative – Exploratory API for

    local experiments and use case determination – Downloads Gemini Nano into Google Chrome – Model can be shared across origins – Uses native APIs directly – Fine-tuning API might follow in the future Generative AI power on the web Making web apps smarter with WebGPU and WebNN Prompt API https://developer.chrome.com/docs/ai/built-in
  18. First Glance Generative AI power on the web Making web

    apps smarter with WebGPU and WebNN Prompt API
  19. Demo: Smart Form Filler Generative AI power on the web

    Making web apps smarter with WebGPU and WebNN Prompt API DEMO
  20. Comparison 22,98 33,96 19,08 38,75 564,63 0 100 200 300

    400 500 600 WebLLM (Mistral-7b, M1) WebLLM (Mistral-7b, M3) OpenAI (GPT-4) Azure OpenAI (GPT-4) Groq (Mixtral-8x7b) Tokens/sec Generative AI power on the web Making web apps smarter with WebGPU and WebNN Performance WebLLM/Groq: Own tests (23.03.2024), OpenAI/Azure OpenAI: https://mcplusa.com/comparing-performance-of-openai-gpt-4-and-microsoft-azure-gpt-4/ (31.08.2023)
  21. Text-to-image model Generates 512x512px images from a prompt Runs on

    “commodity” hardware (with 8 GB VRAM) Open-source Generative AI power on the web Making web apps smarter with WebGPU and WebNN Stable Diffusion Prompt: A guinea pig eating a watermelon
  22. Specialized version of the Stable Diffusion model for the web

    2 GB in size Subject to usage conditions: https://huggingface.co/runwayml/stable- diffusion-v1-5#uses No npm package this time Currently incompatible with Angular & esbuild due to Wasm imports Generative AI power on the web Making web apps smarter with WebGPU and WebNN Web Stable Diffusion
  23. https://websd.mlc.ai/ Generative AI power on the web Making web apps

    smarter with WebGPU and WebNN Web Stable Diffusion DEMO
  24. Advantages – Data does not leave the browser – High

    availability (offline support) – Low latency – Stability (external API changes) – Low cost Generative AI power on the web Making web apps smarter with WebGPU and WebNN Local AI Models
  25. Disadvantages – Lower quality than closed-source models – High system

    requirements (RAM, GPU) – Large model size, high initial bandwidth requirements, models cannot be shared across origins – Model initialization and inference are relatively slow – WebGPU and WebNN are currently only supported by Chromium- based browsers on macOS and Windows (WebNN only behind a flag) – Prompt API is only an exploratory API Generative AI power on the web Making web apps smarter with WebGPU and WebNN Local AI Models
  26. Transformers.js JavaScript library to run Hugging Face transformers in the

    browser Supports most of the models https://xenova.github.io/transformers.js/ Generative AI power on the web Making web apps smarter with WebGPU and WebNN Alternatives
  27. – Cloud-based models (especially OpenAI/GPT) remain the most potent models

    and are easier to integrate (for now) – Due to their size and high system requirements, local generative AI models are currently rather interesting for very special scenarios (e.g., high privacy demands, offline availability) – Small, specialized models are an interesting alternative (if available) – Open-source GenAI models are becoming more compact and efficient – Vendors are beginning to ship AI models with their devices – Devices are becoming more powerful for AI tasks Generative AI power on the web Making web apps smarter with WebGPU and WebNN Summary