Built-in AI APIs & WebNN: AI right in your browser, local and offline-capable

More and more developers intend to integrate Generative AI features into their applications. Until now, this path has practically always led to the cloud—but it doesn't have to be like that! Currently, various promising approaches exist to running AI models directly on the user's computer. With Chromium's new Built-in AI APIs, we can now bring Large Language Models to your browser: locally and offline-capable. The W3C's Web Neural Network API (WebNN) will grant AI models access to the device's Neural Processing Unit (NPU). The advantages of these approaches are obvious: Locally executed AI models are available offline, the user data does not leave the device, and all this is even free of charge thanks to open-source models. In this talk, Christian Liebel, Thinktecture's representative at W3C, will present the approaches to make your single-page app smarter. We will discuss use cases and show the advantages and disadvantages of each solution.

Transcript

1. Built-in AI APIs & WebNN AI right in your browser, local and offline-capable Christian Liebel @christianliebel Consultant
2. Hello, it’s me. Built-in AI APIs & WebNN Christian Liebel W3C WebML WG & CG TAG Associate X: @christianliebel Bluesky: @christianliebel.com Angular, PWA & Generative AI Microsoft MVP & Google GDE (Angular, Web) AI right in your browser, local and offline-capable
3. Examples Built-in AI APIs & WebNN Generative AI Cloud Providers AI right in your browser, local and offline-capable
4. Drawbacks Built-in AI APIs & WebNN Generative AI Cloud Providers Require a (stable) internet connection Subject to network latency and server availability Data is transferred to the cloud service Require a subscription AI right in your browser, local and offline-capable
5. Can we run GenAI models locally? Built-in AI APIs & WebNN AI right in your browser, local and offline-capable
6. Bring Your Own AI (BYOAI) – Libraries – WebLLM – Transfomers.js – Frameworks – ONNX Runtime – TensorFlow.js – APIs – WebNN – Cross-Origin Storage Built-in AI (BIAI) – Writing Assistance APIs – Summarizer API – Writer API – Rewriter API – Translator & Language Detector APIs – Prompt API Built-in AI APIs & WebNN Local AI Inference AI right in your browser, local and offline-capable
7. https://webllm.mlc.ai/ Built-in AI APIs & WebNN WebLLM DEMO AI right in your browser, local and offline-capable
8. On NPM Built-in AI APIs & WebNN WebLLM AI right in your browser, local and offline-capable
9. Storing model files locally Built-in AI APIs & WebNN WebLLM Internet Website HTML/JS Cache with model files Hugging Face Note: Due to the Same-Origin Policy, models cannot be shared across origins. AI right in your browser, local and offline-capable
10. Model Size Comparison Model:Parameters Size phi3:3b 2.2 GB mistral:7b 4.1 GB llama3:8b 4.7 GB gemma2:9b 5.4 GB gemma2:27b 16 GB llama3:70b 40 GB Built-in AI APIs & WebNN WebLLM AI right in your browser, local and offline-capable
11. https://huggingface.co/docs/transformers.js/index Built-in AI APIs & WebNN Transformers.js DEMO AI right in your browser, local and offline-capable
12. – Grants web apps access to the device’s CPU, GPU and Neural Processing Unit (NPU) – In specification by the WebML Working Group at W3C – Implementation in progress in Chromium (behind a flag) – Better performance for specific workloads Built-in AI APIs & WebNN WebNN Source: https://webmachinelearning.github.io/webnn-intro/ DEMO AI right in your browser, local and offline-capable
13. Built-in AI APIs & WebNN Why should you care? DEMO AI right in your browser, local and offline-capable
14. about://flags Enables WebNN API à Enabled Enables experimental WebNN API features à Enabled Built-in AI APIs & WebNN WebNN AI right in your browser, local and offline-capable
15. Drawbacks Built-in AI APIs & WebNN WebNN Models can’t be shared across origins Inference is fast, but doesn’t reach full native speed AI right in your browser, local and offline-capable
16. https://github.com/explainers-by-googlers/cross-origin-storage Built-in AI APIs & WebNN Cross-Origin Storage AI right in your browser, local and offline-capable
17. – Initiative by Google Chrome – Exploratory APIs for local experiments and use case determination – Downloads AI models into Google Chrome – Models are shared across origins – Uses native APIs directly (full performance) Built-in AI APIs & WebNN Built-in AI https://developer.chrome.com/docs/ai/built-in AI right in your browser, local and offline-capable
18. Incubated by the WebML CG Built-in AI APIs & WebNN Built-in AI APIs https://webmachinelearning.github.io/incubations/ DEMO AI right in your browser, local and offline-capable
19. Built-in AI APIs & WebNN Multimodal Models AI right in your browser, local and offline-capable
20. Built-in AI APIs & WebNN Built-in AI APIs Operating System Website HTML/JS Browser Internet Apple Intelligence Gemini Nano AI right in your browser, local and offline-capable
21. about://on-device-internals https://www.google.com/chrome/canary/ about://flags Enables optimization guide on device à EnabledBypassPerfRequirement (API) for Gemini Nano à Enabled Built-in AI APIs & WebNN Built-in AI APIs AI right in your browser, local and offline-capable
22. TypeScript Definitions Built-in AI APIs & WebNN Built-in AI APIs AI right in your browser, local and offline-capable
23. Built-in AI APIs & WebNN Chatbots DEMO AI right in your browser, local and offline-capable
24. Built-in AI APIs & WebNN Categorization DEMO AI right in your browser, local and offline-capable
25. Built-in AI APIs & WebNN Realtime Models DEMO AI right in your browser, local and offline-capable
26. (Cloud only!) Built-in AI APIs & WebNN Multimodal Realtime Models DEMO AI right in your browser, local and offline-capable
27. Pros & Cons + Data does not leave the browser (privacy) + High availability (offline support) + Low latency + Stability (no external API changes) + Low cost – Lower response quality – Less capable – High system (RAM, GPU) and bandwidth requirements – Large model size, models cannot always be shared – Model initialization and inference are relatively slow – APIs are experimental Built-in AI APIs & WebNN On-device AI Models AI right in your browser, local and offline-capable
28. Thank you for your kind attention! Christian Liebel @christianliebel christian.liebel@thinktecture.com