Slide 1
Slide 1 text
Prompt API & WebNN
The AI Revolution Right in Your Browser
Christian Liebel
@christianliebel
Consultant
Slide 2
Slide 2 text
Examples
Prompt API & WebNN
Generative AI Cloud Providers
The AI Revolution Right in Your Browser
Slide 3
Slide 3 text
Drawbacks
Prompt API & WebNN
Generative AI Cloud Providers
Require a (stable) internet connection
Subject to network latency and server availability
Data is transferred to the cloud service
Require a subscription
The AI Revolution Right in Your Browser
Slide 4
Slide 4 text
Can we run GenAI models locally?
Prompt API & WebNN
The AI Revolution Right in Your Browser
Slide 5
Slide 5 text
Bring Your Own AI (BYOAI)
– Libraries
– WebLLM
– Transfomers.js
– Frameworks
– ONNX Runtime
– TensorFlow.js
– APIs
– WebNN
– Cross-Origin Storage
Built-in AI (BIAI)
– Writing Assistance APIs
– Summarizer API
– Writer API
– Rewriter API
– Translator & Language
Detector APIs
– Prompt API
Prompt API & WebNN
Local AI Inference
The AI Revolution Right in Your Browser
Slide 6
Slide 6 text
https://webllm.mlc.ai/
Prompt API & WebNN
WebLLM
DEMO
The AI Revolution Right in Your Browser
Slide 7
Slide 7 text
On NPM
Prompt API & WebNN
WebLLM
The AI Revolution Right in Your Browser
Slide 8
Slide 8 text
Storing model files locally
Prompt API & WebNN
WebLLM
Internet
Website
HTML/JS
Cache with
model files
Hugging
Face
Note: Due to the Same-Origin Policy, models cannot be shared across origins.
The AI Revolution Right in Your Browser
Slide 9
Slide 9 text
Model Size Comparison
Model:Parameters Size
phi3:3b 2.2 GB
mistral:7b 4.1 GB
llama3:8b 4.7 GB
gemma2:9b 5.4 GB
gemma2:27b 16 GB
llama3:70b 40 GB
Prompt API & WebNN
WebLLM
The AI Revolution Right in Your Browser
Slide 10
Slide 10 text
https://huggingface.co/docs/transformers.js/index
Prompt API & WebNN
Transformers.js
DEMO
The AI Revolution Right in Your Browser
Slide 11
Slide 11 text
– Grants web apps access to
the device’s CPU, GPU and
Neural Processing Unit (NPU)
– In specification by the WebML
Working Group at W3C
– Implementation in progress in
Chromium (behind a flag)
– Better performance for
specific workloads
Prompt API & WebNN
WebNN
Source: https://webmachinelearning.github.io/webnn-intro/
DEMO
The AI Revolution Right in Your Browser
Slide 12
Slide 12 text
Prompt API & WebNN
WebNN
DEMO
The AI Revolution Right in Your Browser
Slide 13
Slide 13 text
about://flags
Enables WebNN API à Enabled
Enables experimental WebNN API features à Enabled
Prompt API & WebNN
WebNN
The AI Revolution Right in Your Browser
Slide 14
Slide 14 text
Drawbacks
Prompt API & WebNN
WebNN
Models can’t be shared across origins
Inference is fast, but doesn’t reach full native speed
The AI Revolution Right in Your Browser
Slide 15
Slide 15 text
https://github.com/explainers-by-googlers/cross-origin-storage
Prompt API & WebNN
Cross-Origin Storage
The AI Revolution Right in Your Browser
Slide 16
Slide 16 text
– Initiative by Google Chrome
– Exploratory APIs for local
experiments and use case
determination
– Downloads AI models into
Google Chrome
– Models are shared across origins
– Uses native APIs directly
(full performance)
Prompt API & WebNN
Built-in AI
https://developer.chrome.com/docs/ai/built-in
The AI Revolution Right in Your Browser
Slide 17
Slide 17 text
Incubated by the WebML CG
Prompt API & WebNN
Built-in AI APIs
https://webmachinelearning.github.io/incubations/
DEMO
The AI Revolution Right in Your Browser
Slide 18
Slide 18 text
Prompt API & WebNN
Multimodal Models
The AI Revolution Right in Your Browser
Slide 19
Slide 19 text
Prompt API & WebNN
Built-in AI APIs
Operating
System
Website
HTML/JS
Browser Internet
Apple Intelligence
Gemini Nano
The AI Revolution Right in Your Browser
Slide 20
Slide 20 text
about://on-device-internals
https://www.google.com/chrome/canary/
about://flags
Enables optimization guide on device à
EnabledBypassPerfRequirement
(API) for Gemini Nano à Enabled
Prompt API & WebNN
Built-in AI APIs
The AI Revolution Right in Your Browser
Slide 21
Slide 21 text
TypeScript Definitions
Prompt API & WebNN
Built-in AI APIs
The AI Revolution Right in Your Browser
Slide 22
Slide 22 text
Rule-based algorithms are
limited in their capabilities.
Prompt API & WebNN
Why should you care?
The AI Revolution Right in Your Browser
Slide 23
Slide 23 text
Use AI to implement use cases
that are difficult or impossible
to implement using rule-based
algorithms.
Prompt API & WebNN
Why should you care?
The AI Revolution Right in Your Browser
Slide 24
Slide 24 text
Prompt API & WebNN
Chatbots
DEMO
The AI Revolution Right in Your Browser
Slide 25
Slide 25 text
Prompt API & WebNN
Categorization
DEMO
The AI Revolution Right in Your Browser
Slide 26
Slide 26 text
(Cloud only!)
Prompt API & WebNN
Multimodal Realtime Models
DEMO
The AI Revolution Right in Your Browser
Slide 27
Slide 27 text
Prompt API & WebNN
The AI Revolution Right in Your Browser
Slide 28
Slide 28 text
Prompt API & WebNN
Realtime Models
DEMO
The AI Revolution Right in Your Browser
Slide 29
Slide 29 text
Pros & Cons
+ Data does not leave the
browser (privacy)
+ High availability
(offline support)
+ Low latency
+ Stability
(no external API changes)
+ Low cost
– Lower response quality
– Less capable
– High system (RAM, GPU) and
bandwidth requirements
– Large model size, models
cannot always be shared
– Model initialization and
inference are relatively slow
– APIs are experimental
Prompt API & WebNN
On-device AI Models
The AI Revolution Right in Your Browser
Slide 30
Slide 30 text
Thank you
for your kind attention!
Christian Liebel
@christianliebel
[email protected]