How WebAssembly Enhances Edge AI Operations

KubeCon + CloudNativeCon Japan, Japan Community Day

Transcript

1. How WebAssembly Enhances Edge AI Operations Japan Community Day July 28, 2026 Soichiro Ueda, Ai Nozaki
2. Self Introduction Soichiro Ueda @saza-ku @saza_ku • A Newbie Infrastructure Engineer • Interest: Cloud Computing, Systems Software Ai Nozaki @ainozaki @ainno321 • Ph.D student at the University of Tokyo • Interest: Hardware Acceleration, Systems Software 2
3. Edge AI operation
4. Rise of Edge AI • Edge AI runs inference directly on edge devices ◦ Enables low-latency local decisions ◦ Protects privacy-sensitive data Measuring advertising effectiveness at 7-Eleven [1] Biometric monitoring by SlateSafety [2] [1]https://www.sony-semicon.com/en/news/2024/2024042401.html [2]https://www.edgeimpulse.com/case-studies/slate-safety-predicting-heat-exhaustion
5. Edge AI Requires Updates after Deployment • Edge AI follows a continuous lifecycle ◦ Update to newer models ◦ Update pre/postprocessing logic Support updating model via over-the-air (OTA) Collect Data Deploy Improve Model
6. Problem of Edge AI operation • Device-specific artifacts
7. Problem of Edge AI operation • Device-specific artifacts Python package container NVIDIA Jetson OS: Ubuntu CPU: Arm Cortex-A
8. Problem of Edge AI operation • Device-specific artifacts Python package container STM specific firmware NVIDIA Jetson STM OS: Ubuntu OS: FreeRTOS/Zephyr CPU: Arm Cortex-A CPU: Arm Cortex-M
9. Problem of Edge AI operation • Device-specific artifacts Python package container STM specific firmware NVIDIA Jetson STM OS: Ubuntu OS: FreeRTOS/Zephyr CPU: Arm Cortex-A CPU: Arm Cortex-M ESP specific firmware ESP OS: FreeRTOS CPU: Xtensa
10. The Problem Grows with Scale • Device-specific operation does not scale ◦ More device families, more build and deploy pipelines ◦ More model and application versions, more artifacts New model New model (C++) Device specific⚠ App Device specific⚠ firm ware
11. Toward Cloud-like Operation for Edge AI • Containers provide a common deployment unit ◦ Applications share a common packaging model ◦ Orchestrators manage life cycles of applications Can we bring this operational model to edge devices?
12. Why existing cloud techniques don’t fit? • Resource constrained devices cannot run containers Feasible but costs much No Linux No Linux ex. nvidia/l4t-pytorch: 5.6GB NVIDIA Jetson STM OS: Ubuntu OS: FreeRTOS/Zephyr CPU: Arm Cortex-A CPU: Arm Cortex-M ESP OS: FreeRTOS CPU: Xtensa
13. Why existing cloud techniques don’t fit? • Container images remain architecture-dependent NVIDIA Jetson OS: Ubuntu CPU: Arm Cortex-A Dev env OS: Ubuntu CPU: x86-64
14. Existing Approaches for the Edge • K3s ◦ Lightweight Kubernetes distribution ◦ Still requires Linux • KubeEdge ◦ Extends Kubernetes management to edge nodes ◦ Edge devices don’t typically execute workloads ▪ KubeEdge manages them as device resources
15. Our Focus: WebAssembly and Kubernetes
16. Our Focus: WebAssembly • Wasm is a portable binary format ◦ Compiled from multiple programming languages ◦ Portable across CPUs and OSs ◦ Sandboxed for isolated execution
17. WASI: System Interfaces for Wasm • WASI provides standardized interfaces for Wasm app ◦ ex. Files and streams ◦ ex. Networking Wasm App. Wasm App. WASI Wasm runtime Host kernel
18. WASI-NN • Standardized WASI for Neural Networks ◦ Wasm application invokes inference through WASI-NN ◦ Inference engines remain inside the runtime Wasm App. Wasm App. WASI-NN Wasm runtime Inference Engine • • • • load() set_input() compute() get_output()
19. WebAssembly for Edge Devices • Wasm provides a portable and smaller deployment unit ◦ Does not require a full Linux userspace ◦ Portable across CPU architectures 2.47GB 100MB → Wasm is a promising deployment format for edge devices
20. Approach: WebAssembly × Kubernetes Lightweight Packaging containerd/runwasi Continuous Updating / Lifecycle Management
21. Approach: WebAssembly × Kubernetes Q. IoT Device as a Kubernetes Node?
22. How to manage IoT Devices with Kubernetes Container Runtime Kubelet Linux Regular Kubernetes Node
23. How to manage IoT Devices with Kubernetes Only runs on Linux Container Runtime Wasm Runtime Kubelet ？ Linux Embedded OS Doesn’t run on small devices Regular Kubernetes Node IoT Device
24. How to manage IoT Devices with Kubernetes 💡 Edge server runs Kubelet instead Proxy Kubelet Wasm Runtime Linux Embedded OS Edge Server IoT Device
25. Implementations
26. Wasm Runtime: waiot • Wasm Runtime targeting small IoT devices (around 500KiB RAM) • Use WebAssembly Micro Runtime (WAMR) ◦ Manage the runtime ◦ Works like containerd / CRI-O ▪ containerd ↔ waiot ▪ runc ↔ WAMR bytecodealliance/wasm-micro-runtime WAMR waiot Embedded OS • Supported Platforms: ◦ ESP32 ◦ STM32 (WIP) github.com/mewz-project/waiot
27. Proxy Kubelet: waiot-kubelet • Works as Kubelets for IoT devices ◦ Runs on Linux Edge Servers ◦ Cooperates with waiot waiot-kubelet • Watch Pod Resources → Make waiot run Wasm ◦ Call remote waiot, instead of local container runtime mewz-project/waiot-kubelet waiot waiot waiot
28. How Proxy Kubelet Works control plane waiot-kubelet waiot waiot device-1 device-2
29. How Proxy Kubelet Works control plane created → scheduled to device-1 waiot-kubelet waiot waiot device-1 device-2
30. How Proxy Kubelet Works control plane device-1 pull waiot-kubelet waiot waiot device-1 device-2
31. How Proxy Kubelet Works control plane device-1 waiot-kubelet send waiot waiot device-1 device-2
32. How Proxy Kubelet Works control plane device-1 waiot-kubelet waiot run device-1 waiot device-2
33. Virtual Kubelet • Library to create your own Kubelet implementation • Defines Interfaces that must be implemented for Kubelet ◦ e.g. CreatePod, UpdatePod, DeletePod, etc virtual-kubelet/virtual-kubelet
34. Pod Deployment Time
35. Future Works
36. Future Work: NPU/GPU Support Wasm App. Wasm App. WASI-NN Wasm runtime Inference Engine
37. Future Work: NPU/GPU Support Wasm App. Wasm App. WASI-NN Wasm runtime ARM TrustZone (TEE) Secure World Inference Engine
38. Future Work: NPU/GPU Support Wasm App. Wasm App. WASI-NN Wasm runtime Secure World Inference Engine
39. Conclusion • How to manage Edge AI models? ◦ Frequent Updates ◦ Constrained Resources → Wasm meets the Edge AI requirements • How to manage Wasm on IoT devices? → Make devices Kubernetes nodes ◦ waiot: Wasm Runtime ◦ waiot-kubelet: Proxy Kubelet • Future Works: AI Inference with GPU/NPU, TrustZone