Computer Vision at the Edge with WebAssembly

COMPUTER VISION AT THE EDGE WITH WEBASSEMBLY
MIGUEL ÁNGEL CABRERA - PIPELESS.AI / ANGEL DE MIGUEL - BROADCOM
Computer vision is a field of Artificial Intelligence with very challenging constraints. To process data in real-time, your application must process every frame in microseconds. Often, these applications make decisions based on the images, making it impossible to move these workloads to the cloud due to network delay, connection issues, and privacy concerns, among other reasons.

In this talk, you will learn about how WebAssembly allows us to build applications that run on any device. We will deep dive into how Pipeless, an Open-Source computer vision framework leverages this WASM capability.

From a few devices to large fleets, you can run your computer vision applications using the same code. These applications are small Wasm Components that Pipeless plugs into the video streams to process and analyze them. It leverages the Wasm Component-Model, so developers can write these components in many different programming languages.

After this talk, you will understand the different challenges of the computer vision ecosystem and how WASM, Pipeless, and the Component-Model simplify them significantly, providing an amazing development experience in a space that currently has a high entry barrier.

Transcript

1. COMPUTER VISION AT THE EDGE WITH WEBASSEMBLY Miguel Angel Cabrera @miguelaeh_ Angel M De Miguel @_angelmm Pipeless
2. COMPUTER VISION (CV) Pipeless
3. A SIMPLE CV APP Model Inference Runtime GPU Output Pipeline Input Pipeline Post-processing Pre-processing CPU video streams video streams Pipeless 60 FPS / 16ms per frame Raw frame Video
4. CHALLENGES OF COMPUTER VISION • Parallelize frame processing? • Latency? Pipeless Performance and stability • How do you start with computer vision? • What languages and frameworks can you use? Development experience • How to distribute a computer vision platform? • How to share processing logic? Distribution
5. PIPELESS.AI Pipeless is an open-source framework that takes care of everything you need to develop and deploy computer vision applications in just minutes https://github.com/pipeless-ai/pipeless Pipeless
6. ARCHITECTURE webcam, RTSP, WebRTC, … f f f f actionable data streams (JSON payloads) Pipeless pre-process post-process process (inference) frame data OpenVINO CoreML ONNX … Raw RGB structured JSON
7. Pipeless pre-process.py process.json post-process.py PIPELESS HELLO WORLD
8. Pipeless
9. Pipeless Pipeless Original: • FPS: 24 • Duration: 9s Processing: • FPS: 31 • Duration: 7s
10. WASM? Pipeless
11. PIPELESS 6 MONTHS AGO… Pipeless • A Python project • Filters (user-defined) were written in Python • Performance issues (processing frames in parallel) • Big size artifacts Pipeless
12. • Provide tools to simplify filters' development. • Develop filters in multiple languages. CHALLENGES OF COMPUTER VISION Pipeless Performance and stability Developer Experience • Ensure filters perform properly and you can parallelize them. • Filters run in any environment / device. • Simplify filter distribution and sharing them. Distribution
13. Pipeless
14. PIPELESS 🖤 WASM Pipeless pre-process post-process process (inference) frame data OpenVINO CoreML ONNX … Raw RGB structured JSON • Define filters as Wasm components, so you can use different languages. • Instantiate multiple filters in parallel and isolate them. • Filters run in any environment where Pipeless run. • Define a proper interface (with the Component Model) and compose filters, making small and shareable units.
15. PIPELESS EVOLUTION Python Rust + Pyo3 Rust + Pyo3 + Wasm (exp) Pipeless
16. IMPLEMENTATION DETAILS Pipeless
17. STATEFUL VS STATELESS A component per thread. N-cores run in parallel. A single component, with internal state, shared among all threads and with sequential sorted processing. Pipeless lock run release Stateless run run run Stateful
18. PYTHON ON WASM Pipeless Python user code componetize-py • (Computer Vision) users love Python. • Wasm unlocks parallel processing (avoid GIL limitations) at the cost of memory. • Running a Python script in Wasm is significantly slower.
19. AVOID COPYING FRAMES Latency is critical. Copying raw frames to WASM memory is too slow… ptr host-ref HashMap frame resize, normalize, draw, permute, etc. Pipeless common operations API invoke with host-ref WIT
20. DEMO Pipeless
21. Pipeless Original: • FPS: 24 • Duration: 9s Processing: • FPS: ~8 • Duration: 29s
22. Benefits • Securely run untrusted code • Easy to share filters • Multi-language support • Get rid of Python GIL limitations Challenges • Python components are slow • Missing support for critical libraries • Complex workarounds for no-copy RECAP A bright future awaits us with Wasm! Pipeless
23. THANKS! Miguel Angel Cabrera @miguelaeh_ Angel M De Miguel @_angelmm Pipeless ⭐ Star Pipeless on GitHub! https://github.com/pipeless-ai/pipeless