Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Crazy computer vision stuff in the browser
Search
Zeno Rocha
April 28, 2015
Programming
5
1k
Crazy computer vision stuff in the browser
Zeno Rocha
April 28, 2015
Tweet
Share
More Decks by Zeno Rocha
See All by Zeno Rocha
The Next Generation of Developer-First Products
zenorocha
1
610
7 Habits of Highly Productive Developers
zenorocha
1
400
7 Hábitos de Desenvolvedores Altamente Produtivos
zenorocha
1
510
What's new in the Liferay Community
zenorocha
0
690
Launching Liferay Projects Faster with WeDeploy
zenorocha
1
560
How Liferay fits into the real of latest technologies
zenorocha
0
610
Estoicismo e JavaScript
zenorocha
3
1.1k
Por que ninguém se importa com seu novo projeto open source?
zenorocha
2
980
Como investir em... você!
zenorocha
1
570
Other Decks in Programming
See All in Programming
レガシープロジェクトで最大限AIの恩恵を受けられるようClaude Codeを利用する
tk1351
2
1k
実践!App Intents対応
yuukiw00w
1
350
CEDEC 2025 『ゲームにおけるリアルタイム通信への QUIC導入事例の紹介』
segadevtech
3
970
GitHub Copilotの全体像と活用のヒント AI駆動開発の最初の一歩
74th
8
3.2k
モバイルアプリからWebへの横展開を加速した話_Claude_Code_実践術.pdf
kazuyasakamoto
0
260
新世界の理解
koriym
0
140
ソフトウェアテスト徹底指南書の紹介
goyoki
1
110
コンテキストエンジニアリング Cursor編
kinopeee
1
700
AI OCR API on Lambdaを Datadogで可視化してみた
nealle
0
180
エンジニアのための”最低限いい感じ”デザイン入門
shunshobon
0
130
TROCCO×dbtで実現する人にもAIにもやさしいデータ基盤
nealle
0
330
物語を動かす行動"量" #エンジニアニメ
konifar
14
5.4k
Featured
See All Featured
The MySQL Ecosystem @ GitHub 2015
samlambert
251
13k
Fantastic passwords and where to find them - at NoRuKo
philnash
51
3.4k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
7
820
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
Typedesign – Prime Four
hannesfritz
42
2.8k
Art, The Web, and Tiny UX
lynnandtonic
302
21k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
131
19k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
3k
Making Projects Easy
brettharned
117
6.3k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
53
2.9k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
229
22k
Testing 201, or: Great Expectations
jmmastey
45
7.6k
Transcript
CRAZY COMPUTER vision stuff in the browser
.html .css .js
zenorocha
None
None
None
None
None
computer vision ? what is
Persistence of vision
Persistence of vision
Motion perception
Motion perception
vimeo.com/108331968
Face detection
Face recognition
Autonomous robots
Autonomous robots
Virtual reality
Augmented reality
Augmented reality
how to do that?
None
None
how to do that in the browser?
1. Access the camera
None
Request user’s webcam navigator.getUserMedia({ video: true }, onSuccess, onFail);
Request user’s webcam navigator.getUserMedia({ video: true }, onSuccess, onFail);
Request user’s webcam navigator.getUserMedia({ video: true }, onSuccess, onFail);
2. Reproduce in a video
<video>
Get User Media callback function onSuccess(stream) { var video =
document.querySelector('video'); video.src = URL.createObjectURL(stream); video.onloadedmetadata = function(e) { // Done }; }
Get User Media callback function onSuccess(stream) { var video =
document.querySelector('video'); video.src = URL.createObjectURL(stream); video.onloadedmetadata = function(e) { // Done }; }
Get User Media callback function onSuccess(stream) { var video =
document.querySelector('video'); video.src = URL.createObjectURL(stream); video.onloadedmetadata = function(e) { // Done }; }
Get User Media callback function onSuccess(stream) { var video =
document.querySelector('video'); video.src = URL.createObjectURL(stream); video.onloadedmetadata = function(e) { // Done }; }
3. Obtain pixel matrix
<canvas>
4. Tracking algorithms
Request accepted Request access to the camera Reproduce camera <video>
Obtain pixel matrix <canvas> Tracking algorithms Result Summary
tracking.js
None
Maira Bello Team eduardo lundgren java pablo carvalho zeno rocha
100% JavaScript! Open source No dependencies Simple and intuitive API
Automatic setup Built-in tracking algorithms Easily extensible High performance Unit & performance regression tests ~ 7 Kb
tracker color
Demo
var tracker = new tracking.ColorTracker('magenta'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
var tracker = new tracking.ColorTracker('magenta'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
var tracker = new tracking.ColorTracker('magenta'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
tracker object
Demo
var tracker = new tracking.ObjectTracker('face'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
var tracker = new tracking.ObjectTracker('face'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
var tracker = new tracking.ObjectTracker('face'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
tracker custom
var MyTracker = function() { MyTracker.base(this, 'constructor'); }; tracking.inherits(MyTracker, tracking.Tracker);
MyTracker.prototype.track = function(pxs, width, height) { // Your code here this.emit('track', { // Your code here }); };
var MyTracker = function() { MyTracker.base(this, 'constructor'); }; tracking.inherits(MyTracker, tracking.Tracker);
MyTracker.prototype.track = function(pxs, width, height) { // Your code here this.emit('track', { // Your code here }); };
var MyTracker = function() { MyTracker.base(this, 'constructor'); }; tracking.inherits(MyTracker, tracking.Tracker);
MyTracker.prototype.track = function(pxs, width, height) { // Your code here this.emit('track', { // Your code here }); };
var MyTracker = function() { MyTracker.base(this, 'constructor'); }; tracking.inherits(MyTracker, tracking.Tracker);
MyTracker.prototype.track = function(pxs, width, height) { // Your code here this.emit('track', { // Your code here }); };
var tracker = new tracking.MyTracker(); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
var tracker = new tracking.MyTracker(); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
var tracker = new tracking.MyTracker(); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
who is using it?
kdzwinel.github.io/JS-face-tracking-demo
thedevilini.com/#masks
lessons learned
24 fps 60 fps 1000ms / 24fps = 41ms per
frame
24 fps 60 fps 1000ms / 24fps = 41ms per
frame
None
None
None
Int32Array Uint16Array Float64Array Uint8ClampedArray [ ] Typed arrays
jsperf.com/tracking-js-arrays
jsperf.com/tracking-js-arithmetic
Demo
None
None
web components > > -
how we create a component nowadays?
1. Never create! Just use a plugin
2. Copy & paste someone’s code
3. And hope it works
3. And hope it works
web components
web components Templates
web components Templates Custom Elements
web components Templates Custom Elements Shadow DOM
web components HTML Imports Templates Custom Elements Shadow DOM
None
var tracker = new tracking.ColorTracker('magenta'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); }); Remember?
Using Web Components… <video is="video-color-tracking" target="magenta" camera="true" ontrack="onResult" </video>
more ?
trackingjs.com
Where the magic happens Your comfort zone
Thanks! zenorocha.com @zenorocha
Thanks! zenorocha.com @zenorocha