Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Crazy computer vision stuff in the browser
Zeno Rocha
April 28, 2015
Programming
5
660
Crazy computer vision stuff in the browser
Zeno Rocha
April 28, 2015
Tweet
Share
More Decks by Zeno Rocha
See All by Zeno Rocha
The Next Generation of Developer-First Products
zenorocha
0
140
7 Habits of Highly Productive Developers
zenorocha
0
62
7 Hábitos de Desenvolvedores Altamente Produtivos
zenorocha
1
50
What's new in the Liferay Community
zenorocha
0
390
Launching Liferay Projects Faster with WeDeploy
zenorocha
1
280
How Liferay fits into the real of latest technologies
zenorocha
0
180
Estoicismo e JavaScript
zenorocha
3
620
Por que ninguém se importa com seu novo projeto open source?
zenorocha
2
530
Como investir em... você!
zenorocha
1
300
Other Decks in Programming
See All in Programming
Power Automateドリブンのチームマネジメント
hanaseleb
0
200
BASE BANKチームの技術選定と歴史 / how to decide technology selection for startup
budougumi0617
0
1.4k
Scrum Fest Osaka 2022/5年で200人になったスタートアップの アジャイル開発の歴史とリアル
atamaplus
1
960
エンジニアによる事業指標計測のススメ
doyaaaaaken
1
190
無限スクロールビューライブラリ 二つの設計思想比較
harumak
0
270
Beyond Micro Frontends: Frontend Moduliths for the Enterprise @enterjs2022
manfredsteyer
PRO
0
200
Get Ready for Jakarta EE 10
ivargrimstad
0
1.1k
開発速度を5倍早くするVSCodeの拡張機能を作った
purp1eeeee
2
160
IE Graduation (IE の功績を讃える)
jxck
20
12k
io22 extended What's new in app performance
veronikapj
0
350
How we run a Realtime Puzzle Fighting Game on AWS Serverless
falken
0
250
ゴーファーくんと辿るプログラミング言語の歴史/history-of-programming-languages-with-gopher
iwasiman
7
3.7k
Featured
See All Featured
Making Projects Easy
brettharned
98
4.3k
StorybookのUI Testing Handbookを読んだ
zakiyama
5
2.3k
Testing 201, or: Great Expectations
jmmastey
21
5.4k
Large-scale JavaScript Application Architecture
addyosmani
499
110k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
37
3.3k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
11
4.7k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
7
1.1k
The Language of Interfaces
destraynor
148
20k
What's in a price? How to price your products and services
michaelherold
229
9.4k
Why You Should Never Use an ORM
jnunemaker
PRO
47
7.6k
Robots, Beer and Maslow
schacon
152
7.1k
Docker and Python
trallard
27
1.6k
Transcript
CRAZY COMPUTER vision stuff in the browser
.html .css .js
zenorocha
None
None
None
None
None
computer vision ? what is
Persistence of vision
Persistence of vision
Motion perception
Motion perception
vimeo.com/108331968
Face detection
Face recognition
Autonomous robots
Autonomous robots
Virtual reality
Augmented reality
Augmented reality
how to do that?
None
None
how to do that in the browser?
1. Access the camera
None
Request user’s webcam navigator.getUserMedia({ video: true }, onSuccess, onFail);
Request user’s webcam navigator.getUserMedia({ video: true }, onSuccess, onFail);
Request user’s webcam navigator.getUserMedia({ video: true }, onSuccess, onFail);
2. Reproduce in a video
<video>
Get User Media callback function onSuccess(stream) { var video =
document.querySelector('video'); video.src = URL.createObjectURL(stream); video.onloadedmetadata = function(e) { // Done }; }
Get User Media callback function onSuccess(stream) { var video =
document.querySelector('video'); video.src = URL.createObjectURL(stream); video.onloadedmetadata = function(e) { // Done }; }
Get User Media callback function onSuccess(stream) { var video =
document.querySelector('video'); video.src = URL.createObjectURL(stream); video.onloadedmetadata = function(e) { // Done }; }
Get User Media callback function onSuccess(stream) { var video =
document.querySelector('video'); video.src = URL.createObjectURL(stream); video.onloadedmetadata = function(e) { // Done }; }
3. Obtain pixel matrix
<canvas>
4. Tracking algorithms
Request accepted Request access to the camera Reproduce camera <video>
Obtain pixel matrix <canvas> Tracking algorithms Result Summary
tracking.js
None
Maira Bello Team eduardo lundgren java pablo carvalho zeno rocha
100% JavaScript! Open source No dependencies Simple and intuitive API
Automatic setup Built-in tracking algorithms Easily extensible High performance Unit & performance regression tests ~ 7 Kb
tracker color
Demo
var tracker = new tracking.ColorTracker('magenta'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
var tracker = new tracking.ColorTracker('magenta'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
var tracker = new tracking.ColorTracker('magenta'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
tracker object
Demo
var tracker = new tracking.ObjectTracker('face'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
var tracker = new tracking.ObjectTracker('face'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
var tracker = new tracking.ObjectTracker('face'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
tracker custom
var MyTracker = function() { MyTracker.base(this, 'constructor'); }; tracking.inherits(MyTracker, tracking.Tracker);
MyTracker.prototype.track = function(pxs, width, height) { // Your code here this.emit('track', { // Your code here }); };
var MyTracker = function() { MyTracker.base(this, 'constructor'); }; tracking.inherits(MyTracker, tracking.Tracker);
MyTracker.prototype.track = function(pxs, width, height) { // Your code here this.emit('track', { // Your code here }); };
var MyTracker = function() { MyTracker.base(this, 'constructor'); }; tracking.inherits(MyTracker, tracking.Tracker);
MyTracker.prototype.track = function(pxs, width, height) { // Your code here this.emit('track', { // Your code here }); };
var MyTracker = function() { MyTracker.base(this, 'constructor'); }; tracking.inherits(MyTracker, tracking.Tracker);
MyTracker.prototype.track = function(pxs, width, height) { // Your code here this.emit('track', { // Your code here }); };
var tracker = new tracking.MyTracker(); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
var tracker = new tracking.MyTracker(); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
var tracker = new tracking.MyTracker(); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
who is using it?
kdzwinel.github.io/JS-face-tracking-demo
thedevilini.com/#masks
lessons learned
24 fps 60 fps 1000ms / 24fps = 41ms per
frame
24 fps 60 fps 1000ms / 24fps = 41ms per
frame
None
None
None
Int32Array Uint16Array Float64Array Uint8ClampedArray [ ] Typed arrays
jsperf.com/tracking-js-arrays
jsperf.com/tracking-js-arithmetic
Demo
None
None
web components > > -
how we create a component nowadays?
1. Never create! Just use a plugin
2. Copy & paste someone’s code
3. And hope it works
3. And hope it works
web components
web components Templates
web components Templates Custom Elements
web components Templates Custom Elements Shadow DOM
web components HTML Imports Templates Custom Elements Shadow DOM
None
var tracker = new tracking.ColorTracker('magenta'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); }); Remember?
Using Web Components… <video is="video-color-tracking" target="magenta" camera="true" ontrack="onResult" </video>
more ?
trackingjs.com
Where the magic happens Your comfort zone
Thanks! zenorocha.com @zenorocha
Thanks! zenorocha.com @zenorocha