Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Crazy computer vision stuff in the browser
Search
Zeno Rocha
April 28, 2015
Programming
5
820
Crazy computer vision stuff in the browser
Zeno Rocha
April 28, 2015
Tweet
Share
More Decks by Zeno Rocha
See All by Zeno Rocha
The Next Generation of Developer-First Products
zenorocha
1
410
7 Habits of Highly Productive Developers
zenorocha
1
270
7 Hábitos de Desenvolvedores Altamente Produtivos
zenorocha
1
300
What's new in the Liferay Community
zenorocha
0
550
Launching Liferay Projects Faster with WeDeploy
zenorocha
1
440
How Liferay fits into the real of latest technologies
zenorocha
0
380
Estoicismo e JavaScript
zenorocha
3
870
Por que ninguém se importa com seu novo projeto open source?
zenorocha
2
750
Como investir em... você!
zenorocha
1
460
Other Decks in Programming
See All in Programming
CDKコントリビュートの最初の壁を越えよう! -簡単issueの見つけ方-
badmintoncryer
3
310
TCAとKMPを用いた新規動画配信アプリ 「ABEMA Live」の設計
tomu28
2
130
Try creating your own orderedmap
kazamori
1
260
Direct Style Effect Systems The Print[A] ExampleA Comprehension Aid
philipschwarz
PRO
0
200
“Seeing Like a Programmer”—Resiliency, Limits, and Moral Hazards in Software Engineering (LambdaConf 2024)
chriskrycho
0
280
冗長なエラーログを削減し、スタックトレースを手に入れる / Reducing Verbose Error Logs and Obtaining Stack Traces
upamune
0
1.1k
ServerAction で Progressive Enhancement はどこまで頑張れるか? / progressive-enhancement-with-server-action
takefumiyoshii
6
460
GitHub Copilotのススメ
marcy731
1
240
R言語の環境構築と基礎 Tokyo.R 112
bob3bob3
0
280
SIMD Parallel Programming with the Vector API
josepaumard
0
240
Balkan Ruby 2024 — How and why to run SQLite on Rails in production
fractaledmind
0
100
Micro Frontends for Java Microservices - Utah JUG 2024
mraible
PRO
1
110
Featured
See All Featured
How to name files
jennybc
65
93k
Rebuilding a faster, lazier Slack
samanthasiow
74
8.3k
Git: the NoSQL Database
bkeepers
PRO
423
63k
The MySQL Ecosystem @ GitHub 2015
samlambert
244
12k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
34
8.9k
How To Stay Up To Date on Web Technology
chriscoyier
782
250k
4 Signs Your Business is Dying
shpigford
176
21k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
221
21k
Automating Front-end Workflow
addyosmani
1357
200k
Six Lessons from altMBA
skipperchong
22
3k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
228
16k
Atom: Resistance is Futile
akmur
260
25k
Transcript
CRAZY COMPUTER vision stuff in the browser
.html .css .js
zenorocha
None
None
None
None
None
computer vision ? what is
Persistence of vision
Persistence of vision
Motion perception
Motion perception
vimeo.com/108331968
Face detection
Face recognition
Autonomous robots
Autonomous robots
Virtual reality
Augmented reality
Augmented reality
how to do that?
None
None
how to do that in the browser?
1. Access the camera
None
Request user’s webcam navigator.getUserMedia({ video: true }, onSuccess, onFail);
Request user’s webcam navigator.getUserMedia({ video: true }, onSuccess, onFail);
Request user’s webcam navigator.getUserMedia({ video: true }, onSuccess, onFail);
2. Reproduce in a video
<video>
Get User Media callback function onSuccess(stream) { var video =
document.querySelector('video'); video.src = URL.createObjectURL(stream); video.onloadedmetadata = function(e) { // Done }; }
Get User Media callback function onSuccess(stream) { var video =
document.querySelector('video'); video.src = URL.createObjectURL(stream); video.onloadedmetadata = function(e) { // Done }; }
Get User Media callback function onSuccess(stream) { var video =
document.querySelector('video'); video.src = URL.createObjectURL(stream); video.onloadedmetadata = function(e) { // Done }; }
Get User Media callback function onSuccess(stream) { var video =
document.querySelector('video'); video.src = URL.createObjectURL(stream); video.onloadedmetadata = function(e) { // Done }; }
3. Obtain pixel matrix
<canvas>
4. Tracking algorithms
Request accepted Request access to the camera Reproduce camera <video>
Obtain pixel matrix <canvas> Tracking algorithms Result Summary
tracking.js
None
Maira Bello Team eduardo lundgren java pablo carvalho zeno rocha
100% JavaScript! Open source No dependencies Simple and intuitive API
Automatic setup Built-in tracking algorithms Easily extensible High performance Unit & performance regression tests ~ 7 Kb
tracker color
Demo
var tracker = new tracking.ColorTracker('magenta'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
var tracker = new tracking.ColorTracker('magenta'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
var tracker = new tracking.ColorTracker('magenta'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
tracker object
Demo
var tracker = new tracking.ObjectTracker('face'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
var tracker = new tracking.ObjectTracker('face'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
var tracker = new tracking.ObjectTracker('face'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
tracker custom
var MyTracker = function() { MyTracker.base(this, 'constructor'); }; tracking.inherits(MyTracker, tracking.Tracker);
MyTracker.prototype.track = function(pxs, width, height) { // Your code here this.emit('track', { // Your code here }); };
var MyTracker = function() { MyTracker.base(this, 'constructor'); }; tracking.inherits(MyTracker, tracking.Tracker);
MyTracker.prototype.track = function(pxs, width, height) { // Your code here this.emit('track', { // Your code here }); };
var MyTracker = function() { MyTracker.base(this, 'constructor'); }; tracking.inherits(MyTracker, tracking.Tracker);
MyTracker.prototype.track = function(pxs, width, height) { // Your code here this.emit('track', { // Your code here }); };
var MyTracker = function() { MyTracker.base(this, 'constructor'); }; tracking.inherits(MyTracker, tracking.Tracker);
MyTracker.prototype.track = function(pxs, width, height) { // Your code here this.emit('track', { // Your code here }); };
var tracker = new tracking.MyTracker(); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
var tracker = new tracking.MyTracker(); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
var tracker = new tracking.MyTracker(); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
who is using it?
kdzwinel.github.io/JS-face-tracking-demo
thedevilini.com/#masks
lessons learned
24 fps 60 fps 1000ms / 24fps = 41ms per
frame
24 fps 60 fps 1000ms / 24fps = 41ms per
frame
None
None
None
Int32Array Uint16Array Float64Array Uint8ClampedArray [ ] Typed arrays
jsperf.com/tracking-js-arrays
jsperf.com/tracking-js-arithmetic
Demo
None
None
web components > > -
how we create a component nowadays?
1. Never create! Just use a plugin
2. Copy & paste someone’s code
3. And hope it works
3. And hope it works
web components
web components Templates
web components Templates Custom Elements
web components Templates Custom Elements Shadow DOM
web components HTML Imports Templates Custom Elements Shadow DOM
None
var tracker = new tracking.ColorTracker('magenta'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); }); Remember?
Using Web Components… <video is="video-color-tracking" target="magenta" camera="true" ontrack="onResult" </video>
more ?
trackingjs.com
Where the magic happens Your comfort zone
Thanks! zenorocha.com @zenorocha
Thanks! zenorocha.com @zenorocha