Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Crazy computer vision stuff in the browser

Crazy computer vision stuff in the browser

Zeno Rocha

April 28, 2015
Tweet

More Decks by Zeno Rocha

Other Decks in Programming

Transcript

  1. CRAZY
    COMPUTER
    vision
    stuff
    in the browser

    View Slide

  2. .html
    .css
    .js

    View Slide

  3. zenorocha

    View Slide

  4. View Slide

  5. View Slide

  6. View Slide

  7. View Slide

  8. View Slide

  9. computer
    vision ?
    what is

    View Slide

  10. Persistence
    of vision

    View Slide

  11. Persistence
    of vision

    View Slide

  12. Motion
    perception

    View Slide

  13. Motion
    perception

    View Slide

  14. vimeo.com/108331968

    View Slide

  15. Face detection

    View Slide

  16. Face recognition

    View Slide

  17. Autonomous robots

    View Slide

  18. Autonomous robots

    View Slide

  19. Virtual reality

    View Slide

  20. Augmented reality

    View Slide

  21. Augmented reality

    View Slide

  22. how to
    do that?

    View Slide

  23. View Slide

  24. View Slide

  25. how to
    do that
    in the browser?

    View Slide

  26. 1. Access the camera

    View Slide

  27. View Slide

  28. Request user’s webcam
    navigator.getUserMedia({
    video: true
    }, onSuccess, onFail);

    View Slide

  29. Request user’s webcam
    navigator.getUserMedia({
    video: true
    }, onSuccess, onFail);

    View Slide

  30. Request user’s webcam
    navigator.getUserMedia({
    video: true
    }, onSuccess, onFail);

    View Slide

  31. 2. Reproduce in a video

    View Slide


  32. View Slide

  33. Get User Media callback
    function onSuccess(stream) {
    var video = document.querySelector('video');
    video.src = URL.createObjectURL(stream);
    video.onloadedmetadata = function(e) {
    // Done
    };
    }

    View Slide

  34. Get User Media callback
    function onSuccess(stream) {
    var video = document.querySelector('video');
    video.src = URL.createObjectURL(stream);
    video.onloadedmetadata = function(e) {
    // Done
    };
    }

    View Slide

  35. Get User Media callback
    function onSuccess(stream) {
    var video = document.querySelector('video');
    video.src = URL.createObjectURL(stream);
    video.onloadedmetadata = function(e) {
    // Done
    };
    }

    View Slide

  36. Get User Media callback
    function onSuccess(stream) {
    var video = document.querySelector('video');
    video.src = URL.createObjectURL(stream);
    video.onloadedmetadata = function(e) {
    // Done
    };
    }

    View Slide

  37. 3. Obtain pixel matrix

    View Slide


  38. View Slide

  39. 4. Tracking algorithms

    View Slide

  40. Request
    accepted
    Request access
    to the camera
    Reproduce

    camera

    Obtain pixel
    matrix

    Tracking
    algorithms
    Result
    Summary

    View Slide

  41. tracking.js

    View Slide

  42. View Slide

  43. Maira
    Bello
    Team
    eduardo
    lundgren
    java
    pablo
    carvalho
    zeno
    rocha

    View Slide

  44. 100% JavaScript!
    Open source
    No dependencies
    Simple and intuitive API
    Automatic setup
    Built-in tracking

    algorithms
    Easily extensible
    High performance
    Unit & performance

    regression tests
    ~ 7 Kb

    View Slide

  45. tracker
    color

    View Slide

  46. Demo

    View Slide

  47. var tracker = new tracking.ColorTracker('magenta');
    tracking.track('#video', tracker, {
    camera: true
    });
    tracker.on('track', function(event) {
    event.data.forEach(function(rect) {
    // Your code goes here
    });
    });

    View Slide

  48. var tracker = new tracking.ColorTracker('magenta');
    tracking.track('#video', tracker, {
    camera: true
    });
    tracker.on('track', function(event) {
    event.data.forEach(function(rect) {
    // Your code goes here
    });
    });

    View Slide

  49. var tracker = new tracking.ColorTracker('magenta');
    tracking.track('#video', tracker, {
    camera: true
    });
    tracker.on('track', function(event) {
    event.data.forEach(function(rect) {
    // Your code goes here
    });
    });

    View Slide

  50. tracker
    object

    View Slide

  51. Demo

    View Slide

  52. var tracker = new tracking.ObjectTracker('face');
    tracking.track('#video', tracker, {
    camera: true
    });
    tracker.on('track', function(event) {
    event.data.forEach(function(rect) {
    // Your code goes here
    });
    });

    View Slide

  53. var tracker = new tracking.ObjectTracker('face');
    tracking.track('#video', tracker, {
    camera: true
    });
    tracker.on('track', function(event) {
    event.data.forEach(function(rect) {
    // Your code goes here
    });
    });

    View Slide

  54. var tracker = new tracking.ObjectTracker('face');
    tracking.track('#video', tracker, {
    camera: true
    });
    tracker.on('track', function(event) {
    event.data.forEach(function(rect) {
    // Your code goes here
    });
    });

    View Slide

  55. tracker
    custom

    View Slide

  56. var MyTracker = function() {
    MyTracker.base(this, 'constructor');
    };
    tracking.inherits(MyTracker, tracking.Tracker);
    MyTracker.prototype.track = function(pxs, width, height) {
    // Your code here
    this.emit('track', {
    // Your code here
    });
    };

    View Slide

  57. var MyTracker = function() {
    MyTracker.base(this, 'constructor');
    };
    tracking.inherits(MyTracker, tracking.Tracker);
    MyTracker.prototype.track = function(pxs, width, height) {
    // Your code here
    this.emit('track', {
    // Your code here
    });
    };

    View Slide

  58. var MyTracker = function() {
    MyTracker.base(this, 'constructor');
    };
    tracking.inherits(MyTracker, tracking.Tracker);
    MyTracker.prototype.track = function(pxs, width, height) {
    // Your code here
    this.emit('track', {
    // Your code here
    });
    };

    View Slide

  59. var MyTracker = function() {
    MyTracker.base(this, 'constructor');
    };
    tracking.inherits(MyTracker, tracking.Tracker);
    MyTracker.prototype.track = function(pxs, width, height) {
    // Your code here
    this.emit('track', {
    // Your code here
    });
    };

    View Slide

  60. var tracker = new tracking.MyTracker();
    tracking.track('#video', tracker, {
    camera: true
    });
    tracker.on('track', function(event) {
    event.data.forEach(function(rect) {
    // Your code goes here
    });
    });

    View Slide

  61. var tracker = new tracking.MyTracker();
    tracking.track('#video', tracker, {
    camera: true
    });
    tracker.on('track', function(event) {
    event.data.forEach(function(rect) {
    // Your code goes here
    });
    });

    View Slide

  62. var tracker = new tracking.MyTracker();
    tracking.track('#video', tracker, {
    camera: true
    });
    tracker.on('track', function(event) {
    event.data.forEach(function(rect) {
    // Your code goes here
    });
    });

    View Slide

  63. who is
    using it?

    View Slide

  64. kdzwinel.github.io/JS-face-tracking-demo

    View Slide

  65. thedevilini.com/#masks

    View Slide

  66. lessons
    learned

    View Slide

  67. 24 fps
    60 fps
    1000ms / 24fps = 41ms per frame

    View Slide

  68. 24 fps
    60 fps
    1000ms / 24fps = 41ms per frame

    View Slide

  69. View Slide

  70. View Slide

  71. View Slide

  72. Int32Array
    Uint16Array
    Float64Array
    Uint8ClampedArray
    [ ]
    Typed arrays

    View Slide

  73. jsperf.com/tracking-js-arrays

    View Slide

  74. jsperf.com/tracking-js-arithmetic

    View Slide

  75. Demo

    View Slide

  76. View Slide

  77. View Slide

  78. web components
    >
    >
    -

    View Slide

  79. how we
    create a
    component
    nowadays?

    View Slide

  80. 1. Never create! Just use a plugin

    View Slide

  81. 2. Copy & paste someone’s code

    View Slide

  82. 3. And hope it works

    View Slide

  83. 3. And hope it works

    View Slide

  84. web
    components

    View Slide

  85. web
    components
    Templates

    View Slide

  86. web
    components
    Templates
    Custom
    Elements

    View Slide

  87. web
    components
    Templates
    Custom
    Elements
    Shadow
    DOM

    View Slide

  88. web
    components
    HTML
    Imports
    Templates
    Custom
    Elements
    Shadow
    DOM

    View Slide

  89. View Slide

  90. var tracker = new tracking.ColorTracker('magenta');
    tracking.track('#video', tracker, {
    camera: true
    });
    tracker.on('track', function(event) {
    event.data.forEach(function(rect) {
    // Your code goes here
    });
    });
    Remember?

    View Slide

  91. Using Web Components…
    target="magenta"
    camera="true"
    ontrack="onResult"

    View Slide

  92. more ?

    View Slide

  93. trackingjs.com

    View Slide

  94. Where
    the magic
    happens
    Your comfort zone

    View Slide

  95. Thanks!
    zenorocha.com
    @zenorocha

    View Slide

  96. Thanks!
    zenorocha.com
    @zenorocha

    View Slide