Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How the Browsers Sees Your Face (microConf)

mciastek
October 25, 2018

How the Browsers Sees Your Face (microConf)

Snapchat's Filters and Apple's Animoji are one of the most famous examples of computer vision in popular applications. Based on complex algorithms, computer vision can be intimidating at the first time. Luckily for us, Web Developers, processing images and face detection are much easier to work with than before.

Mainly a domain of native apps, face detection is now available in our browsers. Thanks to modern web APIs and libraries, we are now able to process the image in real time on the front-end side. With face detection, you can go even a one step further and make the browser to recognize your face!

This talk will give you a brief overview of face detection concepts. I will show you some popular methods of detecting the face using the web camera and how to use them in your applications.

Presented during microConf, 25th of October 2018

mciastek

October 25, 2018
Tweet

More Decks by mciastek

Other Decks in Technology

Transcript

  1. About me • 4+ years of experience • projects in

    React, Angular (1&2+), Ember and Vue.js • I love to play with browser APIs
  2. Face Detection Face Recognition 3D games Push Notifications Real time

    connections Cryptocurrencies Databases Machine Learning IoT VR / AR Computer Vision
  3. Face Recognition Face Detection Face Detection Face Recognition 3D games

    Push Notifications Real time connections Cryptocurrencies Databases Machine Learning IoT VR / AR Computer Vision !==
  4. How to do it in JS • Node.js + OpenCV

    • Tracking.js • face-api.js • Native Face Detection
  5. Node.js + OpenCV • Library - opencv4nodejs • OpenCV 3.x

    • CascadeClassifier (Viola-Jones) with HAAR features • Web Cam stream by WebSocket
  6. const cv = require('opencv4nodejs') const DEFAULT_OPTIONS = { scaleFactor: 1.1,

    minNeighbors: 10 } class ImageProcessor { constructor () { this.classifier = new cv.CascadeClassifier(cv.HAAR_FRONTALFACE_ALT2) } async process (data, options) { const base64String = data.replace('data:image/jpeg;base64', '') const decoded = cv.imdecode(Buffer.from(base64String, 'base64')) const start = Date.now() try { const { objects } = await this.classifier.detectMultiScaleAsync( decoded.bgrToGray(), options || DEFAULT_OPTIONS ) return { objects, time: Date.now() - start } } catch (error) { throw error } } }
  7. getPixels (source) { this.pixelsCanvasCtx.drawImage( source, 0, 0, this.pixelsCanvas.width, this.pixelsCanvas.height )

    return this.pixelsCanvas.toDataURL('image/jpeg') } process (source) { super.process() const encodedPixels = this.getPixels(source) this.socket.emit('frame', { data: encodedPixels, options: DETECTION_OPTIONS }) }
  8. getResults (pixels) { return this.classifiers.map(classifier => tracking.ViolaJones.detect( pixels, this.overlayWidth, this.overlayHeight,

    defaults.initialScale, defaults.scaleFactor, defaults.stepSize, defaults.edgesDensity, classifier ) ) } prepare () { return new Promise((resolve, reject) => { try { this.classifiers = CLASSIFIERS.map( name => tracking.ViolaJones.classifiers[name] ) resolve() } catch (error) { reject(error) } }) }
  9. process (source) { super.process() const pixels = this.getPixels(source) const [face]

    = this.getResults(pixels) this.options.onUpdate() face.forEach(coords => { drawRect(this.overlayCtx, coords, this.drawOptions) }) }
  10. face-api.js • Tensorflow.js under the hood • Pre-trained model •

    MTCNN (Multi-task Cascaded Convolutional Neural Networks) • 5 Point Face Landmarks • Has other models like: SSD Mobilenet v1, Tiny Yolo v2, Face Recognition and 68 Point Face Landmark
  11. const modelFetcher = () => { let loaded = false

    return () => { if (!loaded) { return faceapi.loadMtcnnModel('/') .then(() => { loaded = true return loaded }) } return Promise.resolve() } } const fetchModel = modelFetcher() class FaceApiAdapter extends Adapter { prepare () { return fetchModel() } }
  12. async process (source) { super.process() const { results } =

    await faceapi.nets.mtcnn.forwardWithStats( source, MTCNN_PARAMS ) this.options.onUpdate() if (results) { results.forEach(({ faceDetection, faceLandmarks }) => { if (faceDetection.score < MIN_CONFIDENCE) { return } faceapi.drawDetection( this.overlayEl, faceDetection.forSize(this.overlayWidth, this.overlayHeight), { ...this.drawOptions, withScore: false } ) faceapi.drawLandmarks( this.overlayEl, faceLandmarks.forSize(this.overlayWidth, this.overlayHeight), { lineWidth: 4, color: 'red' } ) }) } }
  13. Native Face Detection • Shape Detection API • Can detect

    faces, texts and barcodes • Native implementation of face detectors • E.g. VNDetectFaceLandmarksRequest for Apple
  14. How to enable in Chrome 69+ • Go to: chrome://flags/#enable-

    experimental-web-platform- features • Enable the experimental flag and reload browser • Create instance of window.FaceDetector
  15. async process (source) { super.process() const scale = getSourceScale(source) const

    faces = await this.detector.detect(source) this.options.onUpdate() faces.forEach(face => { const { width, height, top, left } = face.boundingBox drawRect( this.overlayCtx, { x: left * scale, y: top * scale, width: width * scale, height: height * scale }, this.drawOptions ) }) }
  16. Detection time Elapsed time (ms) 0 100 200 300 400

    Image 1 2 3 4 5 6 7 8 9 10 face-api tracking.js Native API OpenCV
  17. Further reading 1/2 • Realtime JavaScript Face Tracking and Face

    Recognition using face-api.js’ MTCNN Face Detector by Vincent Mühler • Face Detection using Haar Cascades • Official repository of Shape Detection API • A Review on Face Detection Methods by Dr Qaim Mehdi Rizvi • Face detection & recognition with Javascript by Antonio Russo • Let’s play with Chrome’s Face Detection API by João Miguel Cunha
  18. Further reading 2/2 • Snapchat’s Filters: How computer vision recognizes

    your face by James Le • Face Detection For Beginners by Divyansh Dwivedi • Viola–Jones object detection framework • Learning where you are looking at (in the browser) by Max Schumacher • A Study of Various Face Detection Methods by Varsha Gupta and Dipesh Sharma