How the Browser Sees Your Face

Snapchat's Filters and Apple's Animoji are one of the most famous examples of computer vision in popular applications. Based on complex algorithms, computer vision can be intimidating at the first time. Luckily for us, Web Developers, processing images and face detection are much easier to work with than before.

Mainly a domain of native apps, face detection is now available in our browsers. Thanks to modern web APIs and libraries, we are now able to process the image in real time on the front-end side. With face detection, you can go even a one step further and make the browser to recognize your face!

This talk will give you a brief overview of face detection concepts. I will show you some popular methods of detecting the face using the web camera and how to use them in your applications.

Presented during meet.js Summit 2018 in Poznań

Transcript

1. How the Browser Sees Your Face Meet.js Summit 2018 Poznań
2. About me • Front-end Developer in SwingDev • I love to play with browser APIs • I like to go on vacation sometimes https://github.com/mciastek https://medium.com/@mciastek https://twitter.com/mciastek Mirek Ciastek
3. Front-end is great!
4. Face Detection Face Recognition 3D games Push Notifications Real time connections Cryptocurrencies Databases Machine Learning IoT VR / AR Computer Vision
5. Face Recognition Face Detection Face Detection Face Recognition 3D games Push Notifications Real time connections Cryptocurrencies Databases Machine Learning IoT VR / AR Computer Vision !==
6. Face Detection
7. Face recognition
8. Face Detection
12. How it works Get input Normalize Analyze Result
13. Face Detection Methods • Knowledge-based • Template matching • Appearance-based • Feature-based
15. Face Detection in JS
16. How to do it in JS • Node.js + OpenCV • Tracking.js • face-api.js • Native Face Detection
17. Node.js + OpenCV • Library - opencv4nodejs • OpenCV 3.x • CascadeClassifier (Viola-Jones) with HAAR features • Web Cam stream by WebSocket
19. const cv = require('opencv4nodejs') const DEFAULT_OPTIONS = { scaleFactor: 1.1, minNeighbors: 10 } class ImageProcessor { constructor () { this.classifier = new cv.CascadeClassifier(cv.HAAR_FRONTALFACE_ALT2) } async process (data, options) { const base64String = data.replace('data:image/jpeg;base64', '') const decoded = cv.imdecode(Buffer.from(base64String, 'base64')) const start = Date.now() try { const { objects } = await this.classifier.detectMultiScaleAsync( decoded.bgrToGray(), options || DEFAULT_OPTIONS ) return { objects, time: Date.now() - start } } catch (error) { throw error } } }
20. getPixels (source) { this.pixelsCanvasCtx.drawImage( source, 0, 0, this.pixelsCanvas.width, this.pixelsCanvas.height ) return this.pixelsCanvas.toDataURL('image/jpeg') } process (source) { super.process() const encodedPixels = this.getPixels(source) this.socket.emit('frame', { data: encodedPixels, options: DETECTION_OPTIONS }) }
21. Tracking.js • Viola-Jones classifier (the same as in OpenCV) • Uses HAAR features detections
22. getResults (pixels) { return this.classifiers.map(classifier => tracking.ViolaJones.detect( pixels, this.overlayWidth, this.overlayHeight, defaults.initialScale, defaults.scaleFactor, defaults.stepSize, defaults.edgesDensity, classifier ) ) } prepare () { return new Promise((resolve, reject) => { try { this.classifiers = CLASSIFIERS.map( name => tracking.ViolaJones.classifiers[name] ) resolve() } catch (error) { reject(error) } }) }
23. process (source) { super.process() const pixels = this.getPixels(source) const [face] = this.getResults(pixels) this.options.onUpdate() face.forEach(coords => { drawRect(this.overlayCtx, coords, this.drawOptions) }) }
25. face-api.js • Tensorflow.js under the hood • Pre-trained model • MTCNN (Multi-task Cascaded Convolutional Neural Networks) • 5 Point Face Landmarks • Has other models like: SSD Mobilenet v1, Tiny Yolo v2, Face Recognition and 68 Point Face Landmark
28. const modelFetcher = () => { let loaded = false return () => { if (!loaded) { return faceapi.loadMtcnnModel('/') .then(() => { loaded = true return loaded }) } return Promise.resolve() } } const fetchModel = modelFetcher() class FaceApiAdapter extends Adapter { prepare () { return fetchModel() } }
29. async process (source) { super.process() const { results } = await faceapi.nets.mtcnn.forwardWithStats( source, MTCNN_PARAMS ) this.options.onUpdate() if (results) { results.forEach(({ faceDetection, faceLandmarks }) => { if (faceDetection.score < MIN_CONFIDENCE) { return } faceapi.drawDetection( this.overlayEl, faceDetection.forSize(this.overlayWidth, this.overlayHeight), { ...this.drawOptions, withScore: false } ) faceapi.drawLandmarks( this.overlayEl, faceLandmarks.forSize(this.overlayWidth, this.overlayHeight), { lineWidth: 4, color: 'red' } ) }) } }
30. Native Face Detection • Shape Detection API • Can detect faces, texts and barcodes • Native implementation of face detectors • E.g. VNDetectFaceLandmarksRequest for Apple
32. How to enable in Chrome 69+ • Go to: chrome://flags/#enable-experimental-web- platform-features • Enable the experimental flag and reload browser • Create instance of window.FaceDetector
33. async process (source) { super.process() const scale = getSourceScale(source) const faces = await this.detector.detect(source) this.options.onUpdate() faces.forEach(face => { const { width, height, top, left } = face.boundingBox drawRect( this.overlayCtx, { x: left * scale, y: top * scale, width: width * scale, height: height * scale }, this.drawOptions ) }) }
34. Benchmark
37. Face detections Detections number 0 2,5 5 7,5 10 face-api tracking.js Native API OpenCV
38. Detection time Elapsed time (ms) 0 100 200 300 400 Image 1 2 3 4 5 6 7 8 9 10 face-api tracking.js Native API OpenCV
39. Available online Go to https://face-detection-benchmark.herokuapp.com/ Code https://github.com/mciastek/face-detection-benchmark
40. Further reading 1/2 • Realtime JavaScript Face Tracking and Face Recognition using face- api.js’ MTCNN Face Detector by Vincent Mühler • Face Detection using Haar Cascades • Official repository of Shape Detection API • A Review on Face Detection Methods by Dr Qaim Mehdi Rizvi • Face detection & recognition with Javascript by Antonio Russo • Let’s play with Chrome’s Face Detection API by João Miguel Cunha
41. Further reading 2/2 • Snapchat’s Filters: How computer vision recognizes your face by James Le • Face Detection For Beginners by Divyansh Dwivedi • Viola–Jones object detection framework • Learning where you are looking at (in the browser) by Max Schumacher • A Study of Various Face Detection Methods by Varsha Gupta and Dipesh Sharma
43. https://github.com/mciastek https://medium.com/@mciastek https://twitter.com/mciastek Thank you!