COSCUP 2017 - Face off

How to build a high-performance and accuracy face-detection camera app.

Transcript

1. Face/off How to build a high-performance and accuracy face-detection camera app. Boy, Prada @PicCollage
2. PicCollage 160,000,000 Downloads
3. Why Face Detection is important for us?
4. Snapchat Snow Because it’s fun
5. Most of photos are selfie
6. Face = ROI Region of Interest
7. PicCollage Before After
8. 空氣照相機
9. 單⾝身不寂寞 Never Alone https://play.google.com/store/apps/details?id=com.cardinalblue.android.neveralone
10. BigBite https://itunes.apple.com/us/app/big-bite-speed-eating-contest/id1239133098?mt=8
11. Overall Architecture Camera Operation Face Detection Render 33ms Camera Operation Face Detection Render 33ms
12. Assumption only discuss client-side solution.
13. Face Detection on Android? Camera Operation Face Detection Render 33ms
14. android.media.FaceDetector http://developer.android.com/intl/zh-tw/reference/android/media/FaceDetector.html android.media.FaceDetector.Face float getFaceDistance() void getMidPoint(PointF) x
15. android.hardware.Camera .FaceDetectionListener http://developer.android.com/intl/zh-tw/reference/android/hardware/Camera.FaceDetectionListener.html Camera.Face PointF leftEye PointF rightEye PointF mouth Rect rect x x x
16. Google Mobile Vision SDK
17. Landmark http://android-developers.blogspot.tw/2015/08/face-detection-in-google-play-services.htm gms.vision.face.Face List<Landmark>getLandmarks()
18. Smile Mode gms.vision.face.Face float getIsLeftEyeOpenProbability() float getIsRightEyeOpenProbability() float getIsSmilingProbability()
19. Face Rotation gms.vision.face.Face float getEulerY() float getEulerZ()
20. It’s Beta Version Samsung S4 didn’t work https://github.com/googlesamples/android-vision/issues/39 https://github.com/googlesamples/android-vision/issues/21 Preview/Capture Resolution is bad It depends on Google Play Service some china devices, or kindle fire might not work
21. Open CV
22. Sample Code cascadeFile = new File(cascadeDir, "lbpcascade_frontalface.xml"); nativeDetector = new DetectionBasedTracker( cascadeFile.getAbsolutePath(), 0); nativeDetector.setMinFaceSize(FACE_SIZE); MatOfRect faces = new MatOfRect(); nativeDetector.detect(PREVIEW_BUFFER, faces); Rect[] facesArray = faces.toArray(); https://github.com/bearprada/android_camera_with_opencv_sample
23. You lost one man
24. Nobody Blink
25. 阿飄偵測器?
26. OpenCV for Android
27. Dlib modern C++ toolkit containing machine learning algorithms
28. Dlib We need more landmarks for build a mini- game with your face.
29. Dlib It is 68 landmarks! how to integrate with Dlib in your android project http://slides.com/boywang/face-landmarks-in-your-android-app/ http://blog.dlib.net/2014/08/real-time-face-pose-estimation.html
30. Dlib
31. Dlib : experiment result
32. Dlib : improve finding face rectangle
33. Summary media.Face Camera.Face gms.vision .face Open CV Dlib footprint 0MB 0MB ~85KB ~20MB ~80MB camera integration x o o o x easy to use ⭐⭐⭐ ⭐⭐⭐ ⭐⭐ ⭐ ⭐ min-SDK version 1 14 9 9 9 Face Information ⭐ ⭐ ⭐⭐ ⭐⭐⭐ ⭐⭐⭐ Speed(rect) N/a ⭐ ⭐ ⭐⭐ ⭐ Speed (landmarks) ⭐ ⭐ ⭐⭐ n/a ⭐⭐⭐
34. How to pick-up the face detection api? 1. only do want the face rectangle > android.media.FaceDetector 2. want to add something on the eyes > android.hardware.Camera.FaceDetectionListener 3. want to add something on the mouse/cheek > mobile vision API 4. want to detect the user's face emotion/activity. > OpenCV or Dlib
35. Camera Camera Operation Face Detection Render 33ms
36. What's wrong with the Android Camera? • Too many configuration. • Preview buffer is rotated. • It’s YUV format! • Don't have good update mechanism for preview buffer.
37. How to implement the photo burst feature on Android?
38. 3rd-party libraries solve configuration problem • CWCA-Camera • https://github.com/bearprada/cwac-camera • Material Camera • https://github.com/afollestad/material-camera ref : https://academy.realm.io/posts/tools-and-libraries-for-common-android-problems/
39. How to convert YUV2RGB • Java • C++/C + JNI • RenderScript (code) API 17 • YUVImage (code) API 8
40. Why Reactive? • A lot of callbacks and member fields just for using camera. • What if detecting faces and landmarks takes too long and new frame is delivered? (back-pressure)
41. Callback Hell
42. At least have three member fields for safely releasing resource in onPause()
43. Goal for more simple and clean code 1 2 3 4 React to UI
44. Convert setSurfaceTextureListener to REACTIVE way Step 1 Step 2 Step 3
45. Convert setSurfaceTextureListener to REACTIVE way (cont.) Step 4 Step 5 Step 6
46. If the permissions are all granted. The exception would be propagated to the last observer.
48. Surface available - TextureViewObservable Open camera - Camera2Observable Create capture session - Camera2CaptureSessionObservable Detect faces and landmarks - Camera2ImageReaderObservable
49. Result • Business logic for opening camera is distributed to TextureViewObservable.java, Camera2Observable.java, Camera2CaptureSessionObservable.java and Camera2ImageReaderObservable.java.
 Observable classes are reusable. • Less member fields in the Activity/Fragment for camera. • ObservableTransformer gives the picture of sequence flow.
50. Backpressure Sol#1 Sol#2 What if detecting faces and landmarks takes too long and new frame is delivered?
51. Just one line
52. Live Demo
53. Future work • Improve the rendering performance by OpenGL. • Parallel computing • Draw the 3D objects https://github.com/nekocode/ CameraFilter Camera Operation Face Detection Render 33ms
54. Thanks
55. Dlib DEMO https://www.youtube.com/watch?v=9G7jTN_MnmE