Codelab: Realtime Face Detection on Android with Firebase

Transcript

1. FACE DETECTION FIREBASE ML KIT CODELAB https://github.com/shredderskelton/facedetectionlab Nick Skelton
2. NICK SKELTON EGO SLIDE [twi] @nshred [git] shredderskelton [web] nickskelton.info
3. AGENDA OUR GOAL FOR THE DAY: BUILD A SIMPLE SNAPCHAT APP ▸ Setup our playground. ▸ Build a simple camera app, take a photo and display the photo to the user. ▸ Add the special sauce ▸ Use Firebase ML Vision Kit to draw a simple box around all faces in the photo. ▸ Get creative ▸ Add a funny overlay, alter the image, do something fun! ▸ Get all SnapChatty. ▸ Do all of the above, except in realtime.
4. ANY COOL IDEAS?
5. SOME THEORY
6. THEORY WHAT IS FIREBASE? ▸ Product interconnectivity ▸ No need for a backend ▸ Scales by default
7. THEORY FIREBASE ML (MACHINE LEARNING) KIT ▸ Offline ▸ Intuitive ▸ Extensible
8. THEORY FACE DETECTION TERMINOLOGY - CONTOURS
9. THEORY FACE DETECTION TERMINOLOGY - LANDMARKS
10. THEORY FACE DETECTION TERMINOLOGY ▸ Classification ▸ Smiling. ▸ Identification Number.
11. THEORY FIREBASE VISION FACE CLASS
12. THEORY GOTCHAS ▸ Some features don’t work well together, esp for real time. eg. contours and ANYTHING ELSE! ▸ Some features work but for one face only (again contours). ▸ … library is still in Beta, so the API can have breaking changes.
13. ENOUGH TALK. LET’S START CODING
14. PART 1. SCAFFOLDING
15. SCAFFOLDING SETUP PERMISSIONS implementation "com.tbruyelle.rxpermissions2:rxpermissions:0.9.5@aar" implementation 'io.reactivex.rxjava2:rxjava:2.2.8' <uses-permission android:name="android.permission.CAMERA"/>
16. SCAFFOLDING SETUP PERMISSIONS private lateinit var rxPermissions: RxPermissions private lateinit var disposable: Disposable override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) rxPermissions = RxPermissions(this) setContentView(R.layout.activity_main) requestPermissions() } override fun onResume() { super.onResume() if (rxPermissions.isGranted(Manifest.permission.CAMERA)) startCamera() } private fun requestPermissions() { disposable = rxPermissions.request(Manifest.permission.CAMERA) .subscribe { granted -> if (!granted) finish() } } private fun startCamera() { TODO("No camera yet") }
17. SCAFFOLDING SETUP CAMERA ▸ Fotoapparat ▸ Very simple and elegant implementation for most simple photo apps. https://github.com/RedApparat/Fotoapparat
18. SCAFFOLDING SETUP CAMERA implementation ‘io.fotoapparat.fotoapparat:library:2.2.0' <io.fotoapparat.view.CameraView android:id="@+id/cameraView" android:layout_width="match_parent" android:layout_height="match_parent" app:layout_constraintBottom_toBottomOf="parent" app:layout_constraintEnd_toEndOf="parent" app:layout_constraintStart_toStartOf="parent" app:layout_constraintTop_toTopOf="parent" /> foto = Fotoapparat( context = this, view = cameraView, scaleType = ScaleType.CenterCrop, lensPosition = front() ) foto.takePicture() .toBitmap() .whenAvailable { TODO("Took photo") }
19. END OF PART 1. GO FORTH AND CREATE YOUR PLAYGROUND
20. PART 2. ADD THE SECRET SAUCE
21. DETECTING A FACE DETECTING FACES. STEP 2
22. DETECTING A FACE SETUP FIREBASE ▸ Go to console.firebase.google.com ▸ Add your App to “Project Settings”
23. DETECTING A FACE SETUP FIREBASE ▸ App Id/Package Id ▸ Don’t worry about SHA1 ▸ Download config ▸ Add Firebase ML Vision Gradle Dependency
24. DETECTING A FACE SOME SEPARATION OF CONCERNS ▸ We are going to setup 2 new interfaces: ▸ Detector ▸ Graphic
25. DETECTING A FACE DETECTOR ▸ What information do you want from Firebase? ▸ Contours? Landmarks? Classification? Speed or Accuracy? ▸ Different detectors for different data. interface FaceDetector { fun process(bitmap: Bitmap, callback: (Bitmap) -> Unit) }
26. DETECTING A FACE GRAPHICS ▸ What are we going to draw on the photo? Boxes? Dots? Funny sunglasses? ▸ Separate the drawing from the detection ▸ Original photo + information about the faces detected => New photo interface Graphic { fun draw(bitmap: Bitmap, faces: List<FirebaseVisionFace>): Bitmap }
27. DETECTING A FACE OUR FIRST DETECTOR ▸ Draw a box around all the faces in the image.
28. DETECTING A FACE BOX DETECTOR
29. DETECTING A FACE MY FIRST DETECTOR ▸ Draw a box around all the faces in the image. ▸ Configure the Firebase options in you detector: private val options = FirebaseVisionFaceDetectorOptions.Builder() .setPerformanceMode(FirebaseVisionFaceDetectorOptions.FAST) .build() private val detector: FirebaseVisionFaceDetector = FirebaseVision.getInstance().getVisionFaceDetector(options)
30. DETECTING A FACE OUR FIRST GRAPHIC ▸ Simple use case: draw a box around all the faces in the image. ▸ Draw boxes around each face detected override fun draw(bitmap: Bitmap, faces: List<FirebaseVisionFace>): Bitmap { val canvas = Canvas(bitmap) faces.forEach { face -> canvas.drawRect(face.boundingBox, whitePaint) } return bitmap }
31. DETECTING A FACE FIREBASE VISION FACE CLASS
32. DETECTING A FACE FIREBASE VISION FACE CLASS
33. DETECTING A FACE END OF PART 2. DETECT AWAY
34. PART 3. THE FUN PART.
35. FUN! SOME IDEAS: CREATE A NEW DETECTOR AND GRAPHIC THAT… ▸ Outlines the features of a face. ▸ Draws a funny graphic over faces. ▸ Alters the faces themselves.
36. PART 4. KEEPING IT REAL-TIME
37. PERFORMANCE PERFORMANCE PERFORMANCE
39. REAL TIME UI THREAD ▸ Don’t render twice. ▸ Render the overlay and input image in a single step. ▸ Use SurfaceView ▸ Use Camera or Camera2 ▸ See the Firebase Android Quickstart repository
41. REAL TIME THROTTLING ▸ Dropped frame-rate. ▸ Keep your threads under control. ▸ Reduce the camera frame rate. ▸ Detector options.
42. REAL TIME CAMERA TWEAKING ▸ Resolution. ▸ Frame rate. ▸ Filling the screen. ▸ Buffers. ▸ Camera2?
43. REAL TIME DETECTOR PARAMETERS ▸ Experiment ▸ Use minimum specs possible ▸ Avoid certain combinations
44. IMPLEMENT YOUR OWN PROCESSOR/GRAPHIC COMBINATION
45. BYE BYE DENMARK!