ML을 여행하는 안드로이드 히치하이커를 위한 안내서

DevFest Pangyo 2019

Transcript

1. MLਸ ৈ೯ೞח ൤஖ೞ੉ழܳ ਤೠ উղࢲ - ML Vision - উࢿਊ, ֎੉ߡਢొ @fornewid @SOUP
2. Ҋ޹੄ द੘
3. ൤஖ೞ੉ழ੄ ࢤп ✨
4. ൤஖ೞ੉ழ੄ അप ✨
5. On-device ML ٜ݅ӝ ݾ಴
6. On-device ML ࢎਊೞӝ ݾ಴
7. Take a photo Use magic Save a image
8. CameraX Use magic Save a image
9. M L A N D R O I D H I T C H H I K E R Take a photo
10. 1. from Camera
11. Camera1 API (API 21 ޷݅) Camera2 API (API 21 ੉࢚) CameraX API (API 21 ੉࢚, alpha) 1. from Camera
12. 01010101010101 010101 010101 01010101010101
13. Opt 1. CameraX View in ⍺!
14. CameraX View in ⍺ • app/build.gradle implementation 'androidx.camera:camera-view:1.0.0-alpha03'
15. CameraX View in ⍺ • In Layout XML: <androidx.camera.view.CameraView ... app:scaleType="centerCrop|centerInside" app:lensFacing="back|front|none" app:captureMode="image|video|mixed" app:flash="off|on|auto" app:pinchToZoomEnabled="true|false" />
16. CameraX View in ⍺ • In Kotlin: // In Activity cameraView.bindToLifecycle(this) // In Fragment cameraView.bindToLifecycle(viewLifecycleOwner)
17. CameraX View in ⍺ • ImageCapture cameraView.takePicture(Executor, OnImageCapturedListener() { override fun onCaptureSuccess( image: ImageProxy, rotationDegrees: Int) {...} }) // or cameraView.takePicture(File, Executor, OnImageSavedListener { override fun onImageSaved(file: File) {...} })
18. CameraX View in ⍺ • FrameProcessor ❌
19. CameraX View in custom
20. CameraX View in custom • FrameProcessor cameraView.setAnalyzer(object : ImageAnalysis.Analyzer { override fun analyze(image: ImageProxy, rotationDegrees: Int) { val mediaImage: Image? = image.image ... } })
21. Opt 2. natario1 / CameraView!
22. natario1 / CameraView • app/build.gradle implementation 'com.otaliastudios:cameraview:2.4.0' Link: https://github.com/natario1/CameraView
23. natario1 / CameraView • In Layout XML: <com.otaliastudios.cameraview.CameraView ... app:cameraFacing="back|front" app:cameraMode="picture|video" app:cameraFlash="on|auto|torch|off" app:cameraGesturePinch="none|zoom|exposureCorrection" />
24. natario1 / CameraView • In Kotlin: // In Activity cameraView.setLifecycleOwner(this) // In Fragment cameraView.setLifecycleOwner(viewLifecycleOwner)
25. natario1 / CameraView • ImageCapture cameraView.addCameraListener(object : CameraListener() { @UiThread override fun onPictureTaken(result: PictureResult) { @WorkerThread result.toBitmap(object : BitmapCallback { @UiThread override fun onBitmapReady(bitmap: Bitmap?) { ... } }) } }) cameraView.takePicture()
26. natario1 / CameraView • FrameProcessor cameraView.addFrameProcessor(object : FrameProcessor { @WorkerThread override fun process(frame: Frame) { val data: ByteArray = frame.data val width: Int = frame.size.width val height: Int = frame.size.height val format: Int = frame.format // android.graphics.ImageFormat val rotation: Int = frame.rotation // 0, 90, 180, 270 } })
27. Wrap-up: Camera Source CameraX Bitmap android.graphics CameraView ByteArray byte[] File java.io Image android.media
28. • FirebaseVisionImageܳ ੉ਊೞৈ औѱ ߸ജೡ ࣻ ੓׮. ੉੹ ױ҅ীࢲ оઉৡ ؘ੉ఠܳ Bitmap ഋక۽ ߸ജ೧ঠ ೠ׮. 2. Any to Bitmap Bitmap ByteArray File Image Bitmap
29. FirebaseVisionImage
30. • app/build.gradle FirebaseVisionImage implementation 'com.google.firebase:firebase-ml-vision:24.0.0' Link: https://firebase.google.com/.../firebase/ml/vision/common/FirebaseVisionImage
31. • ByteArray to Bitmap FirebaseVisionImage val bitmap = FirebaseVisionImage.fromByteArray( frame.data, // ByteArray FirebaseVisionImageMetadata.Builder() .setFormat(FirebaseVisionImageMetadata.IMAGE_FORMAT_NV21) .setWidth(frame.size.width) .setHeight(frame.size.height) .setRotation(FirebaseVisionImageMetadata.ROTATION_90) .build()) .bitmap
32. • Uri to Bitmap FirebaseVisionImage val bitmap = FirebaseVisionImage .fromFilePath(context, uri) // Context, Uri .bitmap
33. FirebaseVisionImage • Media Image to Bitmap val bitmap = FirebaseVisionImage .fromMediaImage( image, // android.media.Image FirebaseVisionImageMetadata.ROTATION_90 ) .bitmap
34. Done!
35. CameraX Use magic Save a image
36. CameraX Use magic Save a image
37. CameraX TF Lite / ML Kit Save a image
38. M L A N D R O I D H I T C H H I K E R Use magic
39. Opt 1. Tensorflow Lite
40. Model TFLite
41. ) Interpreter (
42. ) Interpreter (
43. ) Interpreter ( Bitmap
44. ) Interpreter ( Bitmap Output
45. ) Interpreter ( Array<…<FloatArray>…> Array<…<FloatArray>…>
46. Tensorflow Lite // module.gradle android { aaptOptions { noCompress "tflite" } } dependencies { implementation 'org.tensorflow:tensorflow-lite:1.14.0' }
47. Tensorflow Lite // TF Lite Model assets/ detect.tflite
48. Tensorflow Lite val model = context.loadModelFile(" detect.tflite ") @Throws(IOException::class) fun Context.loadModelFile(path: String): ByteBuffer { return assets.openFd(path).run { FileInputStream(fileDescriptor).channel.map( FileChannel.MapMode.READ_ONLY, startOffset, declaredLength ) } }
49. Tensorflow Lite val model = context.loadModelFile(" detect.tflite ") val options = Interpreter.Options().apply { setNumThreads(2) } this.interpreter = Interpreter(model, options)
50. Object Detection Image Segmentation Hosted models.. Pose Estimation Link: https://www.tensorflow.org/lite/guide/hosted_models
51. Object Detection Pose Estimation Image Segmentation
52. Array<…<FloatArray>…> Array<…<FloatArray>…> Object Detection
53. Array<Array<Array<FloatArray>>>> Array<…<FloatArray>…> Image ࣻ Width Height Channel ࣻ Object Detection
54. Array<Array<Array<FloatArray>>>> Array<…<FloatArray>…> 1 300 300 3 Object Detection
55. Link: https://www.tensorflow.org/lite/performance/post_training_quantization float32 float16 int byte • ݽ؛ ੿ഛب (Ѣ੄ زੌ) • ݽ؛ ௼ӝ (хࣗ) • CPU ߂ ೞ٘ਝয оࣘӝ ૑োदр (хࣗ) Quantization
56. Link: https://www.tensorflow.org/lite/performance/post_training_quantization 0x112233 • ݽ؛ ੿ഛب (Ѣ੄ زੌ) • ݽ؛ ௼ӝ (хࣗ) • CPU ߂ ೞ٘ਝয оࣘӝ ૑োदр (хࣗ) Quantization
57. Link: https://www.tensorflow.org/lite/performance/post_training_quantization 0x112233 0x11 [0, 255] 0x22 [0, 255] 0x33 [0, 255] 17 34 51 byte byte byte • ݽ؛ ੿ഛب (Ѣ੄ زੌ) • ݽ؛ ௼ӝ (хࣗ) • CPU ߂ ೞ٘ਝয оࣘӝ ૑োदр (хࣗ) Quantization
58. Array<Array<Array<FloatArray>>>> Array<…<FloatArray>…> 1 300 300 3 Object Detection
59. Array<Array<Array<ByteArray>>>> Array<…<FloatArray>…> 1 300 300 3 Object Detection
60. Array<…<FloatArray>…> Array<…<FloatArray>…> Object Detection
61. Array<Array<FloatArray>> Array<FloatArray> Array<FloatArray> FloatArray 0 1 2 3 1 1 1 N 4 N N N Object Detection
62. Object Detection: Input // Input (Float model) fun Bitmap.toByteBuffer(): ByteBuffer { val inputSize = FLOAT_TYPE_SIZE * inputWidth * inputHeight * PIXEL_SIZE return ByteBuffer.allocateDirect(inputSize).apply { order(ByteOrder.nativeOrder()) val pixels = IntArray(inputWidth * inputHeight) getPixels(pixels, 0, width, 0, 0, width, height) for (pixel in pixels) { putFloat((pixel shr 16 and 0xFF) / IMAGE_MEAN - IMAGE_OFFSET) // R putFloat((pixel shr 8 and 0xFF) / IMAGE_MEAN - IMAGE_OFFSET) // G putFloat((pixel and 0xFF) / IMAGE_MEAN - IMAGE_OFFSET) // B } } } // const const val IMAGE_MEAN = 128f const val IMAGE_OFFSET = 1f const val FLOAT_TYPE_SIZE = 4 const val PIXEL_SIZE = 3
63. Object Detection: Input // Input (Quantized model) fun Bitmap.toByteBuffer(): ByteBuffer { val inputSize = inputWidth * inputHeight * PIXEL_SIZE return ByteBuffer.allocateDirect(inputSize).apply { order(ByteOrder.nativeOrder()) val pixels = IntArray(inputWidth * inputHeight) getPixels(pixels, 0, width, 0, 0, width, height) for (pixel in pixels) { put((pixelValue shr 16 and 0xFF).toByte()) // R put((pixelValue shr 8 and 0xFF).toByte()) // G put((pixelValue and 0xFF).toByte()) // B } } } // const const val IMAGE_MEAN = 128f const val IMAGE_OFFSET = 1f const val PIXEL_SIZE = 3
64. Object Detection: Output const val NUM_DETECTIONS = 10 // Output val outputMap = hashMapOf<Int, Any>( 0 to Array(1) { Array(NUM_DETECTIONS) { FloatArray(4) } }, 1 to Array(1) { FloatArray(NUM_DETECTIONS) }, 2 to Array(1) { FloatArray(NUM_DETECTIONS) }, 3 to FloatArray(1) )
65. Object Detection val input = inputImage.toByteBuffer() val inputArray = arrayOf<Any>(input) val outputMap = hashMapOf<Int, Any> (...) val interpreter = Interpreter(...) interpreter.runForMultipleInputsOutputs(inputArray, outputMap)
66. Opt 2. ML Kit for Firebase
68. ML Kit: Custom Model • app/build.gradle implementation 'com.google.firebase:firebase-ml-model-interpreter:22.0.0'
69. ML Kit: Custom Model val localModel = FirebaseCustomLocalModel.Builder() .setAssetFilePath("model_name.tflite") .build() val options = FirebaseModelInterpreterOptions.Builder(localModel).build() val interpreter = FirebaseModelInterpreter.getInstance(options)
70. ML Kit: Custom Model val inputs = FirebaseModelInputs.Builder() .add(bitmap.toByteBuffer()) .build() val inputOutputOptions = FirebaseModelInputOutputOptions.Builder() .setInputFormat( 0, FirebaseModelDataType.BYTE, intArrayOf(1, 300, 300, 3)) .setOutputFormat( 0, FirebaseModelDataType.BYTE, intArrayOf(1, 300, 300, 3)) .build()
71. ML Kit: Custom Model interpreter.run(inputs, inputOutputOptions) .addOnSuccessListener { result -> val output: Array<Array<Array<FloatArray>>> = result.getOutput(0) ... } .addOnFailureListener { e -> ... }
72. Done!
73. CameraX TF Lite / ML Kit Save a image
74. CameraX TF Lite / ML Kit Save a image
75. CameraX TF Lite / ML Kit Scoped Storage
76. M L A N D R O I D H I T C H H I K E R Save a image
77. Pre- Q • app/AndroidManifest.xml <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
78. val bitmap: Bitmap = ... val pictures = Environment.getExternalStoragePublicDirectory( Environment.DIRECTORY_PICTURES) val imageFilePath = File(pictures, "file_name.png").absolutePath val out = FileOutputStream(imageFilePath) bitmap.compress(Bitmap.CompressFormat.PNG, 100, out) out.flush() out.close() val uri: Uri? = context.contentResolver.insert( MediaStore.Images.Media.EXTERNAL_CONTENT_URI, contentValuesOf( ImageColumns.DATA to imageFilePath, ImageColumns.DISPLAY_NAME to imageFileName Pre- Q
79. Environment.DIRECTORY_PICTURES) val imageFilePath = File(pictures, "file_name.png").absolutePath val out = FileOutputStream(imageFilePath) bitmap.compress(Bitmap.CompressFormat.PNG, 100, out) out.flush() out.close() val uri: Uri? = context.contentResolver.insert( MediaStore.Images.Media.EXTERNAL_CONTENT_URI, contentValuesOf( ImageColumns.DATA to imageFilePath, ImageColumns.DISPLAY_NAME to imageFileName ) ) // Show Notification! Pre- Q
80. But, deprecated in Q
81. Option 1. Avoid temporarily • app/AndroidManifest.xml <application android:requestLegacyExternalStorage="true" ... Link: https://developer.android.com/.../scoped-storage#opt-out-of-filtered-view
82. • Before: Option 2. Scoped Storage Bitmap External Storage File MediaStore Content Values
83. • After: Option 2. Scoped Storage Bitmap External Storage File MediaStore Content Values
84. • After: Option 2. Scoped Storage Bitmap External Storage File MediaStore Content Values
85. • After: Option 2. Scoped Storage Bitmap External Storage File MediaStore Content Values Pending
86. val bitmap: Bitmap = ... val resolver = context.contentResolver val uri = resolver.insert( MediaStore.Images.Media.EXTERNAL_CONTENT_URI, contentValuesOf( MediaColumns.DISPLAY_NAME to "file_name.png", MediaColumns.IS_PENDING to 1, MediaColumns.DATE_EXPIRES to ..., MediaColumns.RELATIVE_PATH to Environment.DIRECTORY_PICTURES ) ) val out = resolver.openOutputStream(uri) bitmap.compress(Bitmap.CompressFormat.PNG, 100, out) out.flush() out.close() Option 2. Scoped Storage
87. MediaColumns.IS_PENDING to 1, MediaColumns.DATE_EXPIRES to ..., MediaColumns.RELATIVE_PATH to Environment.DIRECTORY_PICTURES ) ) val out = resolver.openOutputStream(uri) bitmap.compress(Bitmap.CompressFormat.PNG, 100, out) out.flush() out.close() resolver.update(uri, contentValuesOf( MediaColumns.IS_PENDING to 0, MediaColumns.DATE_EXPIRES to null ), null, null) Link: https://android-10/android/provider/MediaStore.java#PendingParams Option 2. Scoped Storage
88. Done!
89. • ஠ݫۄо ೙ਃೞ׮ݶ CameraXܳ ࢎਊ೧ࠁࣁਃ. • Pre-L OSਸ ૑ਗೠ׮ݶ, natario1/CameraViewܳ ୶ୌ೤פ׮. • FirebaseVisionImageܳ ࢎਊೞৈ Bitmapਵ۽ औѱ ߄Ե ࣻ ੓णפ׮. • TF Liteܳ ࢎਊೠ׮ݶ, ML Kit - Custom Model ࢎਊਸ Ҋ۰೧ࠁࣁਃ. • Qীࢲח ੉޷૑ܳ ੷੢ೡ ٸ, Scoped Storageܳ Ҋ۰ೞࣁਃ. Summary
90. хࢎ೤פ׮!