ML을 여행하는 안드로이드 히치하이커를 위한 안내서

Slide 1

Slide 1 text

MLਸ ৈ೯ೞח ൤஖ೞ੉ழܳ ਤೠ উղࢲ - ML Vision - উࢿਊ, ֎੉ߡਢొ @fornewid @SOUP

Slide 2

Slide 2 text

Ҋ޹੄ द੘

Slide 3

Slide 3 text

൤஖ೞ੉ழ੄ ࢤп ✨

Slide 4

Slide 4 text

൤஖ೞ੉ழ੄ അप ✨

Slide 5

Slide 5 text

On-device ML ٜ݅ӝ ݾ಴

Slide 6

Slide 6 text

On-device ML ࢎਊೞӝ ݾ಴

Slide 7

Slide 7 text

Take a photo Use magic Save a image

Slide 8

Slide 8 text

CameraX Use magic Save a image

Slide 9

Slide 9 text

M L A N D R O I D H I T C H H I K E R Take a photo

Slide 10

Slide 10 text

1. from Camera

Slide 11

Slide 11 text

Camera1 API (API 21 ޷݅) Camera2 API (API 21 ੉࢚) CameraX API (API 21 ੉࢚, alpha) 1. from Camera

Slide 12

Slide 12 text

01010101010101 010101 010101 01010101010101

Slide 13

Slide 13 text

Opt 1. CameraX View in ⍺!

Slide 14

Slide 14 text

CameraX View in ⍺ • app/build.gradle implementation 'androidx.camera:camera-view:1.0.0-alpha03'

Slide 15

Slide 15 text

CameraX View in ⍺ • In Layout XML:

Slide 16

Slide 16 text

CameraX View in ⍺ • In Kotlin: // In Activity cameraView.bindToLifecycle(this) // In Fragment cameraView.bindToLifecycle(viewLifecycleOwner)

Slide 17

Slide 17 text

CameraX View in ⍺ • ImageCapture cameraView.takePicture(Executor, OnImageCapturedListener() { override fun onCaptureSuccess( image: ImageProxy, rotationDegrees: Int) {...} }) // or cameraView.takePicture(File, Executor, OnImageSavedListener { override fun onImageSaved(file: File) {...} })

Slide 18

Slide 18 text

CameraX View in ⍺ • FrameProcessor ❌

Slide 19

Slide 19 text

CameraX View in custom

Slide 20

Slide 20 text

CameraX View in custom • FrameProcessor cameraView.setAnalyzer(object : ImageAnalysis.Analyzer { override fun analyze(image: ImageProxy, rotationDegrees: Int) { val mediaImage: Image? = image.image ... } })

Slide 21

Slide 21 text

Opt 2. natario1 / CameraView!

Slide 22

Slide 22 text

natario1 / CameraView • app/build.gradle implementation 'com.otaliastudios:cameraview:2.4.0' Link: https://github.com/natario1/CameraView

Slide 23

Slide 23 text

natario1 / CameraView • In Layout XML:

Slide 24

Slide 24 text

natario1 / CameraView • In Kotlin: // In Activity cameraView.setLifecycleOwner(this) // In Fragment cameraView.setLifecycleOwner(viewLifecycleOwner)

Slide 25

Slide 25 text

natario1 / CameraView • ImageCapture cameraView.addCameraListener(object : CameraListener() { @UiThread override fun onPictureTaken(result: PictureResult) { @WorkerThread result.toBitmap(object : BitmapCallback { @UiThread override fun onBitmapReady(bitmap: Bitmap?) { ... } }) } }) cameraView.takePicture()

Slide 26

Slide 26 text

natario1 / CameraView • FrameProcessor cameraView.addFrameProcessor(object : FrameProcessor { @WorkerThread override fun process(frame: Frame) { val data: ByteArray = frame.data val width: Int = frame.size.width val height: Int = frame.size.height val format: Int = frame.format // android.graphics.ImageFormat val rotation: Int = frame.rotation // 0, 90, 180, 270 } })

Slide 27

Slide 27 text

Wrap-up: Camera Source CameraX Bitmap android.graphics CameraView ByteArray byte[] File java.io Image android.media

Slide 28

Slide 28 text

• FirebaseVisionImageܳ ੉ਊೞৈ औѱ ߸ജೡ ࣻ ੓׮. ੉੹ ױ҅ীࢲ оઉৡ ؘ੉ఠܳ Bitmap ഋక۽ ߸ജ೧ঠ ೠ׮. 2. Any to Bitmap Bitmap ByteArray File Image Bitmap

Slide 29

Slide 29 text

FirebaseVisionImage

Slide 30

Slide 30 text

• app/build.gradle FirebaseVisionImage implementation 'com.google.firebase:firebase-ml-vision:24.0.0' Link: https://firebase.google.com/.../firebase/ml/vision/common/FirebaseVisionImage

Slide 31

Slide 31 text

• ByteArray to Bitmap FirebaseVisionImage val bitmap = FirebaseVisionImage.fromByteArray( frame.data, // ByteArray FirebaseVisionImageMetadata.Builder() .setFormat(FirebaseVisionImageMetadata.IMAGE_FORMAT_NV21) .setWidth(frame.size.width) .setHeight(frame.size.height) .setRotation(FirebaseVisionImageMetadata.ROTATION_90) .build()) .bitmap

Slide 32

Slide 32 text

• Uri to Bitmap FirebaseVisionImage val bitmap = FirebaseVisionImage .fromFilePath(context, uri) // Context, Uri .bitmap

Slide 33

Slide 33 text

FirebaseVisionImage • Media Image to Bitmap val bitmap = FirebaseVisionImage .fromMediaImage( image, // android.media.Image FirebaseVisionImageMetadata.ROTATION_90 ) .bitmap

Slide 34

Slide 34 text

Done!

Slide 35

Slide 35 text

CameraX Use magic Save a image

Slide 36

Slide 36 text

CameraX Use magic Save a image

Slide 37

Slide 37 text

CameraX TF Lite / ML Kit Save a image

Slide 38

Slide 38 text

M L A N D R O I D H I T C H H I K E R Use magic

Slide 39

Slide 39 text

Opt 1. Tensorflow Lite

Slide 40

Slide 40 text

Model TFLite

Slide 41

Slide 41 text

) Interpreter (

Slide 42

Slide 42 text

) Interpreter (

Slide 43

Slide 43 text

) Interpreter ( Bitmap

Slide 44

Slide 44 text

) Interpreter ( Bitmap Output

Slide 45

Slide 45 text

) Interpreter ( Array<……> Array<……>

Slide 46

Slide 46 text

Tensorflow Lite // module.gradle android { aaptOptions { noCompress "tflite" } } dependencies { implementation 'org.tensorflow:tensorflow-lite:1.14.0' }

Slide 47

Slide 47 text

Tensorflow Lite // TF Lite Model assets/ detect.tflite

Slide 48

Slide 48 text

Tensorflow Lite val model = context.loadModelFile(" detect.tflite ") @Throws(IOException::class) fun Context.loadModelFile(path: String): ByteBuffer { return assets.openFd(path).run { FileInputStream(fileDescriptor).channel.map( FileChannel.MapMode.READ_ONLY, startOffset, declaredLength ) } }

Slide 49

Slide 49 text

Tensorflow Lite val model = context.loadModelFile(" detect.tflite ") val options = Interpreter.Options().apply { setNumThreads(2) } this.interpreter = Interpreter(model, options)

Slide 50

Slide 50 text

Object Detection Image Segmentation Hosted models.. Pose Estimation Link: https://www.tensorflow.org/lite/guide/hosted_models

Slide 51

Slide 51 text

Object Detection Pose Estimation Image Segmentation

Slide 52

Slide 52 text

Array<……> Array<……> Object Detection

Slide 53

Slide 53 text

Array>>> Array<……> Image ࣻ Width Height Channel ࣻ Object Detection

Slide 54

Slide 54 text

Array>>> Array<……> 1 300 300 3 Object Detection

Slide 55

Slide 55 text

Link: https://www.tensorflow.org/lite/performance/post_training_quantization float32 float16 int byte • ݽ؛ ੿ഛب (Ѣ੄ زੌ) • ݽ؛ ௼ӝ (хࣗ) • CPU ߂ ೞ٘ਝয оࣘӝ ૑োदр (хࣗ) Quantization

Slide 56

Slide 56 text

Link: https://www.tensorflow.org/lite/performance/post_training_quantization 0x112233 • ݽ؛ ੿ഛب (Ѣ੄ زੌ) • ݽ؛ ௼ӝ (хࣗ) • CPU ߂ ೞ٘ਝয оࣘӝ ૑োदр (хࣗ) Quantization

Slide 57

Slide 57 text

Link: https://www.tensorflow.org/lite/performance/post_training_quantization 0x112233 0x11 [0, 255] 0x22 [0, 255] 0x33 [0, 255] 17 34 51 byte byte byte • ݽ؛ ੿ഛب (Ѣ੄ زੌ) • ݽ؛ ௼ӝ (хࣗ) • CPU ߂ ೞ٘ਝয оࣘӝ ૑োदр (хࣗ) Quantization

Slide 58

Slide 58 text

Array>>> Array<……> 1 300 300 3 Object Detection

Slide 59

Slide 59 text

Array>>> Array<……> 1 300 300 3 Object Detection

Slide 60

Slide 60 text

Array<……> Array<……> Object Detection

Slide 61

Slide 61 text

Array> Array Array FloatArray 0 1 2 3 1 1 1 N 4 N N N Object Detection

Slide 62

Slide 62 text

Object Detection: Input // Input (Float model) fun Bitmap.toByteBuffer(): ByteBuffer { val inputSize = FLOAT_TYPE_SIZE * inputWidth * inputHeight * PIXEL_SIZE return ByteBuffer.allocateDirect(inputSize).apply { order(ByteOrder.nativeOrder()) val pixels = IntArray(inputWidth * inputHeight) getPixels(pixels, 0, width, 0, 0, width, height) for (pixel in pixels) { putFloat((pixel shr 16 and 0xFF) / IMAGE_MEAN - IMAGE_OFFSET) // R putFloat((pixel shr 8 and 0xFF) / IMAGE_MEAN - IMAGE_OFFSET) // G putFloat((pixel and 0xFF) / IMAGE_MEAN - IMAGE_OFFSET) // B } } } // const const val IMAGE_MEAN = 128f const val IMAGE_OFFSET = 1f const val FLOAT_TYPE_SIZE = 4 const val PIXEL_SIZE = 3

Slide 63

Slide 63 text

Object Detection: Input // Input (Quantized model) fun Bitmap.toByteBuffer(): ByteBuffer { val inputSize = inputWidth * inputHeight * PIXEL_SIZE return ByteBuffer.allocateDirect(inputSize).apply { order(ByteOrder.nativeOrder()) val pixels = IntArray(inputWidth * inputHeight) getPixels(pixels, 0, width, 0, 0, width, height) for (pixel in pixels) { put((pixelValue shr 16 and 0xFF).toByte()) // R put((pixelValue shr 8 and 0xFF).toByte()) // G put((pixelValue and 0xFF).toByte()) // B } } } // const const val IMAGE_MEAN = 128f const val IMAGE_OFFSET = 1f const val PIXEL_SIZE = 3

Slide 64

Slide 64 text

Object Detection: Output const val NUM_DETECTIONS = 10 // Output val outputMap = hashMapOf( 0 to Array(1) { Array(NUM_DETECTIONS) { FloatArray(4) } }, 1 to Array(1) { FloatArray(NUM_DETECTIONS) }, 2 to Array(1) { FloatArray(NUM_DETECTIONS) }, 3 to FloatArray(1) )

Slide 65

Slide 65 text

Object Detection val input = inputImage.toByteBuffer() val inputArray = arrayOf(input) val outputMap = hashMapOf (...) val interpreter = Interpreter(...) interpreter.runForMultipleInputsOutputs(inputArray, outputMap)

Slide 66

Slide 66 text

Opt 2. ML Kit for Firebase

Slide 67

Slide 67 text

No content

Slide 68

Slide 68 text

ML Kit: Custom Model • app/build.gradle implementation 'com.google.firebase:firebase-ml-model-interpreter:22.0.0'

Slide 69

Slide 69 text

ML Kit: Custom Model val localModel = FirebaseCustomLocalModel.Builder() .setAssetFilePath("model_name.tflite") .build() val options = FirebaseModelInterpreterOptions.Builder(localModel).build() val interpreter = FirebaseModelInterpreter.getInstance(options)

Slide 70

Slide 70 text

ML Kit: Custom Model val inputs = FirebaseModelInputs.Builder() .add(bitmap.toByteBuffer()) .build() val inputOutputOptions = FirebaseModelInputOutputOptions.Builder() .setInputFormat( 0, FirebaseModelDataType.BYTE, intArrayOf(1, 300, 300, 3)) .setOutputFormat( 0, FirebaseModelDataType.BYTE, intArrayOf(1, 300, 300, 3)) .build()

Slide 71

Slide 71 text

ML Kit: Custom Model interpreter.run(inputs, inputOutputOptions) .addOnSuccessListener { result -> val output: Array>> = result.getOutput(0) ... } .addOnFailureListener { e -> ... }

Slide 72

Slide 72 text

Done!

Slide 73

Slide 73 text

CameraX TF Lite / ML Kit Save a image

Slide 74

Slide 74 text

CameraX TF Lite / ML Kit Save a image

Slide 75

Slide 75 text

CameraX TF Lite / ML Kit Scoped Storage

Slide 76

Slide 76 text

M L A N D R O I D H I T C H H I K E R Save a image

Slide 77

Slide 77 text

Pre- Q • app/AndroidManifest.xml

Slide 78

Slide 78 text

val bitmap: Bitmap = ... val pictures = Environment.getExternalStoragePublicDirectory( Environment.DIRECTORY_PICTURES) val imageFilePath = File(pictures, "file_name.png").absolutePath val out = FileOutputStream(imageFilePath) bitmap.compress(Bitmap.CompressFormat.PNG, 100, out) out.flush() out.close() val uri: Uri? = context.contentResolver.insert( MediaStore.Images.Media.EXTERNAL_CONTENT_URI, contentValuesOf( ImageColumns.DATA to imageFilePath, ImageColumns.DISPLAY_NAME to imageFileName Pre- Q

Slide 79

Slide 79 text

Environment.DIRECTORY_PICTURES) val imageFilePath = File(pictures, "file_name.png").absolutePath val out = FileOutputStream(imageFilePath) bitmap.compress(Bitmap.CompressFormat.PNG, 100, out) out.flush() out.close() val uri: Uri? = context.contentResolver.insert( MediaStore.Images.Media.EXTERNAL_CONTENT_URI, contentValuesOf( ImageColumns.DATA to imageFilePath, ImageColumns.DISPLAY_NAME to imageFileName ) ) // Show Notification! Pre- Q

Slide 80

Slide 80 text

But, deprecated in Q

Slide 81

Slide 81 text

Option 1. Avoid temporarily • app/AndroidManifest.xml

Slide 82

Slide 82 text

• Before: Option 2. Scoped Storage Bitmap External Storage File MediaStore Content Values

Slide 83

Slide 83 text

• After: Option 2. Scoped Storage Bitmap External Storage File MediaStore Content Values

Slide 84

Slide 84 text

• After: Option 2. Scoped Storage Bitmap External Storage File MediaStore Content Values

Slide 85

Slide 85 text

• After: Option 2. Scoped Storage Bitmap External Storage File MediaStore Content Values Pending

Slide 86

Slide 86 text

val bitmap: Bitmap = ... val resolver = context.contentResolver val uri = resolver.insert( MediaStore.Images.Media.EXTERNAL_CONTENT_URI, contentValuesOf( MediaColumns.DISPLAY_NAME to "file_name.png", MediaColumns.IS_PENDING to 1, MediaColumns.DATE_EXPIRES to ..., MediaColumns.RELATIVE_PATH to Environment.DIRECTORY_PICTURES ) ) val out = resolver.openOutputStream(uri) bitmap.compress(Bitmap.CompressFormat.PNG, 100, out) out.flush() out.close() Option 2. Scoped Storage

Slide 87

Slide 87 text

MediaColumns.IS_PENDING to 1, MediaColumns.DATE_EXPIRES to ..., MediaColumns.RELATIVE_PATH to Environment.DIRECTORY_PICTURES ) ) val out = resolver.openOutputStream(uri) bitmap.compress(Bitmap.CompressFormat.PNG, 100, out) out.flush() out.close() resolver.update(uri, contentValuesOf( MediaColumns.IS_PENDING to 0, MediaColumns.DATE_EXPIRES to null ), null, null) Link: https://android-10/android/provider/MediaStore.java#PendingParams Option 2. Scoped Storage

Slide 88

Slide 88 text

Done!

Slide 89

Slide 89 text

• ஠ݫۄо ೙ਃೞ׮ݶ CameraXܳ ࢎਊ೧ࠁࣁਃ. • Pre-L OSਸ ૑ਗೠ׮ݶ, natario1/CameraViewܳ ୶ୌ೤פ׮. • FirebaseVisionImageܳ ࢎਊೞৈ Bitmapਵ۽ औѱ ߄Ե ࣻ ੓णפ׮. • TF Liteܳ ࢎਊೠ׮ݶ, ML Kit - Custom Model ࢎਊਸ Ҋ۰೧ࠁࣁਃ. • Qীࢲח ੉޷૑ܳ ੷੢ೡ ٸ, Scoped Storageܳ Ҋ۰ೞࣁਃ. Summary

Slide 90

Slide 90 text

хࢎ೤פ׮!