Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ML을 여행하는 안드로이드 히치하이커를 위한 안내서

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Sungyong An
November 15, 2019

ML을 여행하는 안드로이드 히치하이커를 위한 안내서

DevFest Pangyo 2019

Sungyong An

November 15, 2019

Transcript

90 slides
  1. M L A N D R O I D H

    I T C H H I K E R Take a photo
  2. Camera1 API (API 21 ޷݅) Camera2 API (API 21 ੉࢚)

    CameraX API (API 21 ੉࢚, alpha) 1. from Camera
  3. CameraX View in ⍺ • In Layout XML: <androidx.camera.view.CameraView ...

    app:scaleType="centerCrop|centerInside" app:lensFacing="back|front|none" app:captureMode="image|video|mixed" app:flash="off|on|auto" app:pinchToZoomEnabled="true|false" />
  4. CameraX View in ⍺ • In Kotlin: // In Activity

    cameraView.bindToLifecycle(this) // In Fragment cameraView.bindToLifecycle(viewLifecycleOwner)
  5. CameraX View in ⍺ • ImageCapture cameraView.takePicture(Executor, OnImageCapturedListener() { override

    fun onCaptureSuccess( image: ImageProxy, rotationDegrees: Int) {...} }) // or cameraView.takePicture(File, Executor, OnImageSavedListener { override fun onImageSaved(file: File) {...} })
  6. CameraX View in custom • FrameProcessor cameraView.setAnalyzer(object : ImageAnalysis.Analyzer {

    override fun analyze(image: ImageProxy, rotationDegrees: Int) { val mediaImage: Image? = image.image ... } })
  7. natario1 / CameraView • In Layout XML: <com.otaliastudios.cameraview.CameraView ... app:cameraFacing="back|front"

    app:cameraMode="picture|video" app:cameraFlash="on|auto|torch|off" app:cameraGesturePinch="none|zoom|exposureCorrection" />
  8. natario1 / CameraView • In Kotlin: // In Activity cameraView.setLifecycleOwner(this)

    // In Fragment cameraView.setLifecycleOwner(viewLifecycleOwner)
  9. natario1 / CameraView • ImageCapture cameraView.addCameraListener(object : CameraListener() { @UiThread

    override fun onPictureTaken(result: PictureResult) { @WorkerThread result.toBitmap(object : BitmapCallback { @UiThread override fun onBitmapReady(bitmap: Bitmap?) { ... } }) } }) cameraView.takePicture()
  10. natario1 / CameraView • FrameProcessor cameraView.addFrameProcessor(object : FrameProcessor { @WorkerThread

    override fun process(frame: Frame) { val data: ByteArray = frame.data val width: Int = frame.size.width val height: Int = frame.size.height val format: Int = frame.format // android.graphics.ImageFormat val rotation: Int = frame.rotation // 0, 90, 180, 270 } })
  11. • FirebaseVisionImageܳ ੉ਊೞৈ औѱ ߸ജೡ ࣻ ੓׮. ੉੹ ױ҅ীࢲ оઉৡ

    ؘ੉ఠܳ Bitmap ഋక۽ ߸ജ೧ঠ ೠ׮. 2. Any to Bitmap Bitmap ByteArray File Image Bitmap
  12. • ByteArray to Bitmap FirebaseVisionImage val bitmap = FirebaseVisionImage.fromByteArray( frame.data,

    // ByteArray FirebaseVisionImageMetadata.Builder() .setFormat(FirebaseVisionImageMetadata.IMAGE_FORMAT_NV21) .setWidth(frame.size.width) .setHeight(frame.size.height) .setRotation(FirebaseVisionImageMetadata.ROTATION_90) .build()) .bitmap
  13. FirebaseVisionImage • Media Image to Bitmap val bitmap = FirebaseVisionImage

    .fromMediaImage( image, // android.media.Image FirebaseVisionImageMetadata.ROTATION_90 ) .bitmap
  14. M L A N D R O I D H

    I T C H H I K E R Use magic
  15. Tensorflow Lite // module.gradle android { aaptOptions { noCompress "tflite"

    } } dependencies { implementation 'org.tensorflow:tensorflow-lite:1.14.0' }
  16. Tensorflow Lite val model = context.loadModelFile(" detect.tflite ") @Throws(IOException::class) fun

    Context.loadModelFile(path: String): ByteBuffer { return assets.openFd(path).run { FileInputStream(fileDescriptor).channel.map( FileChannel.MapMode.READ_ONLY, startOffset, declaredLength ) } }
  17. Tensorflow Lite val model = context.loadModelFile(" detect.tflite ") val options

    = Interpreter.Options().apply { setNumThreads(2) } this.interpreter = Interpreter(model, options)
  18. Link: https://www.tensorflow.org/lite/performance/post_training_quantization float32 float16 int byte • ݽ؛ ੿ഛب (Ѣ੄

    زੌ) • ݽ؛ ௼ӝ (хࣗ) • CPU ߂ ೞ٘ਝয оࣘӝ ૑োदр (хࣗ) Quantization
  19. Link: https://www.tensorflow.org/lite/performance/post_training_quantization 0x112233 0x11 [0, 255] 0x22 [0, 255] 0x33

    [0, 255] 17 34 51 byte byte byte • ݽ؛ ੿ഛب (Ѣ੄ زੌ) • ݽ؛ ௼ӝ (хࣗ) • CPU ߂ ೞ٘ਝয оࣘӝ ૑োदр (хࣗ) Quantization
  20. Object Detection: Input // Input (Float model) fun Bitmap.toByteBuffer(): ByteBuffer

    { val inputSize = FLOAT_TYPE_SIZE * inputWidth * inputHeight * PIXEL_SIZE return ByteBuffer.allocateDirect(inputSize).apply { order(ByteOrder.nativeOrder()) val pixels = IntArray(inputWidth * inputHeight) getPixels(pixels, 0, width, 0, 0, width, height) for (pixel in pixels) { putFloat((pixel shr 16 and 0xFF) / IMAGE_MEAN - IMAGE_OFFSET) // R putFloat((pixel shr 8 and 0xFF) / IMAGE_MEAN - IMAGE_OFFSET) // G putFloat((pixel and 0xFF) / IMAGE_MEAN - IMAGE_OFFSET) // B } } } // const const val IMAGE_MEAN = 128f const val IMAGE_OFFSET = 1f const val FLOAT_TYPE_SIZE = 4 const val PIXEL_SIZE = 3
  21. Object Detection: Input // Input (Quantized model) fun Bitmap.toByteBuffer(): ByteBuffer

    { val inputSize = inputWidth * inputHeight * PIXEL_SIZE return ByteBuffer.allocateDirect(inputSize).apply { order(ByteOrder.nativeOrder()) val pixels = IntArray(inputWidth * inputHeight) getPixels(pixels, 0, width, 0, 0, width, height) for (pixel in pixels) { put((pixelValue shr 16 and 0xFF).toByte()) // R put((pixelValue shr 8 and 0xFF).toByte()) // G put((pixelValue and 0xFF).toByte()) // B } } } // const const val IMAGE_MEAN = 128f const val IMAGE_OFFSET = 1f const val PIXEL_SIZE = 3
  22. Object Detection: Output const val NUM_DETECTIONS = 10 // Output

    val outputMap = hashMapOf<Int, Any>( 0 to Array(1) { Array(NUM_DETECTIONS) { FloatArray(4) } }, 1 to Array(1) { FloatArray(NUM_DETECTIONS) }, 2 to Array(1) { FloatArray(NUM_DETECTIONS) }, 3 to FloatArray(1) )
  23. Object Detection val input = inputImage.toByteBuffer() val inputArray = arrayOf<Any>(input)

    val outputMap = hashMapOf<Int, Any> (...) val interpreter = Interpreter(...) interpreter.runForMultipleInputsOutputs(inputArray, outputMap)
  24. ML Kit: Custom Model val localModel = FirebaseCustomLocalModel.Builder() .setAssetFilePath("model_name.tflite") .build()

    val options = FirebaseModelInterpreterOptions.Builder(localModel).build() val interpreter = FirebaseModelInterpreter.getInstance(options)
  25. ML Kit: Custom Model val inputs = FirebaseModelInputs.Builder() .add(bitmap.toByteBuffer()) .build()

    val inputOutputOptions = FirebaseModelInputOutputOptions.Builder() .setInputFormat( 0, FirebaseModelDataType.BYTE, intArrayOf(1, 300, 300, 3)) .setOutputFormat( 0, FirebaseModelDataType.BYTE, intArrayOf(1, 300, 300, 3)) .build()
  26. ML Kit: Custom Model interpreter.run(inputs, inputOutputOptions) .addOnSuccessListener { result ->

    val output: Array<Array<Array<FloatArray>>> = result.getOutput(0) ... } .addOnFailureListener { e -> ... }
  27. M L A N D R O I D H

    I T C H H I K E R Save a image
  28. val bitmap: Bitmap = ... val pictures = Environment.getExternalStoragePublicDirectory( Environment.DIRECTORY_PICTURES)

    val imageFilePath = File(pictures, "file_name.png").absolutePath val out = FileOutputStream(imageFilePath) bitmap.compress(Bitmap.CompressFormat.PNG, 100, out) out.flush() out.close() val uri: Uri? = context.contentResolver.insert( MediaStore.Images.Media.EXTERNAL_CONTENT_URI, contentValuesOf( ImageColumns.DATA to imageFilePath, ImageColumns.DISPLAY_NAME to imageFileName Pre- Q
  29. Environment.DIRECTORY_PICTURES) val imageFilePath = File(pictures, "file_name.png").absolutePath val out = FileOutputStream(imageFilePath)

    bitmap.compress(Bitmap.CompressFormat.PNG, 100, out) out.flush() out.close() val uri: Uri? = context.contentResolver.insert( MediaStore.Images.Media.EXTERNAL_CONTENT_URI, contentValuesOf( ImageColumns.DATA to imageFilePath, ImageColumns.DISPLAY_NAME to imageFileName ) ) // Show Notification! Pre- Q
  30. val bitmap: Bitmap = ... val resolver = context.contentResolver val

    uri = resolver.insert( MediaStore.Images.Media.EXTERNAL_CONTENT_URI, contentValuesOf( MediaColumns.DISPLAY_NAME to "file_name.png", MediaColumns.IS_PENDING to 1, MediaColumns.DATE_EXPIRES to ..., MediaColumns.RELATIVE_PATH to Environment.DIRECTORY_PICTURES ) ) val out = resolver.openOutputStream(uri) bitmap.compress(Bitmap.CompressFormat.PNG, 100, out) out.flush() out.close() Option 2. Scoped Storage
  31. MediaColumns.IS_PENDING to 1, MediaColumns.DATE_EXPIRES to ..., MediaColumns.RELATIVE_PATH to Environment.DIRECTORY_PICTURES )

    ) val out = resolver.openOutputStream(uri) bitmap.compress(Bitmap.CompressFormat.PNG, 100, out) out.flush() out.close() resolver.update(uri, contentValuesOf( MediaColumns.IS_PENDING to 0, MediaColumns.DATE_EXPIRES to null ), null, null) Link: https://android-10/android/provider/MediaStore.java#PendingParams Option 2. Scoped Storage
  32. • ஠ݫۄо ೙ਃೞ׮ݶ CameraXܳ ࢎਊ೧ࠁࣁਃ. • Pre-L OSਸ ૑ਗೠ׮ݶ, natario1/CameraViewܳ

    ୶ୌ೤פ׮. • FirebaseVisionImageܳ ࢎਊೞৈ Bitmapਵ۽ औѱ ߄Ե ࣻ ੓णפ׮. • TF Liteܳ ࢎਊೠ׮ݶ, ML Kit - Custom Model ࢎਊਸ Ҋ۰೧ࠁࣁਃ. • Qীࢲח ੉޷૑ܳ ੷੢ೡ ٸ, Scoped Storageܳ Ҋ۰ೞࣁਃ. Summary

More Decks by Sungyong An

Other Decks in Programming