Share
Embed
Transcript
90 slides
-
MLਸ ৈ೯ೞח ೞழܳ ਤೠ উղࢲ - ML Vision - উࢿਊ,
֎ߡਢొ @fornewid @SOUP -
M L A N D R O I D H
I T C H H I K E R Take a photo -
Camera1 API (API 21 ݅) Camera2 API (API 21 ࢚)
CameraX API (API 21 ࢚, alpha) 1. from Camera -
CameraX View in ⍺ • In Layout XML: <androidx.camera.view.CameraView ...
app:scaleType="centerCrop|centerInside" app:lensFacing="back|front|none" app:captureMode="image|video|mixed" app:flash="off|on|auto" app:pinchToZoomEnabled="true|false" /> -
CameraX View in ⍺ • In Kotlin: // In Activity
cameraView.bindToLifecycle(this) // In Fragment cameraView.bindToLifecycle(viewLifecycleOwner) -
CameraX View in ⍺ • ImageCapture cameraView.takePicture(Executor, OnImageCapturedListener() { override
fun onCaptureSuccess( image: ImageProxy, rotationDegrees: Int) {...} }) // or cameraView.takePicture(File, Executor, OnImageSavedListener { override fun onImageSaved(file: File) {...} }) -
CameraX View in custom • FrameProcessor cameraView.setAnalyzer(object : ImageAnalysis.Analyzer {
override fun analyze(image: ImageProxy, rotationDegrees: Int) { val mediaImage: Image? = image.image ... } }) -
natario1 / CameraView • In Layout XML: <com.otaliastudios.cameraview.CameraView ... app:cameraFacing="back|front"
app:cameraMode="picture|video" app:cameraFlash="on|auto|torch|off" app:cameraGesturePinch="none|zoom|exposureCorrection" /> -
natario1 / CameraView • In Kotlin: // In Activity cameraView.setLifecycleOwner(this)
// In Fragment cameraView.setLifecycleOwner(viewLifecycleOwner) -
natario1 / CameraView • ImageCapture cameraView.addCameraListener(object : CameraListener() { @UiThread
override fun onPictureTaken(result: PictureResult) { @WorkerThread result.toBitmap(object : BitmapCallback { @UiThread override fun onBitmapReady(bitmap: Bitmap?) { ... } }) } }) cameraView.takePicture() -
natario1 / CameraView • FrameProcessor cameraView.addFrameProcessor(object : FrameProcessor { @WorkerThread
override fun process(frame: Frame) { val data: ByteArray = frame.data val width: Int = frame.size.width val height: Int = frame.size.height val format: Int = frame.format // android.graphics.ImageFormat val rotation: Int = frame.rotation // 0, 90, 180, 270 } }) -
Wrap-up: Camera Source CameraX Bitmap android.graphics CameraView ByteArray byte[] File
java.io Image android.media -
• FirebaseVisionImageܳ ਊೞৈ औѱ ߸ജೡ ࣻ . ױ҅ীࢲ оઉৡ
ؘఠܳ Bitmap ഋక۽ ߸ജ೧ঠ ೠ. 2. Any to Bitmap Bitmap ByteArray File Image Bitmap -
• ByteArray to Bitmap FirebaseVisionImage val bitmap = FirebaseVisionImage.fromByteArray( frame.data,
// ByteArray FirebaseVisionImageMetadata.Builder() .setFormat(FirebaseVisionImageMetadata.IMAGE_FORMAT_NV21) .setWidth(frame.size.width) .setHeight(frame.size.height) .setRotation(FirebaseVisionImageMetadata.ROTATION_90) .build()) .bitmap -
• Uri to Bitmap FirebaseVisionImage val bitmap = FirebaseVisionImage .fromFilePath(context,
uri) // Context, Uri .bitmap -
FirebaseVisionImage • Media Image to Bitmap val bitmap = FirebaseVisionImage
.fromMediaImage( image, // android.media.Image FirebaseVisionImageMetadata.ROTATION_90 ) .bitmap -
M L A N D R O I D H
I T C H H I K E R Use magic -
Tensorflow Lite // module.gradle android { aaptOptions { noCompress "tflite"
} } dependencies { implementation 'org.tensorflow:tensorflow-lite:1.14.0' } -
Tensorflow Lite val model = context.loadModelFile(" detect.tflite ") @Throws(IOException::class) fun
Context.loadModelFile(path: String): ByteBuffer { return assets.openFd(path).run { FileInputStream(fileDescriptor).channel.map( FileChannel.MapMode.READ_ONLY, startOffset, declaredLength ) } } -
Tensorflow Lite val model = context.loadModelFile(" detect.tflite ") val options
= Interpreter.Options().apply { setNumThreads(2) } this.interpreter = Interpreter(model, options) -
Link: https://www.tensorflow.org/lite/performance/post_training_quantization float32 float16 int byte • ݽ؛ ഛب (Ѣ
زੌ) • ݽ؛ ӝ (хࣗ) • CPU ߂ ೞ٘ਝয оࣘӝ োदр (хࣗ) Quantization -
Link: https://www.tensorflow.org/lite/performance/post_training_quantization 0x112233 • ݽ؛ ഛب (Ѣ زੌ) • ݽ؛
ӝ (хࣗ) • CPU ߂ ೞ٘ਝয оࣘӝ োदр (хࣗ) Quantization -
Link: https://www.tensorflow.org/lite/performance/post_training_quantization 0x112233 0x11 [0, 255] 0x22 [0, 255] 0x33
[0, 255] 17 34 51 byte byte byte • ݽ؛ ഛب (Ѣ زੌ) • ݽ؛ ӝ (хࣗ) • CPU ߂ ೞ٘ਝয оࣘӝ োदр (хࣗ) Quantization -
Array<Array<FloatArray>> Array<FloatArray> Array<FloatArray> FloatArray 0 1 2 3 1 1
1 N 4 N N N Object Detection -
Object Detection: Input // Input (Float model) fun Bitmap.toByteBuffer(): ByteBuffer
{ val inputSize = FLOAT_TYPE_SIZE * inputWidth * inputHeight * PIXEL_SIZE return ByteBuffer.allocateDirect(inputSize).apply { order(ByteOrder.nativeOrder()) val pixels = IntArray(inputWidth * inputHeight) getPixels(pixels, 0, width, 0, 0, width, height) for (pixel in pixels) { putFloat((pixel shr 16 and 0xFF) / IMAGE_MEAN - IMAGE_OFFSET) // R putFloat((pixel shr 8 and 0xFF) / IMAGE_MEAN - IMAGE_OFFSET) // G putFloat((pixel and 0xFF) / IMAGE_MEAN - IMAGE_OFFSET) // B } } } // const const val IMAGE_MEAN = 128f const val IMAGE_OFFSET = 1f const val FLOAT_TYPE_SIZE = 4 const val PIXEL_SIZE = 3 -
Object Detection: Input // Input (Quantized model) fun Bitmap.toByteBuffer(): ByteBuffer
{ val inputSize = inputWidth * inputHeight * PIXEL_SIZE return ByteBuffer.allocateDirect(inputSize).apply { order(ByteOrder.nativeOrder()) val pixels = IntArray(inputWidth * inputHeight) getPixels(pixels, 0, width, 0, 0, width, height) for (pixel in pixels) { put((pixelValue shr 16 and 0xFF).toByte()) // R put((pixelValue shr 8 and 0xFF).toByte()) // G put((pixelValue and 0xFF).toByte()) // B } } } // const const val IMAGE_MEAN = 128f const val IMAGE_OFFSET = 1f const val PIXEL_SIZE = 3 -
Object Detection: Output const val NUM_DETECTIONS = 10 // Output
val outputMap = hashMapOf<Int, Any>( 0 to Array(1) { Array(NUM_DETECTIONS) { FloatArray(4) } }, 1 to Array(1) { FloatArray(NUM_DETECTIONS) }, 2 to Array(1) { FloatArray(NUM_DETECTIONS) }, 3 to FloatArray(1) ) -
Object Detection val input = inputImage.toByteBuffer() val inputArray = arrayOf<Any>(input)
val outputMap = hashMapOf<Int, Any> (...) val interpreter = Interpreter(...) interpreter.runForMultipleInputsOutputs(inputArray, outputMap) -
ML Kit: Custom Model val localModel = FirebaseCustomLocalModel.Builder() .setAssetFilePath("model_name.tflite") .build()
val options = FirebaseModelInterpreterOptions.Builder(localModel).build() val interpreter = FirebaseModelInterpreter.getInstance(options) -
ML Kit: Custom Model val inputs = FirebaseModelInputs.Builder() .add(bitmap.toByteBuffer()) .build()
val inputOutputOptions = FirebaseModelInputOutputOptions.Builder() .setInputFormat( 0, FirebaseModelDataType.BYTE, intArrayOf(1, 300, 300, 3)) .setOutputFormat( 0, FirebaseModelDataType.BYTE, intArrayOf(1, 300, 300, 3)) .build() -
ML Kit: Custom Model interpreter.run(inputs, inputOutputOptions) .addOnSuccessListener { result ->
val output: Array<Array<Array<FloatArray>>> = result.getOutput(0) ... } .addOnFailureListener { e -> ... } -
M L A N D R O I D H
I T C H H I K E R Save a image -
val bitmap: Bitmap = ... val pictures = Environment.getExternalStoragePublicDirectory( Environment.DIRECTORY_PICTURES)
val imageFilePath = File(pictures, "file_name.png").absolutePath val out = FileOutputStream(imageFilePath) bitmap.compress(Bitmap.CompressFormat.PNG, 100, out) out.flush() out.close() val uri: Uri? = context.contentResolver.insert( MediaStore.Images.Media.EXTERNAL_CONTENT_URI, contentValuesOf( ImageColumns.DATA to imageFilePath, ImageColumns.DISPLAY_NAME to imageFileName Pre- Q -
Environment.DIRECTORY_PICTURES) val imageFilePath = File(pictures, "file_name.png").absolutePath val out = FileOutputStream(imageFilePath)
bitmap.compress(Bitmap.CompressFormat.PNG, 100, out) out.flush() out.close() val uri: Uri? = context.contentResolver.insert( MediaStore.Images.Media.EXTERNAL_CONTENT_URI, contentValuesOf( ImageColumns.DATA to imageFilePath, ImageColumns.DISPLAY_NAME to imageFileName ) ) // Show Notification! Pre- Q -
Option 1. Avoid temporarily • app/AndroidManifest.xml <application android:requestLegacyExternalStorage="true" ... Link:
https://developer.android.com/.../scoped-storage#opt-out-of-filtered-view -
• Before: Option 2. Scoped Storage Bitmap External Storage File
MediaStore Content Values -
• After: Option 2. Scoped Storage Bitmap External Storage File
MediaStore Content Values -
• After: Option 2. Scoped Storage Bitmap External Storage File
MediaStore Content Values -
• After: Option 2. Scoped Storage Bitmap External Storage File
MediaStore Content Values Pending -
val bitmap: Bitmap = ... val resolver = context.contentResolver val
uri = resolver.insert( MediaStore.Images.Media.EXTERNAL_CONTENT_URI, contentValuesOf( MediaColumns.DISPLAY_NAME to "file_name.png", MediaColumns.IS_PENDING to 1, MediaColumns.DATE_EXPIRES to ..., MediaColumns.RELATIVE_PATH to Environment.DIRECTORY_PICTURES ) ) val out = resolver.openOutputStream(uri) bitmap.compress(Bitmap.CompressFormat.PNG, 100, out) out.flush() out.close() Option 2. Scoped Storage -
MediaColumns.IS_PENDING to 1, MediaColumns.DATE_EXPIRES to ..., MediaColumns.RELATIVE_PATH to Environment.DIRECTORY_PICTURES )
) val out = resolver.openOutputStream(uri) bitmap.compress(Bitmap.CompressFormat.PNG, 100, out) out.flush() out.close() resolver.update(uri, contentValuesOf( MediaColumns.IS_PENDING to 0, MediaColumns.DATE_EXPIRES to null ), null, null) Link: https://android-10/android/provider/MediaStore.java#PendingParams Option 2. Scoped Storage -
• ݫۄо ਃೞݶ CameraXܳ ࢎਊ೧ࠁࣁਃ. • Pre-L OSਸ ਗೠݶ, natario1/CameraViewܳ
୶ୌפ. • FirebaseVisionImageܳ ࢎਊೞৈ Bitmapਵ۽ औѱ ߄Ե ࣻ णפ. • TF Liteܳ ࢎਊೠݶ, ML Kit - Custom Model ࢎਊਸ Ҋ۰೧ࠁࣁਃ. • Qীࢲח ܳ ೡ ٸ, Scoped Storageܳ Ҋ۰ೞࣁਃ. Summary