Bringing Machine Learning in Android with MediaPipe - DroidJam 2023

Transcript

1. @arif_faizin Curriculum Developer Lead, Dicoding Indonesia Bringing Machine Learning in Android with MediaPipe DroidJam 2023
2. Let’s Start with Why~ DroidJam 2023
5. DroidJam 2023 65% of consumers have trust in the businesses which use AI technology - Forbes Advisor
7. Intro to ML DroidJam 2023
8. Kai Anak MD Bangkit yang ibunya punya Toko Kelontong
10. Lengkuas Jahe Kencur Kunyit
11. Traditional Programming Rules Input Output
12. Machine Learning Output Input Rules
14. How to? DroidJam 2023
15. On-Cloud On-Device
16. • Lower latency & close knit interactions • Offline availability • Privacy preserving • Cost savings On-Device
17. On-Device ML From Scratch 🤔
18. On-Device ML Using Framework 😎
19. Alternative Framework On-Device ML
23. Deep Dive into MediaPipe DroidJam 2023
24. Model is at the core of an on-device ML solution Model Inference Output Input ML app • Customized to the ad-hoc use cases • Light-weight and efficient • Target to hardware • Sparsified for best performance
25. ML Pipeline streamlines the process from raw inputs to output results Model Inference Output Live Camera ML app Flow Control Post-processing Synchronization Data Preprocessing • Domain-specific processing (e.g. vision / NLP / audio) • E2E acceleration across CPU / GPU / EdgeTPU / DSP • Cross-platform deployment to Android / iOS, web, baremetal
26. Display Live Camera ML app Buffer Management Format Conversion Image Filtering Data Subsampling Data Lifecycle Timestamp Extraction Timestamp Alignment Thread Management GPU/CPU Data Transfer Multi-threaded GPU Compute iOS Metal OpenGL ES Trace Collection Performance Profiling C++ Programming Resource Caching Asset Loading GPU Timing Measurement Cross-platform Abstraction Data Marshalling CPU Affinity Java Native Interface Model Inference Flow Control Post-processing Synchronization Data Preprocessing Both involves a lot of complexity that hinders fast development
27. MediaPipe abstracts this complexity into MediaPipe Tasks Model Inference Display MediaPipe Tasks Live Camera ML app Flow Control Post-processing Synchronization Data Preprocessing
28. … while meeting your custom modeling needs with MediaPipe Model Maker Model Inference Display MediaPipe Model Maker Live Camera ML app Flow Control Post-processing Synchronization Data Preprocessing Custom model
29. No-code GUI with MediaPipe Studio
31. DroidJam 2023
32. Create Image Classification App DroidJam 2023
33. DroidJam 2023 • Create App to get image from Gallery or Camera • Alternative solution ◦ Gallery ▪ PhotoPicker ActivityResultContracts.PickVisualMedia() ▪ Intent ACTION_GET_CONTENT ▪ Intent ACTION_PICK ◦ Camera ▪ Intent ACTION_IMAGE_CAPTURE ▪ ActivityResultContracts.TakePicture() ▪ CameraX Starter Project
38. DroidJam 2023 // 0. build.gradle.kts implementation("com.google.mediapipe:tasks-vision:xxx")
39. DroidJam 2023 // 1. Setup Image Classifier (ImageClassifierHelper.kt) val baseOptionsBuilder = BaseOptions.builder() .setDelegate(Delegate.GPU) // CPU, GPU .setModelAssetPath(MODEL_PATH) val optionsBuilder = ImageClassifier.ImageClassifierOptions.builder() .setScoreThreshold(0.1f) // minimum 10% .setMaxResults(3) .setRunningMode(RunningMode.IMAGE) .setBaseOptions(baseOptionsBuilder.build()) val options = optionsBuilder.build() val imageClassifier = ImageClassifier.createFromOptions(context, options)
40. DroidJam 2023 // 2. Create instance of ImageClassifierHelper // Get Data from Camera (in Activity) // Convert Uri to Bitmap imageUri?.let { uri -> if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.P) { val source = ImageDecoder.createSource(contentResolver, imageUri) ImageDecoder.decodeBitmap(source) } else { MediaStore.Images.Media.getBitmap(contentResolver, uri) }.copy(Bitmap.Config.ARGB_8888, true)?.let { bitmap -> imageClassifierHelper.classifyImage(bitmap) } }
41. Source: https://twitter.com/equasys_de/status/754975190459834368/photo/1
42. DroidJam 2023 // 3. Convert the input Bitmap object to an MPImage object to run inference // ImageClassifierHelper.kt fun classifyImage(bitmap: Bitmap) { val mpImage: MPImage = BitmapImageBuilder(bitmap).build() val imageProcessingOptions = ImageProcessingOptions.builder().build() val startTime = SystemClock.uptimeMillis() imageClassifier?.classify(mpImage, imageProcessingOptions).also { result -> val inferenceTime = SystemClock.uptimeMillis() - startTime imageClassifierListener?.onResults(result, inferenceTime) } if (imageClassifier == null) { imageClassifierListener?.onError( "Image classifier failed to classify." ) } }
43. // Output [Classifications {categories= [ <Category "computer keyboard" (displayName= score=0.41453125 index=621)>, <Category "laptop" (displayName= score=0.35921875 index=509)> ], headIndex=0, headName=Optional[probability] } ] Note: It’s only works in real devices, not in Emulator
44. New skill unlocked!
45. Real Time Classification? DroidJam 2023
46. DroidJam 2023 val cameraProviderFuture = ProcessCameraProvider.getInstance(this) cameraProviderFuture.addListener({ val cameraProvider = cameraProviderFuture.get() val preview = Preview.Builder() .setTargetAspectRatio(AspectRatio.RATIO_4_3) .build() .also { it.setSurfaceProvider(binding.viewFinder.surfaceProvider) } cameraProvider.unbindAll() cameraProvider.bindToLifecycle( this, CameraSelector.DEFAULT_BACK_CAMERA, preview ) }, ContextCompat.getMainExecutor(this)) • Request Permission CAMERA • CameraX implementation Starter Project
47. DroidJam 2023 // 1. Setup Options Configuration (ImageClassifierHelper.kt) val optionsBuilder = ImageClassifier.ImageClassifierOptions.builder() .setScoreThreshold(0.1f) .setMaxResults(3) .setRunningMode(RunningMode.LIVE_STREAM) // IMAGE, VIDEO, LIVE_STREAM .setBaseOptions(baseOptionsBuilder.build()) if (runningMode == RunningMode.LIVE_STREAM) { optionsBuilder.setResultListener(this::returnLivestreamResult) optionsBuilder.setErrorListener(this::returnLivestreamError) }
48. DroidJam 2023 // 2. Setup ImageAnalysis for CameraX (in Activity) val imageAnalyzer = ImageAnalysis.Builder() .setTargetAspectRatio(AspectRatio.RATIO_4_3) .setTargetRotation(binding.viewFinder.display.rotation) .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST) .setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888) .build() .also { it.setAnalyzer(Executors.newSingleThreadExecutor()) { image -> imageClassifierHelper.classifyLiveStreamFrame(image) } } ... cameraProvider.bindToLifecycle( this, CameraSelector.DEFAULT_BACK_CAMERA, preview, imageAnalyzer )
49. DroidJam 2023 // 3. Convert the input Bitmap object to an MPImage object to run inference fun classifyLiveStreamFrame(image: ImageProxy) { ... val mpImage = BitmapImageBuilder(bitmapBuffer).build() // Used for rotating the frame image so it matches our models val imageProcessingOptions = ImageProcessingOptions.builder() .setRotationDegrees(image.imageInfo.rotationDegrees) .build() val frameTime = SystemClock.uptimeMillis() // Run inference imageClassifier?.classifyAsync(mpImage, imageProcessingOptions, frameTime) }
50. “Your focus determines your reality.” – Qui-Gon Jinn
51. Object Detection: A new Hope DroidJam 2023
54. DroidJam 2023 // 1. Setup Object Detector (ObjectDetectorHelper.kt) val baseOptionsBuilder = BaseOptions.builder() .setDelegate(Delegate.GPU) // CPU, GPU .setModelAssetPath(MODEL_PATH) val optionsBuilder = ObjectDetector.ObjectDetectorOptions.builder() .setScoreThreshold(0.1f) .setMaxResults(3) .setRunningMode(RunningMode.LIVE_STREAM) .setBaseOptions(baseOptionsBuilder.build()) val options = optionsBuilder.build() val imageClassifier = ImageClassifier.createFromOptions(context, options)
55. DroidJam 2023 // 2. Setup ImageAnalysis for CameraX (in Activity) val imageAnalyzer = ImageAnalysis.Builder() .setTargetAspectRatio(AspectRatio.RATIO_16_9) // adjust with model .setTargetRotation(binding.viewFinder.display.rotation) .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST) .setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888) .build() .also { it.setAnalyzer(Executors.newSingleThreadExecutor()) { image -> objectDetectorHelper.detectLivestreamFrame(image) } }
56. DroidJam 2023 // 3. Convert the input Bitmap object to an MPImage object to run inference fun detectLiveStreamFrame(image: ImageProxy) { ... val mpImage = BitmapImageBuilder(bitmapBuffer).build() // Used for rotating the frame image so it matches our models val imageProcessingOptions = ImageProcessingOptions.builder() .setRotationDegrees(image.imageInfo.rotationDegrees) .build() val frameTime = SystemClock.uptimeMillis() // Run inference objectDetector?.detectAsync(mpImage, imageProcessingOptions, frameTime) }
57. New skill unlocked!
58. DroidJam 2023 // 4a. Draw Box Using Custom View in XML class OverlayView(context: Context?, attrs: AttributeSet?) : View(context, attrs) { override fun draw(canvas: Canvas) { super.draw(canvas) // Draw bounding box around detected objects val drawableRect = RectF(left, top, right, bottom) canvas.drawRect(drawableRect, boxPaint) // Draw text for detected object canvas.drawText( drawableText, left, top + bounds.height(), textPaint ) } } }
59. DroidJam 2023 // 4b. Create Box Using Composable @Composable fun ResultsOverlay(...) { val detections = results.detections() if (detections != null) { for (detection in detections) { ... Box( modifier = Modifier .border(3.dp, Turquoise) .width(boxWidth.dp) .height(boxHeight.dp) ) Box(modifier = Modifier.padding(3.dp)) { Text( text = resultText, modifier = Modifier .background(Color.Black) .padding(5.dp, 0.dp), color = Color.White, ) } } }
60. Time Skip~ DroidJam 2023
61. 95% Kunyit ML CC CC MD Rp49.999 86% Kencur Rp14.045
62. Summary DroidJam 2023
63. Emang boleh bikin aplikasi AI segampang ini? DroidJam 2023
64. Try the others~
65. DroidJam 2023 References • MediaPipe Documentation • Introducing MediaPipe for On-Device Machine Learning • Introduction to ML on Android with MediaPipe • Easy on-device Machine Learning with MediaPipe • ML Kit: Turnkey APIs to use on-device ML in mobile apps | Session • What's new in Machine Learning for Google Developers
66. "Does my dream have to be success? Can’t it be a person?” – Nam Do San
67. DroidJam 2023 Hatur nuwun! Deck is available at https://speakerdeck.com/arifaizin