Bringing Machine Learning in Android with MediaPipe - DroidJam 2023

Slide 1

Slide 1 text

@arif_faizin Curriculum Developer Lead, Dicoding Indonesia Bringing Machine Learning in Android with MediaPipe DroidJam 2023

Slide 2

Slide 2 text

Let’s Start with Why~ DroidJam 2023

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

DroidJam 2023 65% of consumers have trust in the businesses which use AI technology - Forbes Advisor

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

Intro to ML DroidJam 2023

Slide 8

Slide 8 text

Kai Anak MD Bangkit yang ibunya punya Toko Kelontong

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

Lengkuas Jahe Kencur Kunyit

Slide 11

Slide 11 text

Traditional Programming Rules Input Output

Slide 12

Slide 12 text

Machine Learning Output Input Rules

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

How to? DroidJam 2023

Slide 15

Slide 15 text

On-Cloud On-Device

Slide 16

Slide 16 text

● Lower latency & close knit interactions ● Offline availability ● Privacy preserving ● Cost savings On-Device

Slide 17

Slide 17 text

On-Device ML From Scratch 🤔

Slide 18

Slide 18 text

On-Device ML Using Framework 😎

Slide 19

Slide 19 text

Alternative Framework On-Device ML

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

No content

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

Deep Dive into MediaPipe DroidJam 2023

Slide 24

Slide 24 text

Model is at the core of an on-device ML solution Model Inference Output Input ML app ● Customized to the ad-hoc use cases ● Light-weight and efficient ● Target to hardware ● Sparsified for best performance

Slide 25

Slide 25 text

ML Pipeline streamlines the process from raw inputs to output results Model Inference Output Live Camera ML app Flow Control Post-processing Synchronization Data Preprocessing ● Domain-specific processing (e.g. vision / NLP / audio) ● E2E acceleration across CPU / GPU / EdgeTPU / DSP ● Cross-platform deployment to Android / iOS, web, baremetal

Slide 26

Slide 26 text

Display Live Camera ML app Buffer Management Format Conversion Image Filtering Data Subsampling Data Lifecycle Timestamp Extraction Timestamp Alignment Thread Management GPU/CPU Data Transfer Multi-threaded GPU Compute iOS Metal OpenGL ES Trace Collection Performance Profiling C++ Programming Resource Caching Asset Loading GPU Timing Measurement Cross-platform Abstraction Data Marshalling CPU Affinity Java Native Interface Model Inference Flow Control Post-processing Synchronization Data Preprocessing Both involves a lot of complexity that hinders fast development

Slide 27

Slide 27 text

MediaPipe abstracts this complexity into MediaPipe Tasks Model Inference Display MediaPipe Tasks Live Camera ML app Flow Control Post-processing Synchronization Data Preprocessing

Slide 28

Slide 28 text

… while meeting your custom modeling needs with MediaPipe Model Maker Model Inference Display MediaPipe Model Maker Live Camera ML app Flow Control Post-processing Synchronization Data Preprocessing Custom model

Slide 29

Slide 29 text

No-code GUI with MediaPipe Studio

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

DroidJam 2023

Slide 32

Slide 32 text

Create Image Classification App DroidJam 2023

Slide 33

Slide 33 text

DroidJam 2023 ● Create App to get image from Gallery or Camera ● Alternative solution ○ Gallery ■ PhotoPicker ActivityResultContracts.PickVisualMedia() ■ Intent ACTION_GET_CONTENT ■ Intent ACTION_PICK ○ Camera ■ Intent ACTION_IMAGE_CAPTURE ■ ActivityResultContracts.TakePicture() ■ CameraX Starter Project

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

DroidJam 2023 // 0. build.gradle.kts implementation("com.google.mediapipe:tasks-vision:xxx")

Slide 39

Slide 39 text

DroidJam 2023 // 1. Setup Image Classifier (ImageClassifierHelper.kt) val baseOptionsBuilder = BaseOptions.builder() .setDelegate(Delegate.GPU) // CPU, GPU .setModelAssetPath(MODEL_PATH) val optionsBuilder = ImageClassifier.ImageClassifierOptions.builder() .setScoreThreshold(0.1f) // minimum 10% .setMaxResults(3) .setRunningMode(RunningMode.IMAGE) .setBaseOptions(baseOptionsBuilder.build()) val options = optionsBuilder.build() val imageClassifier = ImageClassifier.createFromOptions(context, options)

Slide 40

Slide 40 text

DroidJam 2023 // 2. Create instance of ImageClassifierHelper // Get Data from Camera (in Activity) // Convert Uri to Bitmap imageUri?.let { uri -> if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.P) { val source = ImageDecoder.createSource(contentResolver, imageUri) ImageDecoder.decodeBitmap(source) } else { MediaStore.Images.Media.getBitmap(contentResolver, uri) }.copy(Bitmap.Config.ARGB_8888, true)?.let { bitmap -> imageClassifierHelper.classifyImage(bitmap) } }

Slide 41

Slide 41 text

Source: https://twitter.com/equasys_de/status/754975190459834368/photo/1

Slide 42

Slide 42 text

DroidJam 2023 // 3. Convert the input Bitmap object to an MPImage object to run inference // ImageClassifierHelper.kt fun classifyImage(bitmap: Bitmap) { val mpImage: MPImage = BitmapImageBuilder(bitmap).build() val imageProcessingOptions = ImageProcessingOptions.builder().build() val startTime = SystemClock.uptimeMillis() imageClassifier?.classify(mpImage, imageProcessingOptions).also { result -> val inferenceTime = SystemClock.uptimeMillis() - startTime imageClassifierListener?.onResults(result, inferenceTime) } if (imageClassifier == null) { imageClassifierListener?.onError( "Image classifier failed to classify." ) } }

Slide 43

Slide 43 text

// Output [Classifications {categories= [ , ], headIndex=0, headName=Optional[probability] } ] Note: It’s only works in real devices, not in Emulator

Slide 44

Slide 44 text

New skill unlocked!

Slide 45

Slide 45 text

Real Time Classification? DroidJam 2023

Slide 46

Slide 46 text

DroidJam 2023 val cameraProviderFuture = ProcessCameraProvider.getInstance(this) cameraProviderFuture.addListener({ val cameraProvider = cameraProviderFuture.get() val preview = Preview.Builder() .setTargetAspectRatio(AspectRatio.RATIO_4_3) .build() .also { it.setSurfaceProvider(binding.viewFinder.surfaceProvider) } cameraProvider.unbindAll() cameraProvider.bindToLifecycle( this, CameraSelector.DEFAULT_BACK_CAMERA, preview ) }, ContextCompat.getMainExecutor(this)) ● Request Permission CAMERA ● CameraX implementation Starter Project

Slide 47

Slide 47 text

DroidJam 2023 // 1. Setup Options Configuration (ImageClassifierHelper.kt) val optionsBuilder = ImageClassifier.ImageClassifierOptions.builder() .setScoreThreshold(0.1f) .setMaxResults(3) .setRunningMode(RunningMode.LIVE_STREAM) // IMAGE, VIDEO, LIVE_STREAM .setBaseOptions(baseOptionsBuilder.build()) if (runningMode == RunningMode.LIVE_STREAM) { optionsBuilder.setResultListener(this::returnLivestreamResult) optionsBuilder.setErrorListener(this::returnLivestreamError) }

Slide 48

Slide 48 text

DroidJam 2023 // 2. Setup ImageAnalysis for CameraX (in Activity) val imageAnalyzer = ImageAnalysis.Builder() .setTargetAspectRatio(AspectRatio.RATIO_4_3) .setTargetRotation(binding.viewFinder.display.rotation) .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST) .setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888) .build() .also { it.setAnalyzer(Executors.newSingleThreadExecutor()) { image -> imageClassifierHelper.classifyLiveStreamFrame(image) } } ... cameraProvider.bindToLifecycle( this, CameraSelector.DEFAULT_BACK_CAMERA, preview, imageAnalyzer )

Slide 49

Slide 49 text

DroidJam 2023 // 3. Convert the input Bitmap object to an MPImage object to run inference fun classifyLiveStreamFrame(image: ImageProxy) { ... val mpImage = BitmapImageBuilder(bitmapBuffer).build() // Used for rotating the frame image so it matches our models val imageProcessingOptions = ImageProcessingOptions.builder() .setRotationDegrees(image.imageInfo.rotationDegrees) .build() val frameTime = SystemClock.uptimeMillis() // Run inference imageClassifier?.classifyAsync(mpImage, imageProcessingOptions, frameTime) }

Slide 50

Slide 50 text

“Your focus determines your reality.” – Qui-Gon Jinn

Slide 51

Slide 51 text

Object Detection: A new Hope DroidJam 2023

Slide 52

Slide 52 text

No content

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

DroidJam 2023 // 1. Setup Object Detector (ObjectDetectorHelper.kt) val baseOptionsBuilder = BaseOptions.builder() .setDelegate(Delegate.GPU) // CPU, GPU .setModelAssetPath(MODEL_PATH) val optionsBuilder = ObjectDetector.ObjectDetectorOptions.builder() .setScoreThreshold(0.1f) .setMaxResults(3) .setRunningMode(RunningMode.LIVE_STREAM) .setBaseOptions(baseOptionsBuilder.build()) val options = optionsBuilder.build() val imageClassifier = ImageClassifier.createFromOptions(context, options)

Slide 55

Slide 55 text

DroidJam 2023 // 2. Setup ImageAnalysis for CameraX (in Activity) val imageAnalyzer = ImageAnalysis.Builder() .setTargetAspectRatio(AspectRatio.RATIO_16_9) // adjust with model .setTargetRotation(binding.viewFinder.display.rotation) .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST) .setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888) .build() .also { it.setAnalyzer(Executors.newSingleThreadExecutor()) { image -> objectDetectorHelper.detectLivestreamFrame(image) } }

Slide 56

Slide 56 text

DroidJam 2023 // 3. Convert the input Bitmap object to an MPImage object to run inference fun detectLiveStreamFrame(image: ImageProxy) { ... val mpImage = BitmapImageBuilder(bitmapBuffer).build() // Used for rotating the frame image so it matches our models val imageProcessingOptions = ImageProcessingOptions.builder() .setRotationDegrees(image.imageInfo.rotationDegrees) .build() val frameTime = SystemClock.uptimeMillis() // Run inference objectDetector?.detectAsync(mpImage, imageProcessingOptions, frameTime) }

Slide 57

Slide 57 text

New skill unlocked!

Slide 58

Slide 58 text

DroidJam 2023 // 4a. Draw Box Using Custom View in XML class OverlayView(context: Context?, attrs: AttributeSet?) : View(context, attrs) { override fun draw(canvas: Canvas) { super.draw(canvas) // Draw bounding box around detected objects val drawableRect = RectF(left, top, right, bottom) canvas.drawRect(drawableRect, boxPaint) // Draw text for detected object canvas.drawText( drawableText, left, top + bounds.height(), textPaint ) } } }

Slide 59

Slide 59 text

DroidJam 2023 // 4b. Create Box Using Composable @Composable fun ResultsOverlay(...) { val detections = results.detections() if (detections != null) { for (detection in detections) { ... Box( modifier = Modifier .border(3.dp, Turquoise) .width(boxWidth.dp) .height(boxHeight.dp) ) Box(modifier = Modifier.padding(3.dp)) { Text( text = resultText, modifier = Modifier .background(Color.Black) .padding(5.dp, 0.dp), color = Color.White, ) } } }

Slide 60

Slide 60 text

Time Skip~ DroidJam 2023

Slide 61

Slide 61 text

95% Kunyit ML CC CC MD Rp49.999 86% Kencur Rp14.045

Slide 62

Slide 62 text

Summary DroidJam 2023

Slide 63

Slide 63 text

Emang boleh bikin aplikasi AI segampang ini? DroidJam 2023

Slide 64

Slide 64 text

Try the others~

Slide 65

Slide 65 text

DroidJam 2023 References ● MediaPipe Documentation ● Introducing MediaPipe for On-Device Machine Learning ● Introduction to ML on Android with MediaPipe ● Easy on-device Machine Learning with MediaPipe ● ML Kit: Turnkey APIs to use on-device ML in mobile apps | Session ● What's new in Machine Learning for Google Developers

Slide 66

Slide 66 text

"Does my dream have to be success? Can’t it be a person?” – Nam Do San

Slide 67

Slide 67 text

DroidJam 2023 Hatur nuwun! Deck is available at https://speakerdeck.com/arifaizin