Applied Machine Learning for Android Developer

This presentation, "Applied Machine Learning for Android Developers," begins with a warm-up activity and an introduction to machine learning concepts. It uses a case study of a grocery store owner's son, Andro, who wants to build a point-of-sale app but doesn't know the names of all the products. The presentation explains how Andro can use image classification to solve this problem and provides step-by-step instructions on how to implement image classification in an Android app using frameworks like ML Kit, TensorFlow Lite, and MediaPipe. It also includes a live demo of image classification and a sample app that showcases collaboration between ML, CC, and MD cohorts.

Transcript

1. This work is licensed under the Apache 2.0 License Applied Machine Learning for Android Developer Fachridan Tio Mu’afa Software Engineer - Android Nusantara Beta Studio (NBS)
2. This work is licensed under the Apache 2.0 License Fachridan Tio Mu’afa Latest Work Experiences: • Software Engineer - Android, NBS 2024 - Present • Android Developer, Amanah Corp. 2023 - 2024 • Android Developer, Fishku Indonesia 2022 - 2023 • Mobile Development (Android) Mentor 2023 - 2024 • Mobile Development (Android) Instructor & Advisor 2024 - Present Education: • UIN Syarif Hidayatullah Jakarta 2019 - 2024 Bachelor of Computer Science Others • Discord Manager Lead, GoogleDevsID 2022 - Present • GDG Co-Organizer, GDG Jakarta 2023 - Present • GDSC Lead, UIN Syarif Hidayatullah 2021 - 2022 About Me
3. This work is licensed under the Apache 2.0 License What AI products do you often use?
4. This work is licensed under the Apache 2.0 License 65% of consumers have trust in the businesses which use AI technology - Forbes Advisor
5. This work is licensed under the Apache 2.0 License Intro to ML
6. This work is licensed under the Apache 2.0 License Andro MD student whose parents owns a grocery store
7. This work is licensed under the Apache 2.0 License Lengkuas (galangal) Jahe (ginger) Kencur (aromatic ginger) Kunyit (turmeric)
8. This work is licensed under the Apache 2.0 License Machine Learning Traditional Programming
9. This work is licensed under the Apache 2.0 License Illustration - Activity Recognition
10. This work is licensed under the Apache 2.0 License Activity Recognition - Using Machine Learning
11. This work is licensed under the Apache 2.0 License Activity Recognition - Using Machine Learning
12. How-to? This work is licensed under the Apache 2.0 License
13. This work is licensed under the Apache 2.0 License On-Cloud On-Device ML Implementations in Android
14. This work is licensed under the Apache 2.0 License On-Cloud • High Performance • Scalability • Centralized Management • Access to Data • No new knowledge needed ◦ Only using networking to send and receive result from the cloud ◦ Using multipart to send file
15. This work is licensed under the Apache 2.0 License Upload File using Retrofit
16. This work is licensed under the Apache 2.0 License • Lower latency & close knit interactions • Offline availability • Privacy preserving • Cost savings On-Device
17. This work is licensed under the Apache 2.0 License Gather Data Train Model Build a Model Pipeline Optimize Model + HW Acceleration On-Device ML From Scratch 🤔 ML Ship it! 🚀 Test on Multiple Devices Integrate with Our App Testing App Performance & Quality Android
18. This work is licensed under the Apache 2.0 License On-Device ML Using Framework 😎
19. This work is licensed under the Apache 2.0 License Alternative Frameworks for On-Device ML Tagline Easy-to-use, production-ready APIs to tackle common task Deploy self pre-trained models for specific use cases Advanced vision solutions for live and streaming media Supports Mobile Mobile & IoT Mobile, Web, Desktop, & IoT Provide built-in models Yes No No Case Flexibility Limited to provided API Very flexible because there is Interpreter API No Interpreter API Unique Solution Entity extraction, Translate, Barcode Scanning Question & Answer, Text Searcher, Style Transfer Hand Landmark, Face Landmark, Pose Landmark
20. This work is licensed under the Apache 2.0 License TensorFlow Lite rebranded to Lite RT 🔥 As per 04 Sept 2024, TensorFlow Lite is officially rebranded to Lite RT. This rebranding is Google’s commitment to supporting today’s thriving various Machine Learning framework ecosystem. Worry not, the changes happening are limited to the name for now, so you can still use your existing TensorFlow Lite code in your project. Please note that future development of TensorFlow Lite will continue under the name LiteRT. More info: TensorFlow Lite is now LiteRT
21. This work is licensed under the Apache 2.0 License
22. This work is licensed under the Apache 2.0 License Image Classification Cat Object Detection Dog, Cat Image Segmentation Dog, Cat
23. This work is licensed under the Apache 2.0 License Pre-process image Step-by-Step Guide to Implement Image Classification App Get image from camera or gallery 1 Prepare the ML models 2 Initialize Task Library 3 4 5 Run inference 6 Show data
24. This work is licensed under the Apache 2.0 License Get Image From Gallery/Camera
25. This work is licensed under the Apache 2.0 License Get Image From Gallery/Camera • Create App to get image from Gallery or Camera • Alternative solution ◦ Gallery ▪ PhotoPicker ActivityResultContracts.PickVisualMedia() ▪ Intent ACTION_GET_CONTENT ▪ Intent ACTION_PICK ◦ Camera ▪ Intent ACTION_IMAGE_CAPTURE ▪ ActivityResultContracts.TakePicture() ▪ CameraX
26. This work is licensed under the Apache 2.0 License launcherGallery.launch(PickVisualMediaRequest( ActivityResultContracts.PickVisualMedia.ImageOnly) ) private val launcherGallery = registerForActivityResult( ActivityResultContracts.PickVisualMedia() ) { uri: Uri? -> } The photo picker is a browsable interface that presents the user with their media library, sorted by date from newest to oldest, and integrates nicely with your app’s experience without requiring media storage permissions! Intent Gallery using Photo Picker
27. This work is licensed under the Apache 2.0 License val intent = Intent(MediaStore.ACTION_IMAGE_CAPTURE) intent.putExtra(MediaStore.EXTRA_OUTPUT, photoUri) launcherIntentCamera.launch(intent) private val launcherIntentCamera = registerForActivityResult( ActivityResultContracts.StartActivityForResult() ) { result -> if (result.resultCode == RESULT_OK) { } } To perform basic camera actions like capturing a photo or video using the device's default camera application, you do not need to integrate with a CameraX library. Instead, use an Intent. Intent Camera
28. This work is licensed under the Apache 2.0 License Prepare the ML Models
29. This work is licensed under the Apache 2.0 License Find ML Model
30. This work is licensed under the Apache 2.0 License Test ML Model
31. This work is licensed under the Apache 2.0 License Test ML Model
32. This work is licensed under the Apache 2.0 License Import ML Model
33. This work is licensed under the Apache 2.0 License Read Metadata from ML Model
34. This work is licensed under the Apache 2.0 License Create Image Classification App
35. This work is licensed under the Apache 2.0 License Setup ImageClassifier private fun setupImageClassifier() { val optionsBuilder = ImageClassifier.ImageClassifierOptions.builder() .setScoreThreshold(0.5f) // minimum 10% .setMaxResults(3) // maximum results displayed val baseOptionsBuilder = BaseOptions.builder() .setNumThreads(4) // number of threads to be used optionsBuilder.setBaseOptions(baseOptionsBuilder.build()) try { imageClassifier = ImageClassifier.createFromFileAndOptions( context, "your_model_name.tflite", // number of optionsBuilder.build() ) } catch (e: IllegalStateException) { listener?.onError(context.getString(R.string.image_classifier_failed)) Log.e(TAG, e.message.toString()) } }
36. This work is licensed under the Apache 2.0 License Convert to TensorImage & Perform Inference val imageProcessor = ImageProcessor.Builder().build() // convert uri to Bitmap if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.P) { val source = ImageDecoder.createSource(context.contentResolver, imageUri) ImageDecoder.decodeBitmap(source) } else { MediaStore.Images.Media.getBitmap(context.contentResolver, imageUri) }.copy(Bitmap.Config.ARGB_8888, true)?.let { bitmap -> // convert Bitmap to TensorImage val tensorImage = imageProcessor.process(TensorImage.fromBitmap(bitmap)) var inferenceTime = SystemClock.uptimeMillis() // run inference val results = imageClassifier?.classify(tensorImage) inferenceTime = SystemClock.uptimeMillis() - inferenceTime listener?.onResults( results, inferenceTime ) }
37. This work is licensed under the Apache 2.0 License Show Output // Output [Classifications {categories= [ <Category "computer keyboard" (displayName= score=0.41453125 index=621)>, <Category "laptop" (displayName= score=0.35921875 index=509)> ], headIndex=0, headName=Optional[probability] } ]
38. This work is licensed under the Apache 2.0 License New skill unlocked!
39. This work is licensed under the Apache 2.0 License Real-time Image Classification
40. This work is licensed under the Apache 2.0 License CameraX val cameraProviderFuture = ProcessCameraProvider.getInstance(this) cameraProviderFuture.addListener({ val cameraProvider = cameraProviderFuture.get() val preview = Preview.Builder().build().also { it.setSurfaceProvider(binding.viewFinder.surfaceProvider) } cameraProvider.unbindAll() cameraProvider.bindToLifecycle( this, cameraSelector, preview ) }, ContextCompat.getMainExecutor(this))
41. This work is licensed under the Apache 2.0 License ImageAnalysis at CameraX val resolutionSelector = ResolutionSelector.Builder() .setAspectRatioStrategy(AspectRatioStrategy.RATIO_16_9_FALLBACK_AUTO_STRATEGY) .build() val imageAnalysis = ImageAnalysis.Builder() .setResolutionSelector(resolutionSelector) .setTargetRotation(binding.viewFinder.display.rotation) .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST) .setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888) .build() imageAnalysis.setAnalyzer(Executors.newSingleThreadExecutor()) { image: ImageProxy -> imageClassifierHelper.classifyLiveImage(image) } ... cameraProvider.bindToLifecycle( this, cameraSelector, preview, imageAnalysis )
42. This work is licensed under the Apache 2.0 License Convert ImageProxy to TensorImage fun classifyLiveImage(image: ImageProxy) { // convert imageProxy to Bitmap val bitmapBuffer = Bitmap.createBitmap( image.width, image.height, Bitmap.Config.ARGB_8888 ) image.use { bitmapBuffer.copyPixelsFromBuffer(image.planes[0].buffer) } image.close() val tensorImage: TensorImage = imageProcessor.process(TensorImage.fromBitmap(bitmapBuffer)) ImageProxy is the name of the wrapper class used to represent images in raw format taken from the camera.
43. This work is licensed under the Apache 2.0 License Perform Inference private fun getOrientationFromRotation(rotation: Int): ImageProcessingOptions.Orientation { return when (rotation) { Surface.ROTATION_270 -> ImageProcessingOptions.Orientation.BOTTOM_RIGHT Surface.ROTATION_180 -> ImageProcessingOptions.Orientation.RIGHT_BOTTOM Surface.ROTATION_90 -> ImageProcessingOptions.Orientation.TOP_LEFT else -> ImageProcessingOptions.Orientation.RIGHT_TOP } } val imageProcessingOptions = ImageProcessingOptions.builder() .setOrientation(getOrientationFromRotation(image.imageInfo.rotationDegrees)) .build() var inferenceTime = SystemClock.uptimeMillis() val results = imageClassifier?.classify(tensorImage, imageProcessingOptions) inferenceTime = SystemClock.uptimeMillis() - inferenceTime classifierListener?.onResults( results, inferenceTime )
44. This work is licensed under the Apache 2.0 License Demo Link https://github.com/fachridantm/ImageObjectDetector
45. This work is licensed under the Apache 2.0 License “Your focus determines your reality.” – Qui-Gon Jinn
46. This work is licensed under the Apache 2.0 License Sample App ML CC CC MD
47. This work is licensed under the Apache 2.0 License Let’s keep connected! THANK YOU fachridantm.me