Tweet
Tweet

Slide 1

Slide 1 text

Exploring AR, ML and Camera-related APIs on Android Now smile (also in 3D)! 🤖📸✨ Walmyr Carvalho Lead Android Engineer @ Polaroid, Google Developer Expert for Android

Slide 2

Slide 2 text

1.Fact: people use cameras a lot!

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Another fact: we’re constantly using cameras through our phones.

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

Instagram SPACE INVADERS: World Defense Bird

Slide 7

Slide 7 text

It became a very important, social tool across time, in good and bad ways.

Slide 8

Slide 8 text

Source: BBC TikTok (@pinkydollreal)

Slide 9

Slide 9 text

It’s such a powerful integration to have in our apps, so we should be mindful on what we’re trying to build.

Slide 10

Slide 10 text

“OK Walmyr, what about Polaroid? which Camera APIs do you use?“

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

Although they’re connected devices, Polaroid cameras are analog by design. But we do have some cool app features using custom Camera APIs, like photo scanning with frame recognition and custom AR pictures! 🪄🦄 Photo Scanner

Slide 13

Slide 13 text

Polaroid Lab - AR Photo Preview

Slide 14

Slide 14 text

2.Android Camera 101

Slide 15

Slide 15 text

If you’re planning to have only basic photo and video functionality on your app, you should just call the regular system Intents for it!

Slide 16

Slide 16 text

// Photo capture intent val REQUEST_IMAGE_CAPTURE = 1 private fun dispatchTakePictureIntent() { val takePictureIntent = Intent(MediaStore.ACTION_IMAGE_CAPTURE) try { startActivityForResult(takePictureIntent, REQUEST_IMAGE_CAPTURE) } catch (e: ActivityNotFoundException) { // Handle error } }

Slide 17

Slide 17 text

// Video capture intent val REQUEST_VIDEO_CAPTURE = 1 private fun dispatchTakeVideoIntent() { Intent(MediaStore.ACTION_VIDEO_CAPTURE).also { takeVideoIntent -> takeVideoIntent.resolveActivity(packageManager)?.also { startActivityForResult(takeVideoIntent, REQUEST_VIDEO_CAPTURE) } ?: run { // Handle error } } }

Slide 18

Slide 18 text

For a more custom usage of Camera APIs, today we have three options: CameraX, Camera2 and Camera.

Slide 19

Slide 19 text

For a more custom usage of Camera APIs, today we have three (ish) options: CameraX, Camera2 and Camera (deprecated).

Slide 20

Slide 20 text

dependencies { def camerax_version = "1.4.0" implementation "androidx.camera:camera-core:${camerax_version}" implementation "androidx.camera:camera-camera2:${camerax_version}" implementation "androidx.camera:camera-lifecycle:${camerax_version}" implementation "androidx.camera:camera-video:${camerax_version}" implementation "androidx.camera:camera-view:${camerax_version}" implementation "androidx.camera:camera-extensions:${camerax_version}" }

Slide 21

Slide 21 text

// Permissions required (define permissions in way you prefer)

Slide 22

Slide 22 text

// Setup a ProcessCameraProvider cameraProvider = ProcessCameraProvider.getInstance(requireContext()).await() // Select lensFacing depending on the available cameras lensFacing = when { hasBackCamera() -> CameraSelector.LENS_FACING_BACK hasFrontCamera() -> CameraSelector.LENS_FACING_FRONT else -> throw IllegalStateException("Back and front camera are unavailable") } // If you want to change the different cameras, you need a CameraSelector val cameraSelector = CameraSelector.Builder().requireLensFacing(lensFacing).build()

Slide 23

Slide 23 text

// Preview preview = Preview.Builder() .setTargetAspectRatio(screenAspectRatio) .setTargetRotation(rotation) .build() // ImageCapture imageCapture = ImageCapture.Builder() .setCaptureMode(ImageCapture.CAPTURE_MODE_MINIMIZE_LATENCY) .setTargetAspectRatio(screenAspectRatio) .setTargetRotation(rotation) .build()

Slide 24

Slide 24 text

try { // Unbind use cases before rebinding cameraProvider.unbindAll() // Bind use cases to camera cameraProvider.bindToLifecycle( this, cameraSelector, preview, imageCapture) } catch(exception: Exception) { Log.e(TAG, "Use case binding failed", exception) }

Slide 25

Slide 25 text

// Get a stable reference of the modifiable image capture use case val imageCapture = imageCapture ?: return // Create time stamped name and MediaStore entry. val name = SimpleDateFormat(FILENAME_FORMAT, Locale.US) .format(System.currentTimeMillis()) val contentValues = ContentValues().apply { put(MediaStore.MediaColumns.DISPLAY_NAME, name) put(MediaStore.MediaColumns.MIME_TYPE, "image/jpeg") if(Build.VERSION.SDK_INT > Build.VERSION_CODES.P) { put(MediaStore.Images.Media.RELATIVE_PATH, "Pictures/CameraX-Image") } }

Slide 26

Slide 26 text

// Create output options object which contains file + metadata val outputOptions = ImageCapture.OutputFileOptions .Builder(contentResolver, MediaStore.Images.Media.EXTERNAL_CONTENT_URI, contentValues) .build()

Slide 27

Slide 27 text

// Set up image capture listener to get the photo action callback imageCapture.takePicture( outputOptions, ContextCompat.getMainExecutor(this), object : ImageCapture.OnImageSavedCallback { override fun onError(exc: ImageCaptureException) { Log.e(TAG, "Photo capture failed: ${exc.message}", exc) } override fun onImageSaved(output: ImageCapture.OutputFileResults){ val msg = "Photo capture succeeded: ${output.savedUri}" Toast.makeText(baseContext, msg, Toast.LENGTH_SHORT).show() Log.d(TAG, msg) } } )

Slide 28

Slide 28 text

These examples and more are available at the official code lab repo: developer.android.com/codelabs/camerax-getting-started

Slide 29

Slide 29 text

CameraX support many other use cases, such as VideoCapture, ImageAnalysis, dual concurrent camera and more, please try it out! 🙏

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

OK, Camera APIs are cool but… is that it? 😬

Slide 32

Slide 32 text

3.AR capabilities

Slide 33

Slide 33 text

What’s AR? 👀

Slide 34

Slide 34 text

Introducing AR Core - Google AR & VR (Youtube)

Slide 35

Slide 35 text

"Augmented reality (AR) is an interactive experience that combines the real world and computer-generated content. The content can span multiple sensory modalities, including visual, auditory, haptic, somatosensory and olfactory." - Wikipedia

Slide 36

Slide 36 text

Android Experiments - Norman AR

Slide 37

Slide 37 text

OK, so how is the best way to make AR experiences on Android? 🤔

Slide 38

Slide 38 text

developers.google.com/ar

Slide 39

Slide 39 text

AR Core tools & capabilities • Motion tracking for relative positioning to your surroundings; • Environmental awareness for anchor and object placing support; • Depth awareness for tracking and distance measurements; • Lighting awareness for illumination checks and color corrections; • Unity and Adobe Aero support; • Many more! Source: Google AR Core

Slide 40

Slide 40 text

Geospace Creator (Adobe Aero + Unity)

Slide 41

Slide 41 text

4. ML capabilities

Slide 42

Slide 42 text

developers.google.com/ml-kit

Slide 43

Slide 43 text

ML Kit overview • Machine Learning common tools, but optimised for mobile usage • Easy-to-implement Vision APIs, such as text, pose and face detection, image labelling, barcode scanning • Natural language functionalities, such as language identification, translation and more; Source: Google Codelabs (mlkit-android)

Slide 44

Slide 44 text

developers.google.com/mediapipe

Slide 45

Slide 45 text

MediaPipe overview • Cross-platform (mobile, web, desktop) solution for common ML tasks; • Empowers some of the ML Kit functionalities under the hood; • More powerful tools to create custom ML use cases; • More applied to real-time perception tasks in a low-latency experience; • Low-code APIs to create custom ML solutions Source: MediaPipe

Slide 46

Slide 46 text

tensorflow.org/lite

Slide 47

Slide 47 text

TensorFlow Lite overview • Really powerful tool to create and train deep learning ML models; • High-level APIs integration, like Keras; • Android, iOS, embedded Linux and microcontroller support; • Diverse language support (Java/Kotlin, Swift, Objective-C, C++ and Python); • Powerful tooling (like monitoring, deployment, etc) powered by its big brother, Tensorflow; Source: TensorFlow Lite (Google)

Slide 48

Slide 48 text

tensorflow.org/lite/examples

Slide 49

Slide 49 text

kaggle.com

Slide 50

Slide 50 text

OK Walmyr, that’s quite a lot! 🤯 Can we combine all those things?

Slide 51

Slide 51 text

Camera hardware + preview 🤳 AR Environment Renderer Google Cloud Vision API Image Classification Model ML Kit Object Detection + TensorFlow Lite offline model +

Slide 52

Slide 52 text

Demo time! 🪄

Slide 53

Slide 53 text

ARCore + ML Kit: github.com/googlesamples/arcore-ml-sample

Slide 54

Slide 54 text

AR Core Depth Lab github.com/googlesamples/arcore-depth-lab

Slide 55

Slide 55 text

5. To summarise

Slide 56

Slide 56 text

Key takes to bring home 1.CameraX is the way to go when it comes to use camera (and its customisations) on Android; 2.ML Kit, AR Core and TensorFlow Lite are really a powerful combination; 3.Cloud Vision APIs can actually extend the capabilities of your project but fetching larger, up to date models. 4.You can use Camera APIS, AR and ML to any purposes, being it either a game or a tool to help people engage with your business! 5.There’s plenty of content available to learn and explore, especially Codelabs; 6.Have fun and… smile! 👋 📸

Slide 57

Slide 57 text

Walmyr Carvalho Lead Android Engineer @ Polaroid, Google Developer Expert for Android Thank you so much! ❤ The slides, links and sources will be available soon, please reach out on Twitter!