Slide 1
Slide 1 text
Smile,
it’s CameraX!
Magda Miu @magdamiu
Squad Lead Developer at Orange
Android Google Developer Expert
Slide 2
Slide 2 text
Birthday party 2015
Slide 3
Slide 3 text
Christmas 2016
Slide 4
Slide 4 text
Tančící dům 2017
Slide 5
Slide 5 text
Pizzaaaaa
Slide 6
Slide 6 text
It's Cocktail Time
Slide 7
Slide 7 text
mDevCamp 2018
Slide 8
Slide 8 text
Just happiness
Slide 9
Slide 9 text
Discovering that the food order has 1 hour delay
Slide 10
Slide 10 text
The last_last_last_..._last_selfie
A Picture Is Worth
a Thousand Words
Photos help us to
tell stories
Slide 11
Slide 11 text
The last_last_last_..._last_selfie
A Picture Is Worth
a Thousand Words
Photos help us to
tell stories
Self-portraits are
about self-image
“looking-glass
self”
Slide 12
Slide 12 text
Total photos taken yearly
Slide 13
Slide 13 text
Digital camera vs Phone vs Tablet
87.5% Phone
10% Digital camera
2.5% Tablet
89.8% Phone
8.2% Digital camera
2% Tablet
90.9% Phone
7.3% Digital camera
1.8% Tablet
Slide 14
Slide 14 text
Challenges
OS flavors
Platform
fragmentation
Camera API
complexity
Slide 15
Slide 15 text
Camera APIs
01
Legacy
android.hardware.Camera
02
Camera 2
android.hardware.camera2
03
CameraX
androidx.camera
Slide 16
Slide 16 text
Challenge Solution
OS flavors
Backward
compatible with
L+
devices
Slide 17
Slide 17 text
Platform
fragmentation
Consistent
behavior
across
devices
Challenge Solution
Slide 18
Slide 18 text
Easy to use
API
Camera API
complexity
Challenge Solution
Slide 19
Slide 19 text
Fewer lines of code by using CameraX vs Camera2
70%
Slide 20
Slide 20 text
CameraX Lifecycle
Lifecycle awareness
Slide 21
Slide 21 text
Use-case-driven approach
UseCase
ImageCapture ImageAnalysis
Preview
Slide 22
Slide 22 text
Preview
01
Get an image on the display
Slide 23
Slide 23 text
Preview - implementation steps
Step 1
Add gradle
dependencies
Step 2
Permission
handling
Step 3
Add
PreviewView in
a layout
Step 4
Get an
instance of
ProcessCamera
Provider
Step 5
Select a
camera and
bind the
lifecycle
Slide 24
Slide 24 text
compileOptions {
sourceCompatibility JavaVersion.VERSION_1_8
targetCompatibility JavaVersion.VERSION_1_8
}
Step 1: gradle setup
Slide 25
Slide 25 text
def camerax = "1.0.0-beta04"
implementation "androidx.camera:camera-camera2:${camerax}"
implementation "androidx.camera:camera-lifecycle:${camerax}"
implementation 'androidx.camera:camera-view:1.0.0-alpha11'
implementation 'androidx.camera:camera-extensions:1.0.0-alpha11'
Step 1: gradle setup
Slide 26
Slide 26 text
if (areAllPermissionsGranted()) {
startCamera()
} else {
ActivityCompat.requestPermissions(
this, PERMISSIONS, CAMERA_REQUEST_PERMISSION_CODE)
}
Step 2: permission
handling
Slide 27
Slide 27 text
if (areAllPermissionsGranted()) {
startCamera()
} else {
ActivityCompat.requestPermissions(
this, PERMISSIONS, CAMERA_REQUEST_PERMISSION_CODE)
}
Step 2: permission
handling
Slide 28
Slide 28 text
Step 3: add PreviewView
in a layout
Slide 29
Slide 29 text
PreviewView
SurfaceView
TextureView
FrameLayout
CameraCharacteristics
#INFO_SUPPORTED
_HARDWARE_LEVEL
_LEGACY
YES NO
ViewGroup
Slide 30
Slide 30 text
val previewView = findViewById(R.id.preview)
// initialize the Preview object (current use case)
val preview = Preview.Builder().build()
Step 3: add PreviewView
in a layout
Slide 31
Slide 31 text
val cameraProviderFuture =
ProcessCameraProvider.getInstance(this)
// used to bind the lifecycle of camera to the lifecycle owner
val cameraProvider: ProcessCameraProvider =
cameraProviderFuture.get()
// add a listener to the cameraProviderFuture
val cameraExecutor = ContextCompat.getMainExecutor(this)
cameraProviderFuture.addListener(Runnable {}, cameraExecutor)
Step 4: instance of
ProcessCameraProvider
Slide 32
Slide 32 text
val cameraProviderFuture =
ProcessCameraProvider.getInstance(this)
// used to bind the lifecycle of camera to the lifecycle owner
val cameraProvider: ProcessCameraProvider =
cameraProviderFuture.get()
// add a listener to the cameraProviderFuture
val cameraExecutor = ContextCompat.getMainExecutor(this)
cameraProviderFuture.addListener(Runnable {}, cameraExecutor)
Step 4: instance of
ProcessCameraProvider
Slide 33
Slide 33 text
val cameraProviderFuture =
ProcessCameraProvider.getInstance(this)
// used to bind the lifecycle of camera to the lifecycle owner
val cameraProvider: ProcessCameraProvider =
cameraProviderFuture.get()
// add a listener to the cameraProviderFuture
val cameraExecutor = ContextCompat.getMainExecutor(this)
cameraProviderFuture.addListener(Runnable {}, cameraExecutor)
Step 4: instance of
ProcessCameraProvider
Slide 34
Slide 34 text
val backCamera = CameraSelector.LENS_FACING_BACK;
val frontCamera = CameraSelector.LENS_FACING_FRONT;
val cameraSelector =
CameraSelector.Builder().requireLensFacing(backCamera).build()
Step 5: select camera
Slide 35
Slide 35 text
// unbind use cases before rebinding
cameraProvider.unbindAll();
// bind the preview use case to camera
camera = cameraProvider.bindToLifecycle(
this as LifecycleOwner,
cameraSelector,
preview
)
preview?.setSurfaceProvider(
previewView.createSurfaceProvider(camera?.cameraInfo))
Step 5: bind the
lifecycle (try-catch)
*Runnable - try catch
Slide 36
Slide 36 text
// unbind use cases before rebinding
cameraProvider.unbindAll();
// bind the preview use case to camera
camera = cameraProvider.bindToLifecycle(
this as LifecycleOwner,
cameraSelector,
preview
)
preview?.setSurfaceProvider(
previewView.createSurfaceProvider(camera?.cameraInfo))
Step 5: bind the
lifecycle (try-catch)
*Runnable - try catch
Slide 37
Slide 37 text
val previewView = findViewById(R.id.preview)
val preview = Preview.Builder().build()
val cameraProviderFuture = ProcessCameraProvider.getInstance(this)
val cameraProvider: ProcessCameraProvider = cameraProviderFuture.get()
val cameraExecutor = ContextCompat.getMainExecutor(this)
cameraProviderFuture.addListener(Runnable {
cameraProvider.unbindAll();
camera = cameraProvider.bindToLifecycle(
this as LifecycleOwner,
cameraSelector,
preview
)
preview?.setSurfaceProvider(
previewView.createSurfaceProvider(camera?.cameraInfo))
}, cameraExecutor)
try catch
Slide 38
Slide 38 text
No content
Slide 39
Slide 39 text
Capture
02
Save high-quality images
Slide 40
Slide 40 text
Capture - implementation steps
Step 1
Create
ImageCapture
reference
Step 2
Add
orientation
event listener
Step 3
Image file
management
Step 4
Call
takePicture()
Step 5
Update the
call to bind
lifecycle
Slide 41
Slide 41 text
val imageCapture = ImageCapture.Builder().build()
Step 1: create
ImageCapture reference
Slide 42
Slide 42 text
// to optimize photo capture for quality
val captureMode = ImageCapture.CAPTURE_MODE_MAXIMIZE_QUALITY
// to optimize photo capture for latency (default)
val captureMode = ImageCapture.CAPTURE_MODE_MINIMIZE_LATENCY
imageCapture = ImageCapture.Builder()
.setCaptureMode(captureMode)
.build()
Step 1: create
ImageCapture reference
*photo capture mode
Slide 43
Slide 43 text
// flash will always be used when taking a picture
val flashMode = ImageCapture.FLASH_MODE_ON
// flash will never be used when taking a picture (default)
val flashMode = ImageCapture.FLASH_MODE_OFF
// flash will be used according to the camera system's
determination
val flashMode = ImageCapture.FLASH_MODE_AUTO
imageCapture = ImageCapture.Builder()
.setFlashMode(flashMode)
.build()
Step 1: create
ImageCapture reference
*flash mode
Slide 44
Slide 44 text
// 16:9 standard aspect ratio
val aspectRatio = AspectRatio.RATIO_16_9
// 4:3 standard aspect ratio (default)
val aspectRatio = AspectRatio.RATIO_4_3
imageCapture = ImageCapture.Builder()
.setTargetAspectRatio(aspectRatio)
.build()
Step 1: create
ImageCapture reference
*aspect ratio
Slide 45
Slide 45 text
val metrics = DisplayMetrics().also {
previewView.display.getRealMetrics(it) }
val screenSize = Size(metrics.widthPixels,
metrics.heightPixels)
imageCapture = ImageCapture.Builder()
.setTargetResolution(screenSize)
.setTargetName("CameraConference")
.build()
Step 1: create
ImageCapture reference
*target resolution and
target name
Slide 46
Slide 46 text
val orientationEventListener = object :
OrientationEventListener(this as Context) {
override fun onOrientationChanged(orientation: Int) {
val rotation: Int = when (orientation) {
in 45..134 -> Surface.ROTATION_270
in 135..224 -> Surface.ROTATION_180
in 225..314 -> Surface.ROTATION_90
else -> Surface.ROTATION_0
}
// default => Display.getRotation()
imageCapture.targetRotation = rotation
}
}
orientationEventListener.enable()
Step 2: add orientation
event listener
Slide 47
Slide 47 text
val file = File(
externalMediaDirs.first(),
"${System.currentTimeMillis()}.jpg"
)
val outputFileOptions =
ImageCapture.OutputFileOptions.Builder(file).build()
Step 3: image file
management
Slide 48
Slide 48 text
imageCapture.takePicture(outputFileOptions, cameraExecutor,
object : ImageCapture.OnImageSavedCallback {
override fun onImageSaved(outputFileResult:
ImageCapture.OutputFileResults) {
// yey!!! :)
}
override fun onError(exception: ImageCaptureException) {
// ohhh!!! :(
}
})
Step 4: call takePicture()
Slide 49
Slide 49 text
// bind the image capture use case to camera
camera = cameraProvider.bindToLifecycle(
this as LifecycleOwner,
cameraSelector,
preview,
imageCapture
)
Step 5: update the call
to bind lifecycle
Slide 50
Slide 50 text
takePicture implementation options
takePicture(Executor, OnImageCapturedCallback)
takePicture(OutputFileOptions, Executor, OnImageSavedCallback)
in-memory buffer of
the captured image
save the captured
image to the
provided file
location
Slide 51
Slide 51 text
val imageCapture = ImageCapture.Builder().build()
val file = File(externalMediaDirs.first(), name)
val output = ImageCapture.OutputFileOptions.Builder(file).build()
imageCapture.takePicture(output, executor,
object : ImageCapture.OnImageSavedCallback {
override fun onImageSaved(ofr: ImageCapture.OutputFileResults) {
// yey!!! :)
}
override fun onError(exception: ImageCaptureException) {
// ohhh!!! :(
}
})
camera = cameraProvider.bindToLifecycle(
this as LifecycleOwner,
cameraSelector,
preview,
imageCapture
)
Slide 52
Slide 52 text
No content
Slide 53
Slide 53 text
Analysis
03
CPU-accessible image for
image processing, computer
vision, ML
Slide 54
Slide 54 text
Analysis - implementation steps
Step 1
Create
ImageAnalysis
reference
Step 2
Define a
custom
analyser
Step 3
Implement
analyze()
method
Step 4
Set the
custom
analyser
Step 5
Update the
call to bind
lifecycle
Slide 55
Slide 55 text
val imageAnalysis = ImageAnalysis.Builder().build()
Step 1: create
ImageAnalysis reference
Slide 56
Slide 56 text
// the executor receives sequentially the frames
val blocking = ImageAnalysis.STRATEGY_BLOCK_PRODUCER
// the executor receives last available frame (default)
val nonBlocking = ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST
val imageAnalysis = ImageAnalysis.Builder()
.setBackpressureStrategy(nonBlocking)
.build()
Step 1: create
ImageAnalysis reference
Slide 57
Slide 57 text
Image analysis working modes
setBackpressureStrategy(ImageAnalysis.STRATEGY_BLOCK_PRODUCER)
● The executor receives sequentially the frames
● We could use getImageQueueDepth() to return the no of images available in
the pipeline, including the image currently analysed
● If analyze() takes longer than the latency of a single frame at the current
framerate => new frames are blocked until the method returns
setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
● The default strategy
● The executor receives last available frame
● If analyze() takes longer than the latency of a single frame at the current
framerate => some frames might be skipped and the method will get the last
frame available in the camera pipeline
blocking mode
non-blocking mode
Slide 58
Slide 58 text
ImageAnalysis.Builder methods
setTargetAspectRatio(int aspectRatio)
● aspectRatio could be RATIO_16_9 or RATIO_4_3
● Default = RATIO_4_3
setTargetName(String targetName)
● Debug purpose
● Default = class canonical name and random UUID
setTargetResolution(Size resolution)
● Set the resolution of the intended target
● Default = 640x480
setTargetRotation(int rotation)
● Set the rotation of the intended target
● Default = Display.getRotation()
Slide 59
Slide 59 text
class PurpleColorAnalyser() : ImageAnalysis.Analyzer {
override fun analyze(image: ImageProxy) {
TODO("Not yet implemented")
}
}
Step 2: define a custom
analyser
Slide 60
Slide 60 text
private var lastAnalyzedTimestamp = 0L
override fun analyze(image: ImageProxy) {
val timestamp = System.currentTimeMillis()
val oneSecond = TimeUnit.SECONDS.toMillis(1)
if (elapsedOneSecond(timestamp, oneSecond)) {
val buffer = image.planes[0].buffer
val data = buffer.toByteArray()
val pixels = data.map { it.toInt() and 0x9370DB }
val averagePurplePixels = pixels.average()
lastAnalyzedTimestamp = timestamp
}
image.close()
}
Step 3: implement
analyze() method
Slide 61
Slide 61 text
imageAnalysis.setAnalyzer(executor, PurpleColorAnalyser())
Step 4: set the custom
analyser
Slide 62
Slide 62 text
// bind the image analysis use case to camera
camera = cameraProvider.bindToLifecycle(
this as LifecycleOwner,
cameraSelector,
imageAnalysis,
preview
)
Step 5: update the call
to bind lifecycle
Slide 63
Slide 63 text
val imageAnalysis = ImageAnalysis.Builder()
.setBackpressureStrategy(nonBlocking)
.build()
val cameraProviderFuture = ProcessCameraProvider.getInstance(this)
val cameraProvider: ProcessCameraProvider = cameraProviderFuture.get()
val executor = ContextCompat.getMainExecutor(this)
imageAnalysis.setAnalyzer(executor, PurpleColorAnalyser())
camera = cameraProvider.bindToLifecycle(
this as LifecycleOwner,
cameraSelector,
imageAnalysis,
preview
)
Slide 64
Slide 64 text
No content
Slide 65
Slide 65 text
Image format
CameraX produces images in YUV_420_888 format.
Slide 66
Slide 66 text
YUV color encoding
This scheme assigns both
brightness and color
values to each pixel.
Slide 67
Slide 67 text
RGB vs YUV
RGB => R = red G = green B = blue
YUV => Y = Luminance U = Chrominance of blue V = Chrominance of red
YUV = YCbCr
● Luminance = refers to the brightness of the pixel = Y (grayscale image)
● Chrominance = refers to the color = UV
Slide 68
Slide 68 text
RGB to YUV
● Y = 0.299R + 0.587G + 0.114B
● U = 0.492(B - Y) => blueness of the pixel
● V = 0.877(R - Y) => redness of the pixel
Slide 69
Slide 69 text
YUV to RGB
● R = 1.164 * Y + 1.596 * V
● G = 1.164 * Y - 0.392 * U - 0.813 * V
● B = 1.164 * Y + 2.017 * U
Slide 70
Slide 70 text
No content
Slide 71
Slide 71 text
Human eye
sees well
sees not
so well
Chroma
Subsampling
Slide 72
Slide 72 text
4:4:4
4:2:2
4:2:0
Slide 73
Slide 73 text
No content
Slide 74
Slide 74 text
4:4:4
Slide 75
Slide 75 text
4:4:4
Slide 76
Slide 76 text
4:4:4
Slide 77
Slide 77 text
4:4:4
Slide 78
Slide 78 text
4:4:4
Slide 79
Slide 79 text
4:4:4
Slide 80
Slide 80 text
4:2:2
Slide 81
Slide 81 text
4:2:2
Slide 82
Slide 82 text
4:2:2
Slide 83
Slide 83 text
4:2:2
Slide 84
Slide 84 text
4:2:2
Slide 85
Slide 85 text
4:2:2
Slide 86
Slide 86 text
4:2:0
Slide 87
Slide 87 text
4:2:0
Slide 88
Slide 88 text
4:2:0
Slide 89
Slide 89 text
4:2:0
Slide 90
Slide 90 text
4:2:0
Slide 91
Slide 91 text
Chroma Subsampling
4:4:4
4:2:2
4:2:0
Luma Chroma Luma + Chroma
Slide 92
Slide 92 text
Data savings
50%
Slide 93
Slide 93 text
Camera Controls
cancelFocusAndMetering()
● Cancels current FocusMeteringAction and clears
AF/AE/AWB regions
● automatic focus (AF), automatic exposure (AE), and
automatic white-balance (AWB)
enableTorch(torch: Boolean) ● Enable the torch or disable the torch.
setLinearZoom(@FloatRange(0.0, 1.0)
linearZoom: Float)
● Sets current zoom by a linear zoom value ranging from
0f to 1f.
setZoomRatio(ratio: Float) ● Sets current zoom by ratio.
startFocusAndMetering(@NonNull
action: FocusMeteringAction)
● Starts a focus and metering action configured by the
FocusMeteringAction.
Slide 94
Slide 94 text
val cameraControl = camera.cameraControl
val cameraInfo = camera.cameraInfo
cameraInfo.torchState.observe(this, Observer { state ->
if (state == TorchState.ON) {
// state on
} else {
// state off
}
})
Slide 95
Slide 95 text
No content
Slide 96
Slide 96 text
CameraView
CameraView
FrameLayout
ViewGroup
● A view that displays a preview of the camera with methods like:
○ takePicture(ImageCapture.OutputFileOptions,
Executor, OnImageSavedCallback)
○ startRecording(File, Executor,
OnVideoSavedCallback)
○ stopRecording()
● Must be opened/closed, since it consumes a high amount of power,
and these actions can be handled by using bindToLifecycle
● CameraCapture is used to setup de capture mode
○ IMAGE
○ MIXED
○ VIDEO
Slide 97
Slide 97 text
CameraX lab-tested devices
Slide 98
Slide 98 text
Extensions
04
Dedicated API for optional
effects like HDR, portrait,
night-mode
Slide 99
Slide 99 text
Extensions architecture
Image source: https://developer.android.com/training/camerax/vendor-extensions
Slide 100
Slide 100 text
ImageCaptureExtender
+enableExtension(cameraSelector)
+isExtensionAvailable(cameraSelector)
BeautyImageCaptureExtender
NightImageCaptureExtender
AutoImageCaptureExtender
BokehImageCaptureExtender
HdrImageCaptureExtender
Slide 101
Slide 101 text
What is bokeh?
The bokeh effect is produced when
the foreground and/or background is
intentionally blurred around a
subject.
Bokeh means "blur" in Japanese.
Slide 102
Slide 102 text
Extensions - implementation steps
Step 1
Create an
Extender
object
Step 2
Enable the
extension
Slide 103
Slide 103 text
val builder = ImageCapture.Builder()
val beautyExtender =
BeautyImageCaptureExtender.create(builder)
Step 1: create an
Extender object
Slide 104
Slide 104 text
if (beautyExtender.isExtensionAvailable(cameraSelector)) {
beautyExtender.enableExtension(cameraSelector)
}
Step 2: enable the
extension
Slide 105
Slide 105 text
CameraX recap
Backward
compatible
with L+
devices
Consistent
behavior
across
devices
Easy
to use
API
Lifecycle
awareness
Use-case
driven
approach
Slide 106
Slide 106 text
Learn more...
● Official documentation
○ CameraX overview
● Medium Articles @androiddevelopers
○ Core principles behind CameraX Jetpack Library - Android Developers
○ Android’s CameraX Jetpack Library is now in Beta!
● Podcast
○ https://androidbackstage.blogspot.com/2019/06/episode-116-camerax.html
● Check latest updates about CameraX
○ https://developer.android.com/jetpack/androidx/releases/camera
● Lab-tested devices
○ https://developer.android.com/training/camerax/devices
Slide 107
Slide 107 text
Thanks!
magdamiu.com
@magdamiu