HELLO THERE, CAMERA 2 API PRESENTED BY HUYEN TUE DAO (for science) 

HOLD ONTO YOUR BUTTS… HOW I THOUGHT THE FUTURE OF CAMERA WOULD BE… HOW IT ACTUALLY WAS… 

ANDROID CAMERA 2 API • Camera 1 vs Camera 2 • Camera 2 Subsystem • Getting started • Steps to Capture an Image • Caveats 3 

CAMERA 1 VS CAMERA 2 4 Straightforward (relatively) Faster to start Limited features Steeper learning curve More control, more features High performance Easier to mess up CAMERA 1: POINT AND SHOOT CAMERA 2: DSLR 

CAMERA SUBSYSTEM, VERSION 1: CAMERA COMPONENTS: ANDROID.COM black box, “one-way stream” preview request, still request, video request 

CAMERA 1 • 3 modes: preview, still capture, video capture • Still: single image • Video: continuous image capture • Preview: low-resolution video • Difficult to… • Implement new modes (Burst: take X pictures ASAP) • Manually control capture and processing 6 

CAMERA 2 • Camera 2: re-work not just update • Based on upgrade of camera subsystem/HAL • Deprecates Camera 1 • More subsystem control, more features • Higher performance • API more efficient and maintainable 7 

CAMERA 2 • Hardware Abstraction Layer v3 • New frame-based request system • Camera subsystem pipeline • API • Completely new system: android.hardware.camera2 • More functionality, more metadata: more verbose • Low-level requests, advanced terminology 8 

CAMERA, ARCHITECTURE ANDROID.COM HAL Android API System service Native camera JNI (Camera 1) 

SUBSYSTEM/HAL3 Preview Still Video Repeating Capture Capture Repeating Capture continual single frame requests single frame requests CAMERA 1 CAMERA 2 Burst Repeating Burst continual execution of X requests execution of X requests … … x x x … Unimplementable • 3 operation modes → single frame capture model 

SUBSYSTEM/HAL3 CAMERA 1 CAMERA 2 • Bitmap callbacks → frame requests output directly to N surfaces still request call back with byte array (bitmap) write to file display on surface frame request SurfaceView TextureView Allocation RenderScript ImageReader N pre-configured surfaces developer MediaCodec MediaRecorder 

HAL3, PIPELINE “CAMERA HAL SUBSYSTEM PIPELINE MODEL” ANDROID.COM 3 block modes: OFF FAST HIGH_QUALITY ! blocks cannot be OFF JPEG processor only one capture at a time 

CAMERA 2: METADATA • Camera characteristics: lens, sensor info • Request configuration, result configuration ! ! • Important for advanced features • HDR + exposure bracketing: require accurate feedback • RAW support: sensor info 13 Result Capture Settings Request Statistics 

CAMERA HAL V3, “CAMERA CORE OPERATION MODEL” ANDROID.COM 

GETTING STARTED: STONES PRINCIPLE • Features dependent on API • Features even more dependent on hardware/specific device 15 WANT NEED Google Play filtering minimumSdk Runtime checking Build.VERSION PackageManager CameraParameters (v1) CameraCharacteristics (v2) STONES PRINCIPLE Separate critical from optional features 

GETTING STARTED: SUPPORTED HARDWARE LEVEL • FULL: Guaranteed to have a certain set of features and meet performance requirements • LIMITED: May have some or none of that set guaranteed by FULL; have to ask • LEGACY: Guaranteed NOT to support certain advanced features • Relaxed performance constraints • Basically a wrapper for < HAL3 cameras running Camera 2. • Explanation and requirements: CameraCharacteristics#INFO_SUPPORTED_HARDWARE_LEVEL 16 

GETTING STARTED: MANIFEST • Permission: android.permission.camera • Just requesting permission for camera implies application requires: • android.hardware.camera (default/back) • android.hardware.camera.autofocus • android.hardware.camera.flash • android.hardware.camera.front • Highly recommended to explicitly specify for optional device features. 17 

GETTING STARTED: MANIFEST • android.hardware.camera… • any (API 17); preferred if either front or back camera is acceptable • external (API 20) • capability.manual_post_processing (API 21) • capability.manual_sensor (API 21) • level.full (API 21) 18 

GETTING STARTED: LEARNING BY EXAMPLE • Android Camera2Basic Sample: • Find camera, open cameras, access characteristics, start preview, take picture • https://github.com/googlesamples/android-Camera2Basic • Android Camera2Raw Sample • Camera2Basic extended to RAW, multiple output buffers, handle multiple requests to queue • Does more/better handling of concurrency • https://github.com/googlesamples/android-Camera2Raw 19 

GETTING STARTED: CAMERA 2 CLASSES • CameraManager (static Camera methods): system service; information/access for cameras • CameraCharacteristics (CameraInfo + CameraParameters): description of camera device • CameraDevice (Camera): create capture session + capture requests; • CameraCaptureSession • configures set of output Surfaces for CameraDevice • gateway for requests 20 

GETTING STARTED: CAMERA 2 CLASSES • CameraMetadata: camera/capture characteristics and controls • CameraRequest(Builder) • tied to particular CameraDevice • targets pre-configured Surface in CameraCaptureSession • CaptureResult/TotalCaptureResult: • request configuration • hardware, processing, controls, and output buffers 21 

GETTING STARTED: BASIC API STEPS 1. Get a list of connected camera IDs. 2. Open camera. 3. Configure target outputs for potential requests. 4. Open a camera capture session. 5. Create a request and set configuration. 6. Submit request through session. 7. Receive resulting image data and metadata. 

HAL SUBSYSTEM, CAMERA MODEL ANDROID.COM 

VERBOSITY + COMPLEXITY: CAMERA 1 VS CAMERA 2 SETUP *async, †background Camera.getNumberOfCameras Camera.open Camera.getCameraInfo Camera#getParameters CameraManager#getCameraIdList CameraManager#openCamera*† CameraManager#getCameraCharacteristics OPEN CAMERA CameraDevice#createCaptureSession*† CameraDevice#createCaptureRequest CaptureRequest.Builder#addTarget CaptureRequest.Builder#set (2x) CaptureRequest.Builder#build CameraCaptureSession#setRepeatingRequest*† Camera#setPreviewDisplay(SurfaceHolder) Camera#startPreview() START PREVIEW PREVIEW SETUP CameraParameters#getSupportedPreviewSizes CameraParameters#setPreviewSize Camera#DisplayOrientation Camera#setParameters #onSurfaceTextureAvailable CameraCharacteristics#get StreamConfigurationMap#get #onSurfaceTextureAvailable TextureView#setAspectRatio TextureView#setTransform CAMERA 1 CAMERA 2 

SIMULTANEOUSLY SETTING STUFF UP CAMERA DEVICE PREVIEW SURFACE Add SurfaceView/ TextureView to layout Implement SurfaceHolder.Callback/ TextureView.SurfaceTextureListener Get camera info Open a camera Get list of camera IDs wait for surface to be created Select preview size from allowed values Adjust Surface size to fit selected preview Create camera capture session wait for camera to open wait for session to configure 

wait for auto-focus callback TAKING A PICTURE: AUTO-FOCUS, AUTO-EXPOSURE, CAPTURE CAMERA 2 Send AF lock request to CameraCaptureSession wait for AF results Create CaptureRequest for auto- exposure pre-capture (trigger) Create CaptureRequest for still capture Re-set configuration of preview request to unlock focus wait for capture to complete CAMERA 1 Camera#takePicture Camera#autoFocus wait for picture callbacks wait for AE results Send still request to CameraCaptureSession Send pre-capture request to CameraCaptureSession Send repeating request to CameraCaptureSession re-start preview is focus locked? is exposure converged? Create CaptureRequest for auto-focus (trigger) 

CAMERA 2 CAVEATS: BASICS • Only grab CameraDevice when needed; release immediately • Same as in Camera 1 • Monopolizing camera -> crash other apps • onResume/onPause, switching cameras • Surface dance: the Setup + Size Samba • Surface must be setup before sending to CameraCaptureSession • Size must be in set allowed by CameraDevice • SCALER_STREAM_CONFIGURATION_MAP 27 

CAMERA 2 CAVEATS: CONCURRENCY • API/Subsystem/HAL3 • Performance; non-blocking requests • Focus on asynchronous work • Methods accept Handler, run callbacks on that thread • Synchronization of fields • Locking mechanism to coordinate camera open/ close so that app does not exit in middles 28 

THANKS FOR COMING! Huyen Tue Dao Randomly Typing ! @queencodemonkey ! [email protected] randomlytyping.com ! speakerdeck.com/randomlytyping 

APPENDIX A: REFERENCES • AnandTech | Understanding Camera Optics & Smartphone Camera Trends, A Presentation by Brian Klug: http://www.anandtech.com/show/6777/ understanding-camera-optics-smartphone-camera-trends • 2013 SIGGRAPH Camera BoF: https://www.khronos.org/assets/uploads/ developers/library/2013-siggraph-camera-bof/Camera- BOF_SIGGRAPH-2013.pdf • All About Symbian | EDoF versus Auto-focus: Understanding the compromises involved: http://www.allaboutsymbian.com/features/item/ 12789_edof_versus_auto-focus_underst.php 30 

APPENDIX A: REFERENCES • Graphics Architecture: https://source.android.com/devices/graphics/ architecture.html • Android | Devices | Camera: http://source.android.com/devices/camera/ • Exposing the Android Camera Stack (Aptina Imaging) • (Pre-reading) Android | Graphics Architecture: https://source.android.com/ devices/graphics/architecture.html • Slides: http://www.embedded-vision.com/academy/ Exposing_the_Android_Camera_Stack.pdf • Video: https://youtu.be/dCfHh2viI-c • Camera2 API on MWC 2015 devices: Galaxy S6, HTC One M9 and more Lollipop devices: http://spectrastudy.com/camera2-api-on-mwc-2015-devices/ 31 

APPENDIX B: GLOSSARY • ISP - Image Signal Processor, specialized microprocessor for processing sensor data: demosaicing, 3A, noise reduction, corrections • demosaicing (or debayering): digital image process to reconstruct full image from image sensor color samples (Bayer filter) • 3A: auto-focus, auto-exposure, auto-white- balance 32