A high-level introduction to Android's Camera 2 API examining the underlying camera subsystem and hardware extraction layer changes that founded a complete re-work of the camera API.
• Still: single image • Video: continuous image capture • Preview: low-resolution video • Difficult to… • Implement new modes (Burst: take X pictures ASAP) • Manually control capture and processing 6
Based on upgrade of camera subsystem/HAL • Deprecates Camera 1 • More subsystem control, more features • Higher performance • API more efficient and maintainable 7
request system • Camera subsystem pipeline • API • Completely new system: android.hardware.camera2 • More functionality, more metadata: more verbose • Low-level requests, advanced terminology 8
single frame requests single frame requests CAMERA 1 CAMERA 2 Burst Repeating Burst continual execution of X requests execution of X requests … … x x x … Unimplementable • 3 operation modes → single frame capture model
requests output directly to N surfaces still request call back with byte array (bitmap) write to file display on surface frame request SurfaceView TextureView Allocation RenderScript ImageReader N pre-configured surfaces developer MediaCodec MediaRecorder
Request configuration, result configuration ! ! • Important for advanced features • HDR + exposure bracketing: require accurate feedback • RAW support: sensor info 13 Result Capture Settings Request Statistics
Features even more dependent on hardware/specific device 15 WANT NEED Google Play filtering <uses-feature /> minimumSdk Runtime checking Build.VERSION PackageManager CameraParameters (v1) CameraCharacteristics (v2) STONES PRINCIPLE Separate critical from optional features
a certain set of features and meet performance requirements • LIMITED: May have some or none of that set guaranteed by FULL; have to ask • LEGACY: Guaranteed NOT to support certain advanced features • Relaxed performance constraints • Basically a wrapper for < HAL3 cameras running Camera 2. • Explanation and requirements: CameraCharacteristics#INFO_SUPPORTED_HARDWARE_LEVEL 16
if either front or back camera is acceptable • external (API 20) • capability.manual_post_processing (API 21) • capability.manual_sensor (API 21) • level.full (API 21) 18
system service; information/access for cameras • CameraCharacteristics (CameraInfo + CameraParameters): description of camera device • CameraDevice (Camera): create capture session + capture requests; • CameraCaptureSession • configures set of output Surfaces for CameraDevice • gateway for requests 20
connected camera IDs. 2. Open camera. 3. Configure target outputs for potential requests. 4. Open a camera capture session. 5. Create a request and set configuration. 6. Submit request through session. 7. Receive resulting image data and metadata.
TextureView to layout Implement SurfaceHolder.Callback/ TextureView.SurfaceTextureListener Get camera info Open a camera Get list of camera IDs wait for surface to be created Select preview size from allowed values Adjust Surface size to fit selected preview Create camera capture session wait for camera to open wait for session to configure
CAMERA 2 Send AF lock request to CameraCaptureSession wait for AF results Create CaptureRequest for auto- exposure pre-capture (trigger) Create CaptureRequest for still capture Re-set configuration of preview request to unlock focus wait for capture to complete CAMERA 1 Camera#takePicture Camera#autoFocus wait for picture callbacks wait for AE results Send still request to CameraCaptureSession Send pre-capture request to CameraCaptureSession Send repeating request to CameraCaptureSession re-start preview is focus locked? is exposure converged? Create CaptureRequest for auto-focus (trigger)
release immediately • Same as in Camera 1 • Monopolizing camera -> crash other apps • onResume/onPause, switching cameras • Surface dance: the Setup + Size Samba • Surface must be setup before sending to CameraCaptureSession • Size must be in set allowed by CameraDevice • SCALER_STREAM_CONFIGURATION_MAP 27
• Focus on asynchronous work • Methods accept Handler, run callbacks on that thread • Synchronization of fields • Locking mechanism to coordinate camera open/ close so that app does not exit in middles 28
Smartphone Camera Trends, A Presentation by Brian Klug: http://www.anandtech.com/show/6777/ understanding-camera-optics-smartphone-camera-trends • 2013 SIGGRAPH Camera BoF: https://www.khronos.org/assets/uploads/ developers/library/2013-siggraph-camera-bof/Camera- BOF_SIGGRAPH-2013.pdf • All About Symbian | EDoF versus Auto-focus: Understanding the compromises involved: http://www.allaboutsymbian.com/features/item/ 12789_edof_versus_auto-focus_underst.php 30
microprocessor for processing sensor data: demosaicing, 3A, noise reduction, corrections • demosaicing (or debayering): digital image process to reconstruct full image from image sensor color samples (Bayer filter) • 3A: auto-focus, auto-exposure, auto-white- balance 32