Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Android Camera 2 API

Android Camera 2 API

A high-level introduction to Android's Camera 2 API examining the underlying camera subsystem and hardware extraction layer changes that founded a complete re-work of the camera API.

Huyen Tue Dao

August 27, 2015
Tweet

More Decks by Huyen Tue Dao

Other Decks in Technology

Transcript

  1. HOLD ONTO YOUR BUTTS… HOW I THOUGHT THE FUTURE OF

    CAMERA WOULD BE… HOW IT ACTUALLY WAS…
  2. ANDROID CAMERA 2 API • Camera 1 vs Camera 2

    • Camera 2 Subsystem • Getting started • Steps to Capture an Image • Caveats 3
  3. CAMERA 1 VS CAMERA 2 4 Straightforward (relatively) Faster to

    start Limited features Steeper learning curve More control, more features High performance Easier to mess up CAMERA 1: POINT AND SHOOT CAMERA 2: DSLR
  4. CAMERA SUBSYSTEM, VERSION 1: CAMERA COMPONENTS: ANDROID.COM black box, “one-way

    stream” preview request, still request, video request
  5. CAMERA 1 • 3 modes: preview, still capture, video capture

    • Still: single image • Video: continuous image capture • Preview: low-resolution video • Difficult to… • Implement new modes (Burst: take X pictures ASAP) • Manually control capture and processing 6
  6. CAMERA 2 • Camera 2: re-work not just update •

    Based on upgrade of camera subsystem/HAL • Deprecates Camera 1 • More subsystem control, more features • Higher performance • API more efficient and maintainable 7
  7. CAMERA 2 • Hardware Abstraction Layer v3 • New frame-based

    request system • Camera subsystem pipeline • API • Completely new system: android.hardware.camera2 • More functionality, more metadata: more verbose • Low-level requests, advanced terminology 8
  8. SUBSYSTEM/HAL3 Preview Still Video Repeating Capture Capture Repeating Capture continual

    single frame requests single frame requests CAMERA 1 CAMERA 2 Burst Repeating Burst continual execution of X requests execution of X requests … … x x x … Unimplementable • 3 operation modes → single frame capture model
  9. SUBSYSTEM/HAL3 CAMERA 1 CAMERA 2 • Bitmap callbacks → frame

    requests output directly to N surfaces still request call back with byte array (bitmap) write to file display on surface frame request SurfaceView TextureView Allocation RenderScript ImageReader N pre-configured surfaces developer MediaCodec MediaRecorder
  10. HAL3, PIPELINE “CAMERA HAL SUBSYSTEM PIPELINE MODEL” ANDROID.COM 3 block

    modes: OFF FAST HIGH_QUALITY ! blocks cannot be OFF JPEG processor only one capture at a time
  11. CAMERA 2: METADATA • Camera characteristics: lens, sensor info •

    Request configuration, result configuration ! ! • Important for advanced features • HDR + exposure bracketing: require accurate feedback • RAW support: sensor info 13 Result Capture Settings Request Statistics
  12. GETTING STARTED: STONES PRINCIPLE • Features dependent on API •

    Features even more dependent on hardware/specific device 15 WANT NEED Google Play filtering <uses-feature /> minimumSdk Runtime checking Build.VERSION PackageManager CameraParameters (v1) CameraCharacteristics (v2) STONES PRINCIPLE Separate critical from optional features
  13. GETTING STARTED: SUPPORTED HARDWARE LEVEL • FULL: Guaranteed to have

    a certain set of features and meet performance requirements • LIMITED: May have some or none of that set guaranteed by FULL; have to ask • LEGACY: Guaranteed NOT to support certain advanced features • Relaxed performance constraints • Basically a wrapper for < HAL3 cameras running Camera 2. • Explanation and requirements: CameraCharacteristics#INFO_SUPPORTED_HARDWARE_LEVEL 16
  14. GETTING STARTED: MANIFEST • Permission: android.permission.camera • Just requesting permission

    for camera implies application requires: • android.hardware.camera (default/back) • android.hardware.camera.autofocus • android.hardware.camera.flash • android.hardware.camera.front • Highly recommended to explicitly specify <uses-feature android:required=“false” /> for optional device features. 17
  15. GETTING STARTED: MANIFEST • android.hardware.camera… • any (API 17); preferred

    if either front or back camera is acceptable • external (API 20) • capability.manual_post_processing (API 21) • capability.manual_sensor (API 21) • level.full (API 21) 18
  16. GETTING STARTED: LEARNING BY EXAMPLE • Android Camera2Basic Sample: •

    Find camera, open cameras, access characteristics, start preview, take picture • https://github.com/googlesamples/android-Camera2Basic • Android Camera2Raw Sample • Camera2Basic extended to RAW, multiple output buffers, handle multiple requests to queue • Does more/better handling of concurrency • https://github.com/googlesamples/android-Camera2Raw 19
  17. GETTING STARTED: CAMERA 2 CLASSES • CameraManager (static Camera methods):

    system service; information/access for cameras • CameraCharacteristics (CameraInfo + CameraParameters): description of camera device • CameraDevice (Camera): create capture session + capture requests; • CameraCaptureSession • configures set of output Surfaces for CameraDevice • gateway for requests 20
  18. GETTING STARTED: CAMERA 2 CLASSES • CameraMetadata: camera/capture characteristics and

    controls • CameraRequest(Builder) • tied to particular CameraDevice • targets pre-configured Surface in CameraCaptureSession • CaptureResult/TotalCaptureResult: • request configuration • hardware, processing, controls, and output buffers 21
  19. GETTING STARTED: BASIC API STEPS 1. Get a list of

    connected camera IDs. 2. Open camera. 3. Configure target outputs for potential requests. 4. Open a camera capture session. 5. Create a request and set configuration. 6. Submit request through session. 7. Receive resulting image data and metadata.
  20. VERBOSITY + COMPLEXITY: CAMERA 1 VS CAMERA 2 SETUP *async,

    †background Camera.getNumberOfCameras Camera.open Camera.getCameraInfo Camera#getParameters CameraManager#getCameraIdList CameraManager#openCamera*† CameraManager#getCameraCharacteristics OPEN CAMERA CameraDevice#createCaptureSession*† CameraDevice#createCaptureRequest CaptureRequest.Builder#addTarget CaptureRequest.Builder#set (2x) CaptureRequest.Builder#build CameraCaptureSession#setRepeatingRequest*† Camera#setPreviewDisplay(SurfaceHolder) Camera#startPreview() START PREVIEW PREVIEW SETUP CameraParameters#getSupportedPreviewSizes CameraParameters#setPreviewSize Camera#DisplayOrientation Camera#setParameters #onSurfaceTextureAvailable CameraCharacteristics#get StreamConfigurationMap#get #onSurfaceTextureAvailable TextureView#setAspectRatio TextureView#setTransform CAMERA 1 CAMERA 2
  21. SIMULTANEOUSLY SETTING STUFF UP CAMERA DEVICE PREVIEW SURFACE Add SurfaceView/

    TextureView to layout Implement SurfaceHolder.Callback/ TextureView.SurfaceTextureListener Get camera info Open a camera Get list of camera IDs wait for surface to be created Select preview size from allowed values Adjust Surface size to fit selected preview Create camera capture session wait for camera to open wait for session to configure
  22. wait for auto-focus callback TAKING A PICTURE: AUTO-FOCUS, AUTO-EXPOSURE, CAPTURE

    CAMERA 2 Send AF lock request to CameraCaptureSession wait for AF results Create CaptureRequest for auto- exposure pre-capture (trigger) Create CaptureRequest for still capture Re-set configuration of preview request to unlock focus wait for capture to complete CAMERA 1 Camera#takePicture Camera#autoFocus wait for picture callbacks wait for AE results Send still request to CameraCaptureSession Send pre-capture request to CameraCaptureSession Send repeating request to CameraCaptureSession re-start preview is focus locked? is exposure converged? Create CaptureRequest for auto-focus (trigger)
  23. CAMERA 2 CAVEATS: BASICS • Only grab CameraDevice when needed;

    release immediately • Same as in Camera 1 • Monopolizing camera -> crash other apps • onResume/onPause, switching cameras • Surface dance: the Setup + Size Samba • Surface must be setup before sending to CameraCaptureSession • Size must be in set allowed by CameraDevice • SCALER_STREAM_CONFIGURATION_MAP 27
  24. CAMERA 2 CAVEATS: CONCURRENCY • API/Subsystem/HAL3 • Performance; non-blocking requests

    • Focus on asynchronous work • Methods accept Handler, run callbacks on that thread • Synchronization of fields • Locking mechanism to coordinate camera open/ close so that app does not exit in middles 28
  25. THANKS FOR COMING! Huyen Tue Dao Randomly Typing ! @queencodemonkey

    ! [email protected] randomlytyping.com ! speakerdeck.com/randomlytyping
  26. APPENDIX A: REFERENCES • AnandTech | Understanding Camera Optics &

    Smartphone Camera Trends, A Presentation by Brian Klug: http://www.anandtech.com/show/6777/ understanding-camera-optics-smartphone-camera-trends • 2013 SIGGRAPH Camera BoF: https://www.khronos.org/assets/uploads/ developers/library/2013-siggraph-camera-bof/Camera- BOF_SIGGRAPH-2013.pdf • All About Symbian | EDoF versus Auto-focus: Understanding the compromises involved: http://www.allaboutsymbian.com/features/item/ 12789_edof_versus_auto-focus_underst.php 30
  27. APPENDIX A: REFERENCES • Graphics Architecture: https://source.android.com/devices/graphics/ architecture.html • Android

    | Devices | Camera: http://source.android.com/devices/camera/ • Exposing the Android Camera Stack (Aptina Imaging) • (Pre-reading) Android | Graphics Architecture: https://source.android.com/ devices/graphics/architecture.html • Slides: http://www.embedded-vision.com/academy/ Exposing_the_Android_Camera_Stack.pdf • Video: https://youtu.be/dCfHh2viI-c • Camera2 API on MWC 2015 devices: Galaxy S6, HTC One M9 and more Lollipop devices: http://spectrastudy.com/camera2-api-on-mwc-2015-devices/ 31
  28. APPENDIX B: GLOSSARY • ISP - Image Signal Processor, specialized

    microprocessor for processing sensor data: demosaicing, 3A, noise reduction, corrections • demosaicing (or debayering): digital image process to reconstruct full image from image sensor color samples (Bayer filter) • 3A: auto-focus, auto-exposure, auto-white- balance 32