How to use multi-camera API in Android

Transcript

1. How to use multi-camera API in Android shibuya.apk #38 -

   report from KotlinConf 2019 (2020/01/17) @kurun_pan

2. Self-introduction @kurun_pan @Kurun-pan Software Engineer @Sony Linux / Android /εϓϥτΡʔϯΛѪ͢ΔΤϯδχΞ

   Kurun / ͘ΔΜ

3. Support multi-camera API from Android 9 https://source.android.com/devices/camera/multi-camera

4. Multi-camera use case • Ұ൪ଟ͍Ϣʔεέʔε͸ɺϢʔβͷζʔϜૢ࡞ʹ࿈ಈͨ͠ΧϝϥϨϯζʢྫ͑͹ɺ޿֯/๬ԕϨ ϯζ㱻ඪ४Ϩϯζʣͷ੾Γସ͑ • ͦͷଞɺεςϨΦΧϝϥͰDepth (Ԟߦ͖) ΋ࢉग़Ͱ͖ͨΓ΋

   ޿֯Χϝϥ ඪ४Χϝϥ

5. Logical and physical camera from Android 9 • CameraManagerܦ༝ͰऔಘՄೳͳΧϝϥ(ID)ʹlogical cameraͱphysical

   cameraͷ2छྨ͕ଘࡏ • ྫ͑͹ɺlogical-cameraΛར༻ͯ͠ɺಉ࣌ʹෳ਺ΧϝϥͷετϦʔϜऔಘͳͲ͕Մೳ ʙ
   "OESPJE
    "OESPJE
   
   ʙ %FWJDF
   WFOEPST "QQ
   %FWFMPQFST ͋͘·Ͱ΋આ໌༻ͷΠϝʔδߏ੒Ͱ͢ MPHJDBMDBNFSB #BDL
   DBNFSB
    #BDL
   DBNFSB
    #BDL
   DBNFSB MPHJDBMDBNFSB 'SPOU
   DBNFSB
    'SPOU
   DBNFSB
    'SPOU
   DBNFSB #BDL
   DBNFSB
    #BDL
   DBNFSB
    #BDL
   DBNFSB MPHJDBMDBNFSB 'SPOU
   DBNFSB
    'SPOU
   DBNFSB
    %FWJDF
   WFOEPST "QQ
   %FWFMPQFST MPHJDBMDBNFSB #BDL
   DBNFSB
    #BDL
   DBNFSB
    $BNFSB
   *% $BNFSB
   *% QIZTJDBM
   DBNFSB MPHJDBM
   DBNFSB 'SPOU
   DBNFSB 'SPOU
   DBNFSB
    'SPOU
   DBNFSB
    $BNFSB
   *% $BNFSB
   *% $BNFSB
   *% $BNFSB
   *% $BNFSB
   *% $BNFSB
   *%

6. Multi-camera API (Camera2) Ͱ ෳ਺Χϝϥͷಉ੍࣌ޚ͢Δํ๏ʹ͍ͭͯ

7. Step 1. Get camera IDs CameraManager#getCameraIdListͰcamera idͷҰཡΛऔಘ ͨͩ͠ɺlogical-cameraʹଐ͢Δphysical-cameraͷID͸ؚ·Εͳ͍ val manager:

   CameraManager = getSystemService(Context.CAMERA_SERVICE) as CameraManager for (cameraId in manager.cameraIdList) { … } #BDL
   DBNFSB
    #BDL
   DBNFSB
    #BDL
   DBNFSB MPHJDBMDBNFSB 'SPOU
   DBNFSB
    'SPOU
   DBNFSB
    %FWJDF
   WFOEPST "QQ
   %FWFMPQFST MPHJDBMDBNFSB #BDL
   DBNFSB
    #BDL
   DBNFSB
    'SPOU
   DBNFSB 'SPOU
   DBNFSB
    'SPOU
   DBNFSB
    $BNFSB
   *% $BNFSB
   *% $BNFSB
   *% $BNFSB
   *% $BNFSB
   *% $BNFSB
   *% ӈਤͷ৔߹ɺDBNFSB*E-JTU͸ ͷΈ
   ˞௨ৗɺ*%͕σϑΥϧτͷഎ໘Χϝϥɺ
   *%͕લ໘Χϝϥ

8. Step 2. Check multi-camera is supported or not for (cameraId

   in manager.cameraIdList) { val characteristics = manager.getCameraCharacteristics(cameraId) // get supported capabilities val capabilities = characteristics.get( CameraCharacteristics.REQUEST_AVAILABLE_CAPABILITIES ) // logical-camera or not? val isLogicalCamera = capabilities!!.contains( CameraCharacteristics.REQUEST_AVAILABLE_CAPABILITIES_LOGICAL_MULTI_CAMERA ) if (isLogicalCamera) { val physicalCameraIds = characteristics.physicalCameraIds … } … } CameraManager#getCameraCharacteristicsͰlogical cameraͷαϙʔτ༗ແνΣοΫ ✓ REQUEST_AVAILABLE_CAPABILITIES_LOGICAL_MULTI_CAMERAͷνΣοΫ ✓ CameraCharacteristics#getPhysicalCameraIdsͰphysical camera idΛऔಘ #BDL
   DBNFSB
    #BDL
   DBNFSB
    #BDL
   DBNFSB %FWJDF
   WFOEPST "QQ
   %FWFMPQFST MPHJDBMDBNFSB #BDL
   DBNFSB
    #BDL
   DBNFSB
    $BNFSB
   *% $BNFSB
   *% $BNFSB
   *% ӈਤͷ৔߹ɺ QIZTJDBM$BNFSB*ET͸ 

9. Step 3. Open logical camera CameraManager#openCameraͰɺΧϝϥIDʹର৅ͷlogical camera idΛࢦఆ ※جຊతͳCamera2Ͱͷ΍Γํͱಉ͘͡ɺstateCallbackͰCameraDeviceΛऔಘ cameraManager.openCamera(logicalCameraId,

   stateCallback, backgroundHander) #BDL
   DBNFSB
    #BDL
   DBNFSB
    #BDL
   DBNFSB %FWJDF
   WFOEPST "QQ
   %FWFMPQFST MPHJDBMDBNFSB #BDL
   DBNFSB
    #BDL
   DBNFSB
    $BNFSB
   *% $BNFSB
   *% $BNFSB
   *%

10. Step 4. Create CaptureSession CaptureSessionͷ࡞੒ɹ※SessionConfiguration͸Multi-camera APIͱͯ͠API 28͔Β௥Ճ ✓ CameraDevice#createCaptureSessionͷҾ਺SessionConfigurationʹର৅ͷphysical camera

    id/TextureViewΛࢦఆ ✓ ͦͷޙ͸Camera2ͷ͓࡞๏ͱಉ༷ ✓ (ੜ੒ͨ͠CaptureSessionʹରͯ͠CaptureSession#setRepeatingRequestΛ࣮ߦͯ͠Ωϟϓνϟ։࢝) private fun createDualCamera(surfaces: List<Surface>) { val outputConfigs = surfaces.mapIndexed { index, surface -> val config = OutputConfiguration(surface) val physicalCameraId = physicalCameraIds.toList()[index] config.setPhysicalCameraId(physicalCameraId) config ^mapIndexed } val executor = Executors.newCachedThreadPool() val sessionConfig = SessionConfiguration( SessionConfiguration.SESSION_REGULAR, outputConfigs, executor, captureStateCallback ) cameraDevice?.createCaptureSession(sessionConfig) } #BDL
    DBNFSB
     #BDL
    DBNFSB
     #BDL
    DBNFSB %FWJDF
    WFOEPST "QQ
    %FWFMPQFST MPHJDBMDBNFSB #BDL
    DBNFSB
     #BDL
    DBNFSB
     $BNFSB
    *% $BNFSB
    *% $BNFSB
    *%

11. ࢖ͬͯΈͨײ૝ (·ͱΊ)

12. Points I noticed after using multi-camera API • ͍͔ͭ͘ͷAndroid 10ػछͰࢼͯ͠Έ͕ͨɺlogical-cameraඇαϙʔτػछ͕ଟͦ͏ͳؾ഑

    ✓ Pixel 3͸αϙʔτ ✓ Galaxy S10 Plus͸ඇαϙʔτ ✓ physical-camera΋શΧϝϥ෼ͷIDΛऔಘͰ͖ͳ͔ͬͨ • logical-cameraͷΈΛࢦఆͯ͠ར༻ͨ͠৔߹ͷڍಈ͕ෆ໌ ✓ Pixel3ͷ৔߹ɺ಺෦తʹ͸Ұͭͷݻఆphysical-cameraΛར༻͍ͯ͠Δ༷ࢠ ➡ ζʔϜʹԠͯ͡޿֯Ϩϯζͱඪ४Ϩϯζͷ੾Γସ͑ͳͲ΋Ұ੾ͳ͠ ➡ CaptureRequestͰઃఆՄೳͳԿ͔ͷઃఆʹ࿈ಈ͍ͯ͠Δʁ • logical-cameraʹؚ·ΕΔphysical-camera͸୯ҰͰͷར༻͕Մೳͩͬͨ

13. Camera Architecture in Android $BNFSB
    )BSEXBSF -JOVY
    ,FSOFM $BNFSB
    %FWJDF
    %SJWFS $BNFSB
    )"- "OESPJE
    'SBNFXPSL
    
    $BNFSB
    
    $BNFSB9

    
    "QQMJDBUJPO εϚʔτϑΥϯ 4P$
    ϕϯμʔʹґଘ "OESPJE
    04
    WFSTJPO
    "1*
    -FWFM ʹґଘ
    ࠓ࣌఺Ͱ͸$BNFSB9͸.VMUJDBNFSB
    "1*ඇαϙʔτ Ұൠతʹϋʔυ΢ΣΞܥ(Bluetooth BLEͳͲ)ͷػೳ͸ػछʹڧ͘ґଘͯ͠͠·͏ͨΊɺ ΞϓϦͰͷϚϧνΧϝϥػೳαϙʔτ͕೉͍͠

14. Thank you!☺