Upgrade to Pro — share decks privately, control downloads, hide ads and more …

AVFoundation, iOS 11 & Lumina

David Okun
November 07, 2017

AVFoundation, iOS 11 & Lumina

Come learn about what’s new in iOS 11 with AVFoundation, as David Okun walks us through new ways to capture still images, live photos, and depth maps via his open source framework, Lumina.

David Okun

November 07, 2017
Tweet

More Decks by David Okun

Other Decks in Programming

Transcript

  1. AVFoundation, iOS 11 & Lumina
    David Okun
    @dokun24

    View Slide

  2. WEIRD SWIFT ATX
    @dokun24

    View Slide

  3. WEIRD SWIFT ATX
    •Meets every Thursday
    @dokun24

    View Slide

  4. WEIRD SWIFT ATX
    •Meets every Thursday
    •6-10 pm
    @dokun24

    View Slide

  5. WEIRD SWIFT ATX
    •Meets every Thursday
    •6-10 pm
    •Been at BennU coffee so far but…
    @dokun24

    View Slide

  6. WEIRD SWIFT ATX
    •Meets every Thursday
    •6-10 pm
    •Been at BennU coffee so far but…
    •…next week is at Galvanize!!!
    @dokun24

    View Slide

  7. What We’ll Cover
    @dokun24

    View Slide

  8. What We’ll Cover
    • A history of AVFoundation’s input and output
    @dokun24

    View Slide

  9. What We’ll Cover
    • A history of AVFoundation’s input and output
    • What changed in iOS 10
    @dokun24

    View Slide

  10. What We’ll Cover
    • A history of AVFoundation’s input and output
    • What changed in iOS 10
    • New methods in iOS 11
    @dokun24

    View Slide

  11. What We’ll Cover
    • A history of AVFoundation’s input and output
    • What changed in iOS 10
    • New methods in iOS 11
    • Capturing Depth Data
    @dokun24

    View Slide

  12. What We’ll Cover
    • A history of AVFoundation’s input and output
    • What changed in iOS 10
    • New methods in iOS 11
    • Capturing Depth Data
    • Lumina
    @dokun24

    View Slide

  13. What We’ll Cover
    • A history of AVFoundation’s input and output
    • What changed in iOS 10
    • New methods in iOS 11
    • Capturing Depth Data
    • Lumina
    • Synchronizing Asynchronously Streaming Data
    @dokun24

    View Slide

  14. What We Won’t Cover
    @dokun24

    View Slide

  15. What We Won’t Cover
    • High Efficiency Image File Format (HEIF)
    @dokun24

    View Slide

  16. What We Won’t Cover
    • High Efficiency Image File Format (HEIF)
    • High Efficiency Video Coding (HEVC)
    @dokun24

    View Slide

  17. What We Won’t Cover
    • High Efficiency Image File Format (HEIF)
    • High Efficiency Video Coding (HEVC)
    • AVCaptureDepthDataOutput
    @dokun24

    View Slide

  18. Reference Material
    @dokun24

    View Slide

  19. Reference Material
    • WWDC 2016, Session 501
    @dokun24

    View Slide

  20. Reference Material
    • WWDC 2016, Session 501
    • WWDC 2017, Session 507
    @dokun24

    View Slide

  21. Reference Material
    • WWDC 2016, Session 501
    • WWDC 2017, Session 507
    • https://github.com/dokun1/lumina
    @dokun24

    View Slide

  22. @dokun24

    View Slide

  23. AVCaptureDevice
    @dokun24

    View Slide

  24. AVCaptureDevice
    AVCaptureDeviceInput
    @dokun24

    View Slide

  25. AVCaptureDevice
    AVCaptureDeviceInput
    AVCaptureSession
    @dokun24

    View Slide

  26. AVCaptureDevice
    AVCaptureDeviceInput
    AVCaptureConnection
    AVCaptureSession
    AVCapturePhotoOutput
    @dokun24

    View Slide

  27. AVCaptureDevice
    AVCaptureDeviceInput
    AVCaptureConnection
    AVCaptureConnection
    AVCaptureSession
    AVCapturePhotoOutput
    AVCaptureMovieFileOutput
    @dokun24

    View Slide

  28. AVCaptureDevice
    AVCaptureDeviceInput
    AVCaptureConnection
    AVCaptureConnection
    AVCaptureConnection
    AVCaptureSession
    AVCapturePhotoOutput
    AVCaptureMovieFileOutput
    AVCaptureMetadataOutput
    @dokun24

    View Slide

  29. AVCaptureDevice
    AVCaptureDeviceInput
    AVCaptureConnection
    AVCaptureConnection
    AVCaptureConnection
    AVCaptureConnection
    AVCaptureSession
    AVCapturePhotoOutput
    AVCaptureMovieFileOutput
    AVCaptureMetadataOutput
    AVCaptureVideoDataOutput
    @dokun24

    View Slide

  30. AVCaptureDevice
    AVCaptureDeviceInput
    AVCaptureConnection
    AVCaptureConnection
    AVCaptureConnection
    AVCaptureConnection
    AVCaptureSession
    AVCapturePhotoOutput
    AVCaptureMovieFileOutput
    AVCaptureMetadataOutput
    AVCaptureVideoDataOutput
    AVCaptureVideoPreviewLayer
    @dokun24

    View Slide

  31. Before iOS 10
    @dokun24

    View Slide

  32. Before iOS 10
    AVCaptureStillImageOutput
    @dokun24

    View Slide

  33. Before iOS 10
    AVCaptureStillImageOutput
    func captureStillImageAsynchronously(from connection:
    AVCaptureConnection, completionHandler handler: @escaping
    (CMSampleBuffer?, Error?) -> Swift.Void)
    @dokun24

    View Slide

  34. Since iOS 10
    @dokun24

    View Slide

  35. Since iOS 10
    AVCapturePhotoOutput
    @dokun24

    View Slide

  36. Since iOS 10
    AVCapturePhotoOutput
    var settings = AVCapturePhotoSettings()
    self.photoOutput.capturePhoto(with: settings, delegate: self)
    @dokun24

    View Slide

  37. Since iOS 10
    @dokun24

    View Slide

  38. Since iOS 10
    AVCapturePhotoSettings
    @dokun24

    View Slide

  39. Since iOS 10
    AVCapturePhotoSettings
    var uniqueID: Int64 // generated on init
    @dokun24

    View Slide

  40. Since iOS 10
    AVCapturePhotoSettings
    var uniqueID: Int64 // generated on init
    var flashMode: AVCaptureDevice.FlashMode
    @dokun24

    View Slide

  41. Since iOS 10
    AVCapturePhotoSettings
    var uniqueID: Int64 // generated on init
    var flashMode: AVCaptureDevice.FlashMode
    var isAutoStillImageStabilizationEnabled: Bool
    @dokun24

    View Slide

  42. Since iOS 10
    AVCapturePhotoSettings
    var uniqueID: Int64 // generated on init
    var flashMode: AVCaptureDevice.FlashMode
    var isAutoStillImageStabilizationEnabled: Bool
    var isHighResolutionPhotoEnabled: Bool
    @dokun24

    View Slide

  43. Since iOS 10
    AVCapturePhotoSettings
    var uniqueID: Int64 // generated on init
    var flashMode: AVCaptureDevice.FlashMode
    var isAutoStillImageStabilizationEnabled: Bool
    var isHighResolutionPhotoEnabled: Bool
    var isAutoDualCameraFusionEnabled: Bool // iOS 10.2+
    @dokun24

    View Slide

  44. Since iOS 10
    @dokun24

    View Slide

  45. Since iOS 10
    AVCapturePhotoOutputDelegate
    @dokun24

    View Slide

  46. Since iOS 10
    AVCapturePhotoOutputDelegate
    func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    @dokun24

    View Slide

  47. Since iOS 10
    AVCapturePhotoOutputDelegate
    func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    func photoOutput(_ output: AVCapturePhotoOutput, willCapturePhotoFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    @dokun24

    View Slide

  48. Since iOS 10
    AVCapturePhotoOutputDelegate
    func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    func photoOutput(_ output: AVCapturePhotoOutput, willCapturePhotoFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    func photoOutput(_ output: AVCapturePhotoOutput, didCapturePhotoFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    @dokun24

    View Slide

  49. Since iOS 10
    AVCapturePhotoOutputDelegate
    func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photoSampleBuffer:
    CMSampleBuffer?, previewPhoto previewPhotoSampleBuffer: CMSampleBuffer?, resolvedSettings:
    AVCaptureResolvedPhotoSettings, bracketSettings: AVCaptureBracketedStillImageSettings?, error: Error?)
    func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    func photoOutput(_ output: AVCapturePhotoOutput, willCapturePhotoFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    func photoOutput(_ output: AVCapturePhotoOutput, didCapturePhotoFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    @dokun24

    View Slide

  50. Since iOS 10
    AVCapturePhotoOutputDelegate
    func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photoSampleBuffer:
    CMSampleBuffer?, previewPhoto previewPhotoSampleBuffer: CMSampleBuffer?, resolvedSettings:
    AVCaptureResolvedPhotoSettings, bracketSettings: AVCaptureBracketedStillImageSettings?, error: Error?)
    func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    func photoOutput(_ output: AVCapturePhotoOutput, willCapturePhotoFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    func photoOutput(_ output: AVCapturePhotoOutput, didCapturePhotoFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    func photoOutput(_ output: AVCapturePhotoOutput, didFinishCaptureFor resolvedSettings:
    AVCaptureResolvedPhotoSettings, error: Error?)
    @dokun24

    View Slide

  51. Since iOS 10
    @dokun24

    View Slide

  52. Since iOS 10
    AVCaptureResolvedPhotoSettings
    @dokun24

    View Slide

  53. Since iOS 10
    AVCaptureResolvedPhotoSettings
    var uniqueID: Int64
    @dokun24

    View Slide

  54. Since iOS 10
    AVCaptureResolvedPhotoSettings
    var uniqueID: Int64
    var photoDimensions: CMVideoDimensions
    @dokun24

    View Slide

  55. Since iOS 10
    AVCaptureResolvedPhotoSettings
    var uniqueID: Int64
    var photoDimensions: CMVideoDimensions
    var livePhotoMovieDimensions: CMVideoDimensions
    @dokun24

    View Slide

  56. Since iOS 10
    AVCaptureResolvedPhotoSettings
    var uniqueID: Int64
    var photoDimensions: CMVideoDimensions
    var livePhotoMovieDimensions: CMVideoDimensions
    var isFlashEnabled: Bool
    @dokun24

    View Slide

  57. Since iOS 10
    AVCaptureResolvedPhotoSettings
    var uniqueID: Int64
    var photoDimensions: CMVideoDimensions
    var livePhotoMovieDimensions: CMVideoDimensions
    var isFlashEnabled: Bool
    var isStillImageStabilizationEnabled: Bool
    @dokun24

    View Slide

  58. Since iOS 10
    AVCaptureResolvedPhotoSettings
    var uniqueID: Int64
    var photoDimensions: CMVideoDimensions
    var livePhotoMovieDimensions: CMVideoDimensions
    var isFlashEnabled: Bool
    var isStillImageStabilizationEnabled: Bool
    var isDualCameraFusionEnabled: Bool // iOS 10.2+
    @dokun24

    View Slide

  59. New in iOS 11
    @dokun24

    View Slide

  60. New in iOS 11
    AVCapturePhotoOutputDelegate
    @dokun24

    View Slide

  61. New in iOS 11
    AVCapturePhotoOutputDelegate
    func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto
    photo: AVCapturePhoto, error: Error?)
    @dokun24

    View Slide

  62. New in iOS 11
    @dokun24

    View Slide

  63. New in iOS 11
    AVCapturePhoto
    @dokun24

    View Slide

  64. New in iOS 11
    AVCapturePhoto
    var pixelBuffer: CVPixelBuffer?
    @dokun24

    View Slide

  65. New in iOS 11
    AVCapturePhoto
    var pixelBuffer: CVPixelBuffer?
    var previewPixelBuffer: CVPixelBuffer?
    @dokun24

    View Slide

  66. New in iOS 11
    AVCapturePhoto
    var pixelBuffer: CVPixelBuffer?
    var previewPixelBuffer: CVPixelBuffer?
    var metadata: [String : Any]
    @dokun24

    View Slide

  67. New in iOS 11
    AVCapturePhoto
    var pixelBuffer: CVPixelBuffer?
    var previewPixelBuffer: CVPixelBuffer?
    var metadata: [String : Any]
    var resolvedSettings: AVCaptureResolvedPhotoSettings
    @dokun24

    View Slide

  68. New in iOS 11
    AVCapturePhoto
    var pixelBuffer: CVPixelBuffer?
    var previewPixelBuffer: CVPixelBuffer?
    var metadata: [String : Any]
    var resolvedSettings: AVCaptureResolvedPhotoSettings
    var photoCount: Int
    @dokun24

    View Slide

  69. New in iOS 11
    AVCapturePhoto
    var pixelBuffer: CVPixelBuffer?
    var previewPixelBuffer: CVPixelBuffer?
    var metadata: [String : Any]
    var resolvedSettings: AVCaptureResolvedPhotoSettings
    var photoCount: Int
    var depthData: AVDepthData?
    @dokun24

    View Slide

  70. New in iOS 11
    AVCapturePhoto
    @dokun24

    View Slide

  71. New in iOS 11
    AVCapturePhoto
    @dokun24

    View Slide

  72. New in iOS 11
    AVCapturePhoto
    func cgImageRepresentation() -> Unmanaged?
    @dokun24

    View Slide

  73. New in iOS 11
    AVCapturePhoto
    func cgImageRepresentation() -> Unmanaged?
    func takeRetainedValue() -> Instance
    @dokun24

    View Slide

  74. New in iOS 11
    AVCapturePhoto
    func cgImageRepresentation() -> Unmanaged?
    func takeRetainedValue() -> Instance
    func takeUnretainedValue() -> Instance
    @dokun24

    View Slide

  75. Back to iOS 10
    @dokun24

    View Slide

  76. Back to iOS 10
    AVCapturePhotoOutputDelegate -> CMSampleBuffer -> UIImage
    @dokun24

    View Slide

  77. Back to iOS 10
    AVCapturePhotoOutputDelegate -> CMSampleBuffer -> UIImage
    guard let imageData = AVCapturePhotoOutput.jpegPhotoDataRepresentation(forJPEGSampleBuffer:
    sampleBuffer, previewPhotoSampleBuffer: nil) else {
    return nil
    }
    guard let dataProvider = CGDataProvider(data: imageData as CFData) else {
    return nil
    }
    guard let cgImage = CGImage(jpegDataProviderSource: dataProvider, decode: nil,
    shouldInterpolate: true, intent: CGColorRenderingIntent.defaultIntent) else {
    return nil
    }
    return UIImage(cgImage: cgImage)
    @dokun24

    View Slide

  78. New in iOS 11
    @dokun24

    View Slide

  79. New in iOS 11
    AVCapturePhoto -> UIImage
    @dokun24

    View Slide

  80. New in iOS 11
    AVCapturePhoto -> UIImage
    guard let cgImage = photo.cgImageRepresentation() else {
    return nil
    }
    return UIImage(cgImage: cgImage.takeUnretainedValue())
    @dokun24

    View Slide

  81. Capturing Depth Data
    @dokun24

    View Slide

  82. Capturing Depth Data
    @dokun24

    View Slide

  83. Capturing Depth Data
    @dokun24

    View Slide

  84. New in iOS 11
    AVCapturePhoto
    @dokun24
    var pixelBuffer: CVPixelBuffer?
    var previewPixelBuffer: CVPixelBuffer?
    var metadata: [String : Any]
    var resolvedSettings: AVCaptureResolvedPhotoSettings
    var photoCount: Int
    var depthData: AVDepthData?

    View Slide

  85. New in iOS 11
    AVCapturePhoto
    @dokun24
    var pixelBuffer: CVPixelBuffer?
    var previewPixelBuffer: CVPixelBuffer?
    var metadata: [String : Any]
    var resolvedSettings: AVCaptureResolvedPhotoSettings
    var photoCount: Int
    var depthData: AVDepthData?

    View Slide

  86. Capturing Depth Data
    @dokun24

    View Slide

  87. Capturing Depth Data
    AVDepthData
    @dokun24

    View Slide

  88. Capturing Depth Data
    AVDepthData
    var depthDataAccuracy: AVDepthData.Accuracy
    @dokun24

    View Slide

  89. Capturing Depth Data
    AVDepthData
    var depthDataAccuracy: AVDepthData.Accuracy
    var isDepthDataFiltered: Bool
    @dokun24

    View Slide

  90. Capturing Depth Data
    AVDepthData
    var depthDataAccuracy: AVDepthData.Accuracy
    var isDepthDataFiltered: Bool
    var depthDataQuality: AVDepthData.Quality
    @dokun24

    View Slide

  91. Capturing Depth Data
    AVDepthData
    var depthDataAccuracy: AVDepthData.Accuracy
    var isDepthDataFiltered: Bool
    var depthDataQuality: AVDepthData.Quality
    var cameraCalibrationData: AVCameraCalibrationData?
    @dokun24

    View Slide

  92. Capturing Depth Data
    AVDepthData
    var depthDataAccuracy: AVDepthData.Accuracy
    var isDepthDataFiltered: Bool
    var depthDataQuality: AVDepthData.Quality
    var cameraCalibrationData: AVCameraCalibrationData?
    var depthDataMap: CVPixelBuffer
    @dokun24

    View Slide

  93. Capturing Depth Data
    @dokun24

    View Slide

  94. Capturing Depth Data
    // in your capture session configuration
    if self.photoOutput.isDepthDataDeliverySupported {
    self.photoOutput.isDepthDataDeliveryEnabled = true
    }
    @dokun24

    View Slide

  95. Capturing Depth Data
    // in your capture session configuration
    if self.photoOutput.isDepthDataDeliverySupported {
    self.photoOutput.isDepthDataDeliveryEnabled = true
    }
    // in your preparation for capturing a still image with AVCapturePhotoSettings
    if self.photoOutput.isDepthDataDeliverySupported {
    settings.isDepthDataDeliveryEnabled = true
    }
    @dokun24

    View Slide

  96. Lumina
    @dokun24

    View Slide

  97. Lumina
    • Gratuitous plugging!
    @dokun24

    View Slide

  98. Lumina
    • Gratuitous plugging!
    • A full service camera framework
    @dokun24

    View Slide

  99. Lumina
    • Gratuitous plugging!
    • A full service camera framework
    • QR codes, bar codes, live photos, the list
    goes on…
    @dokun24

    View Slide

  100. Lumina
    • Gratuitous plugging!
    • A full service camera framework
    • QR codes, bar codes, live photos, the list
    goes on…
    • …and the easiest CoreML integration you’ll
    see
    @dokun24

    View Slide

  101. Lumina
    • Gratuitous plugging!
    • A full service camera framework
    • QR codes, bar codes, live photos, the list
    goes on…
    • …and the easiest CoreML integration you’ll
    see
    @dokun24

    View Slide

  102. Live Demo Of Lumina
    @dokun24

    View Slide

  103. Synchronizing Asynchronously Streaming Data
    @dokun24

    View Slide

  104. Synchronizing Asynchronously Streaming Data
    • AVCaptureVideoDataOutput -> video frames
    @dokun24

    View Slide

  105. Synchronizing Asynchronously Streaming Data
    • AVCaptureVideoDataOutput -> video frames
    • AVCaptureMetadataOutput -> QR codes, barcodes, faces
    @dokun24

    View Slide

  106. Synchronizing Asynchronously Streaming Data
    • AVCaptureVideoDataOutput -> video frames
    • AVCaptureMetadataOutput -> QR codes, barcodes, faces
    • AVCapturePhotoOutput -> still photos
    @dokun24

    View Slide

  107. Synchronizing Asynchronously Streaming Data
    • AVCaptureVideoDataOutput -> video frames
    • AVCaptureMetadataOutput -> QR codes, barcodes, faces
    • AVCapturePhotoOutput -> still photos
    • AVCaptureDepthDataOutput -> depth data maps
    @dokun24

    View Slide

  108. Synchronizing Asynchronously Streaming Data
    • AVCaptureVideoDataOutput -> video frames
    • AVCaptureMetadataOutput -> QR codes, barcodes, faces
    • AVCapturePhotoOutput -> still photos
    • AVCaptureDepthDataOutput -> depth data maps
    • You see the problem here, right?
    @dokun24

    View Slide

  109. Synchronizing Asynchronously Streaming Data
    @dokun24

    View Slide

  110. Synchronizing Asynchronously Streaming Data
    AVCaptureDataOutputSynchronizer // iOS 11.0 +
    @dokun24

    View Slide

  111. Synchronizing Asynchronously Streaming Data
    AVCaptureDataOutputSynchronizer // iOS 11.0 +
    init(dataOutputs: [AVCaptureOutput])
    @dokun24

    View Slide

  112. Synchronizing Asynchronously Streaming Data
    AVCaptureDataOutputSynchronizer // iOS 11.0 +
    init(dataOutputs: [AVCaptureOutput])
    var delegate: AVCaptureDataOutputSynchronizerDelegate
    @dokun24

    View Slide

  113. Synchronizing Asynchronously Streaming Data
    AVCaptureDataOutputSynchronizer // iOS 11.0 +
    init(dataOutputs: [AVCaptureOutput])
    var delegate: AVCaptureDataOutputSynchronizerDelegate
    var delegateCallbackQueue: DispatchQueue?
    @dokun24

    View Slide

  114. Synchronizing Asynchronously Streaming Data
    @dokun24

    View Slide

  115. Synchronizing Asynchronously Streaming Data
    AVCaptureDataOutputSynchronizerDelegate
    @dokun24

    View Slide

  116. Synchronizing Asynchronously Streaming Data
    AVCaptureDataOutputSynchronizerDelegate
    func dataOutputSynchronizer(_ synchronizer:
    AVCaptureDataOutputSynchronizer, didOutput synchronizedDataCollection:
    AVCaptureSynchronizedDataCollection) {
    let collection = synchronizedDataCollection
    }
    @dokun24

    View Slide

  117. Synchronizing Asynchronously Streaming Data
    @dokun24

    View Slide

  118. Synchronizing Asynchronously Streaming Data
    AVCaptureSynchronizedDataCollection // iOS 11.0 +
    @dokun24

    View Slide

  119. Synchronizing Asynchronously Streaming Data
    AVCaptureSynchronizedDataCollection // iOS 11.0 +
    func synchronizedData(for captureOutput: AVCaptureOutput) ->
    AVCaptureSynchronizedData?
    @dokun24

    View Slide

  120. Synchronizing Asynchronously Streaming Data
    AVCaptureSynchronizedDataCollection // iOS 11.0 +
    func synchronizedData(for captureOutput: AVCaptureOutput) ->
    AVCaptureSynchronizedData?
    @dokun24
    var count: Int

    View Slide

  121. Questions?
    @dokun24

    View Slide