Upgrade to Pro — share decks privately, control downloads, hide ads and more …

AVFoundation, iOS 11 & Lumina

David Okun
November 07, 2017

AVFoundation, iOS 11 & Lumina

Come learn about what’s new in iOS 11 with AVFoundation, as David Okun walks us through new ways to capture still images, live photos, and depth maps via his open source framework, Lumina.

David Okun

November 07, 2017
Tweet

More Decks by David Okun

Other Decks in Programming

Transcript

  1. AVFoundation, iOS 11 & Lumina
    David Okun
    @dokun24

    View full-size slide

  2. WEIRD SWIFT ATX
    @dokun24

    View full-size slide

  3. WEIRD SWIFT ATX
    •Meets every Thursday
    @dokun24

    View full-size slide

  4. WEIRD SWIFT ATX
    •Meets every Thursday
    •6-10 pm
    @dokun24

    View full-size slide

  5. WEIRD SWIFT ATX
    •Meets every Thursday
    •6-10 pm
    •Been at BennU coffee so far but…
    @dokun24

    View full-size slide

  6. WEIRD SWIFT ATX
    •Meets every Thursday
    •6-10 pm
    •Been at BennU coffee so far but…
    •…next week is at Galvanize!!!
    @dokun24

    View full-size slide

  7. What We’ll Cover
    @dokun24

    View full-size slide

  8. What We’ll Cover
    • A history of AVFoundation’s input and output
    @dokun24

    View full-size slide

  9. What We’ll Cover
    • A history of AVFoundation’s input and output
    • What changed in iOS 10
    @dokun24

    View full-size slide

  10. What We’ll Cover
    • A history of AVFoundation’s input and output
    • What changed in iOS 10
    • New methods in iOS 11
    @dokun24

    View full-size slide

  11. What We’ll Cover
    • A history of AVFoundation’s input and output
    • What changed in iOS 10
    • New methods in iOS 11
    • Capturing Depth Data
    @dokun24

    View full-size slide

  12. What We’ll Cover
    • A history of AVFoundation’s input and output
    • What changed in iOS 10
    • New methods in iOS 11
    • Capturing Depth Data
    • Lumina
    @dokun24

    View full-size slide

  13. What We’ll Cover
    • A history of AVFoundation’s input and output
    • What changed in iOS 10
    • New methods in iOS 11
    • Capturing Depth Data
    • Lumina
    • Synchronizing Asynchronously Streaming Data
    @dokun24

    View full-size slide

  14. What We Won’t Cover
    @dokun24

    View full-size slide

  15. What We Won’t Cover
    • High Efficiency Image File Format (HEIF)
    @dokun24

    View full-size slide

  16. What We Won’t Cover
    • High Efficiency Image File Format (HEIF)
    • High Efficiency Video Coding (HEVC)
    @dokun24

    View full-size slide

  17. What We Won’t Cover
    • High Efficiency Image File Format (HEIF)
    • High Efficiency Video Coding (HEVC)
    • AVCaptureDepthDataOutput
    @dokun24

    View full-size slide

  18. Reference Material
    @dokun24

    View full-size slide

  19. Reference Material
    • WWDC 2016, Session 501
    @dokun24

    View full-size slide

  20. Reference Material
    • WWDC 2016, Session 501
    • WWDC 2017, Session 507
    @dokun24

    View full-size slide

  21. Reference Material
    • WWDC 2016, Session 501
    • WWDC 2017, Session 507
    • https://github.com/dokun1/lumina
    @dokun24

    View full-size slide

  22. AVCaptureDevice
    @dokun24

    View full-size slide

  23. AVCaptureDevice
    AVCaptureDeviceInput
    @dokun24

    View full-size slide

  24. AVCaptureDevice
    AVCaptureDeviceInput
    AVCaptureSession
    @dokun24

    View full-size slide

  25. AVCaptureDevice
    AVCaptureDeviceInput
    AVCaptureConnection
    AVCaptureSession
    AVCapturePhotoOutput
    @dokun24

    View full-size slide

  26. AVCaptureDevice
    AVCaptureDeviceInput
    AVCaptureConnection
    AVCaptureConnection
    AVCaptureSession
    AVCapturePhotoOutput
    AVCaptureMovieFileOutput
    @dokun24

    View full-size slide

  27. AVCaptureDevice
    AVCaptureDeviceInput
    AVCaptureConnection
    AVCaptureConnection
    AVCaptureConnection
    AVCaptureSession
    AVCapturePhotoOutput
    AVCaptureMovieFileOutput
    AVCaptureMetadataOutput
    @dokun24

    View full-size slide

  28. AVCaptureDevice
    AVCaptureDeviceInput
    AVCaptureConnection
    AVCaptureConnection
    AVCaptureConnection
    AVCaptureConnection
    AVCaptureSession
    AVCapturePhotoOutput
    AVCaptureMovieFileOutput
    AVCaptureMetadataOutput
    AVCaptureVideoDataOutput
    @dokun24

    View full-size slide

  29. AVCaptureDevice
    AVCaptureDeviceInput
    AVCaptureConnection
    AVCaptureConnection
    AVCaptureConnection
    AVCaptureConnection
    AVCaptureSession
    AVCapturePhotoOutput
    AVCaptureMovieFileOutput
    AVCaptureMetadataOutput
    AVCaptureVideoDataOutput
    AVCaptureVideoPreviewLayer
    @dokun24

    View full-size slide

  30. Before iOS 10
    @dokun24

    View full-size slide

  31. Before iOS 10
    AVCaptureStillImageOutput
    @dokun24

    View full-size slide

  32. Before iOS 10
    AVCaptureStillImageOutput
    func captureStillImageAsynchronously(from connection:
    AVCaptureConnection, completionHandler handler: @escaping
    (CMSampleBuffer?, Error?) -> Swift.Void)
    @dokun24

    View full-size slide

  33. Since iOS 10
    @dokun24

    View full-size slide

  34. Since iOS 10
    AVCapturePhotoOutput
    @dokun24

    View full-size slide

  35. Since iOS 10
    AVCapturePhotoOutput
    var settings = AVCapturePhotoSettings()
    self.photoOutput.capturePhoto(with: settings, delegate: self)
    @dokun24

    View full-size slide

  36. Since iOS 10
    @dokun24

    View full-size slide

  37. Since iOS 10
    AVCapturePhotoSettings
    @dokun24

    View full-size slide

  38. Since iOS 10
    AVCapturePhotoSettings
    var uniqueID: Int64 // generated on init
    @dokun24

    View full-size slide

  39. Since iOS 10
    AVCapturePhotoSettings
    var uniqueID: Int64 // generated on init
    var flashMode: AVCaptureDevice.FlashMode
    @dokun24

    View full-size slide

  40. Since iOS 10
    AVCapturePhotoSettings
    var uniqueID: Int64 // generated on init
    var flashMode: AVCaptureDevice.FlashMode
    var isAutoStillImageStabilizationEnabled: Bool
    @dokun24

    View full-size slide

  41. Since iOS 10
    AVCapturePhotoSettings
    var uniqueID: Int64 // generated on init
    var flashMode: AVCaptureDevice.FlashMode
    var isAutoStillImageStabilizationEnabled: Bool
    var isHighResolutionPhotoEnabled: Bool
    @dokun24

    View full-size slide

  42. Since iOS 10
    AVCapturePhotoSettings
    var uniqueID: Int64 // generated on init
    var flashMode: AVCaptureDevice.FlashMode
    var isAutoStillImageStabilizationEnabled: Bool
    var isHighResolutionPhotoEnabled: Bool
    var isAutoDualCameraFusionEnabled: Bool // iOS 10.2+
    @dokun24

    View full-size slide

  43. Since iOS 10
    @dokun24

    View full-size slide

  44. Since iOS 10
    AVCapturePhotoOutputDelegate
    @dokun24

    View full-size slide

  45. Since iOS 10
    AVCapturePhotoOutputDelegate
    func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    @dokun24

    View full-size slide

  46. Since iOS 10
    AVCapturePhotoOutputDelegate
    func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    func photoOutput(_ output: AVCapturePhotoOutput, willCapturePhotoFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    @dokun24

    View full-size slide

  47. Since iOS 10
    AVCapturePhotoOutputDelegate
    func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    func photoOutput(_ output: AVCapturePhotoOutput, willCapturePhotoFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    func photoOutput(_ output: AVCapturePhotoOutput, didCapturePhotoFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    @dokun24

    View full-size slide

  48. Since iOS 10
    AVCapturePhotoOutputDelegate
    func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photoSampleBuffer:
    CMSampleBuffer?, previewPhoto previewPhotoSampleBuffer: CMSampleBuffer?, resolvedSettings:
    AVCaptureResolvedPhotoSettings, bracketSettings: AVCaptureBracketedStillImageSettings?, error: Error?)
    func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    func photoOutput(_ output: AVCapturePhotoOutput, willCapturePhotoFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    func photoOutput(_ output: AVCapturePhotoOutput, didCapturePhotoFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    @dokun24

    View full-size slide

  49. Since iOS 10
    AVCapturePhotoOutputDelegate
    func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photoSampleBuffer:
    CMSampleBuffer?, previewPhoto previewPhotoSampleBuffer: CMSampleBuffer?, resolvedSettings:
    AVCaptureResolvedPhotoSettings, bracketSettings: AVCaptureBracketedStillImageSettings?, error: Error?)
    func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    func photoOutput(_ output: AVCapturePhotoOutput, willCapturePhotoFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    func photoOutput(_ output: AVCapturePhotoOutput, didCapturePhotoFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    func photoOutput(_ output: AVCapturePhotoOutput, didFinishCaptureFor resolvedSettings:
    AVCaptureResolvedPhotoSettings, error: Error?)
    @dokun24

    View full-size slide

  50. Since iOS 10
    @dokun24

    View full-size slide

  51. Since iOS 10
    AVCaptureResolvedPhotoSettings
    @dokun24

    View full-size slide

  52. Since iOS 10
    AVCaptureResolvedPhotoSettings
    var uniqueID: Int64
    @dokun24

    View full-size slide

  53. Since iOS 10
    AVCaptureResolvedPhotoSettings
    var uniqueID: Int64
    var photoDimensions: CMVideoDimensions
    @dokun24

    View full-size slide

  54. Since iOS 10
    AVCaptureResolvedPhotoSettings
    var uniqueID: Int64
    var photoDimensions: CMVideoDimensions
    var livePhotoMovieDimensions: CMVideoDimensions
    @dokun24

    View full-size slide

  55. Since iOS 10
    AVCaptureResolvedPhotoSettings
    var uniqueID: Int64
    var photoDimensions: CMVideoDimensions
    var livePhotoMovieDimensions: CMVideoDimensions
    var isFlashEnabled: Bool
    @dokun24

    View full-size slide

  56. Since iOS 10
    AVCaptureResolvedPhotoSettings
    var uniqueID: Int64
    var photoDimensions: CMVideoDimensions
    var livePhotoMovieDimensions: CMVideoDimensions
    var isFlashEnabled: Bool
    var isStillImageStabilizationEnabled: Bool
    @dokun24

    View full-size slide

  57. Since iOS 10
    AVCaptureResolvedPhotoSettings
    var uniqueID: Int64
    var photoDimensions: CMVideoDimensions
    var livePhotoMovieDimensions: CMVideoDimensions
    var isFlashEnabled: Bool
    var isStillImageStabilizationEnabled: Bool
    var isDualCameraFusionEnabled: Bool // iOS 10.2+
    @dokun24

    View full-size slide

  58. New in iOS 11
    @dokun24

    View full-size slide

  59. New in iOS 11
    AVCapturePhotoOutputDelegate
    @dokun24

    View full-size slide

  60. New in iOS 11
    AVCapturePhotoOutputDelegate
    func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto
    photo: AVCapturePhoto, error: Error?)
    @dokun24

    View full-size slide

  61. New in iOS 11
    @dokun24

    View full-size slide

  62. New in iOS 11
    AVCapturePhoto
    @dokun24

    View full-size slide

  63. New in iOS 11
    AVCapturePhoto
    var pixelBuffer: CVPixelBuffer?
    @dokun24

    View full-size slide

  64. New in iOS 11
    AVCapturePhoto
    var pixelBuffer: CVPixelBuffer?
    var previewPixelBuffer: CVPixelBuffer?
    @dokun24

    View full-size slide

  65. New in iOS 11
    AVCapturePhoto
    var pixelBuffer: CVPixelBuffer?
    var previewPixelBuffer: CVPixelBuffer?
    var metadata: [String : Any]
    @dokun24

    View full-size slide

  66. New in iOS 11
    AVCapturePhoto
    var pixelBuffer: CVPixelBuffer?
    var previewPixelBuffer: CVPixelBuffer?
    var metadata: [String : Any]
    var resolvedSettings: AVCaptureResolvedPhotoSettings
    @dokun24

    View full-size slide

  67. New in iOS 11
    AVCapturePhoto
    var pixelBuffer: CVPixelBuffer?
    var previewPixelBuffer: CVPixelBuffer?
    var metadata: [String : Any]
    var resolvedSettings: AVCaptureResolvedPhotoSettings
    var photoCount: Int
    @dokun24

    View full-size slide

  68. New in iOS 11
    AVCapturePhoto
    var pixelBuffer: CVPixelBuffer?
    var previewPixelBuffer: CVPixelBuffer?
    var metadata: [String : Any]
    var resolvedSettings: AVCaptureResolvedPhotoSettings
    var photoCount: Int
    var depthData: AVDepthData?
    @dokun24

    View full-size slide

  69. New in iOS 11
    AVCapturePhoto
    @dokun24

    View full-size slide

  70. New in iOS 11
    AVCapturePhoto
    @dokun24

    View full-size slide

  71. New in iOS 11
    AVCapturePhoto
    func cgImageRepresentation() -> Unmanaged?
    @dokun24

    View full-size slide

  72. New in iOS 11
    AVCapturePhoto
    func cgImageRepresentation() -> Unmanaged?
    func takeRetainedValue() -> Instance
    @dokun24

    View full-size slide

  73. New in iOS 11
    AVCapturePhoto
    func cgImageRepresentation() -> Unmanaged?
    func takeRetainedValue() -> Instance
    func takeUnretainedValue() -> Instance
    @dokun24

    View full-size slide

  74. Back to iOS 10
    @dokun24

    View full-size slide

  75. Back to iOS 10
    AVCapturePhotoOutputDelegate -> CMSampleBuffer -> UIImage
    @dokun24

    View full-size slide

  76. Back to iOS 10
    AVCapturePhotoOutputDelegate -> CMSampleBuffer -> UIImage
    guard let imageData = AVCapturePhotoOutput.jpegPhotoDataRepresentation(forJPEGSampleBuffer:
    sampleBuffer, previewPhotoSampleBuffer: nil) else {
    return nil
    }
    guard let dataProvider = CGDataProvider(data: imageData as CFData) else {
    return nil
    }
    guard let cgImage = CGImage(jpegDataProviderSource: dataProvider, decode: nil,
    shouldInterpolate: true, intent: CGColorRenderingIntent.defaultIntent) else {
    return nil
    }
    return UIImage(cgImage: cgImage)
    @dokun24

    View full-size slide

  77. New in iOS 11
    @dokun24

    View full-size slide

  78. New in iOS 11
    AVCapturePhoto -> UIImage
    @dokun24

    View full-size slide

  79. New in iOS 11
    AVCapturePhoto -> UIImage
    guard let cgImage = photo.cgImageRepresentation() else {
    return nil
    }
    return UIImage(cgImage: cgImage.takeUnretainedValue())
    @dokun24

    View full-size slide

  80. Capturing Depth Data
    @dokun24

    View full-size slide

  81. Capturing Depth Data
    @dokun24

    View full-size slide

  82. Capturing Depth Data
    @dokun24

    View full-size slide

  83. New in iOS 11
    AVCapturePhoto
    @dokun24
    var pixelBuffer: CVPixelBuffer?
    var previewPixelBuffer: CVPixelBuffer?
    var metadata: [String : Any]
    var resolvedSettings: AVCaptureResolvedPhotoSettings
    var photoCount: Int
    var depthData: AVDepthData?

    View full-size slide

  84. New in iOS 11
    AVCapturePhoto
    @dokun24
    var pixelBuffer: CVPixelBuffer?
    var previewPixelBuffer: CVPixelBuffer?
    var metadata: [String : Any]
    var resolvedSettings: AVCaptureResolvedPhotoSettings
    var photoCount: Int
    var depthData: AVDepthData?

    View full-size slide

  85. Capturing Depth Data
    @dokun24

    View full-size slide

  86. Capturing Depth Data
    AVDepthData
    @dokun24

    View full-size slide

  87. Capturing Depth Data
    AVDepthData
    var depthDataAccuracy: AVDepthData.Accuracy
    @dokun24

    View full-size slide

  88. Capturing Depth Data
    AVDepthData
    var depthDataAccuracy: AVDepthData.Accuracy
    var isDepthDataFiltered: Bool
    @dokun24

    View full-size slide

  89. Capturing Depth Data
    AVDepthData
    var depthDataAccuracy: AVDepthData.Accuracy
    var isDepthDataFiltered: Bool
    var depthDataQuality: AVDepthData.Quality
    @dokun24

    View full-size slide

  90. Capturing Depth Data
    AVDepthData
    var depthDataAccuracy: AVDepthData.Accuracy
    var isDepthDataFiltered: Bool
    var depthDataQuality: AVDepthData.Quality
    var cameraCalibrationData: AVCameraCalibrationData?
    @dokun24

    View full-size slide

  91. Capturing Depth Data
    AVDepthData
    var depthDataAccuracy: AVDepthData.Accuracy
    var isDepthDataFiltered: Bool
    var depthDataQuality: AVDepthData.Quality
    var cameraCalibrationData: AVCameraCalibrationData?
    var depthDataMap: CVPixelBuffer
    @dokun24

    View full-size slide

  92. Capturing Depth Data
    @dokun24

    View full-size slide

  93. Capturing Depth Data
    // in your capture session configuration
    if self.photoOutput.isDepthDataDeliverySupported {
    self.photoOutput.isDepthDataDeliveryEnabled = true
    }
    @dokun24

    View full-size slide

  94. Capturing Depth Data
    // in your capture session configuration
    if self.photoOutput.isDepthDataDeliverySupported {
    self.photoOutput.isDepthDataDeliveryEnabled = true
    }
    // in your preparation for capturing a still image with AVCapturePhotoSettings
    if self.photoOutput.isDepthDataDeliverySupported {
    settings.isDepthDataDeliveryEnabled = true
    }
    @dokun24

    View full-size slide

  95. Lumina
    @dokun24

    View full-size slide

  96. Lumina
    • Gratuitous plugging!
    @dokun24

    View full-size slide

  97. Lumina
    • Gratuitous plugging!
    • A full service camera framework
    @dokun24

    View full-size slide

  98. Lumina
    • Gratuitous plugging!
    • A full service camera framework
    • QR codes, bar codes, live photos, the list
    goes on…
    @dokun24

    View full-size slide

  99. Lumina
    • Gratuitous plugging!
    • A full service camera framework
    • QR codes, bar codes, live photos, the list
    goes on…
    • …and the easiest CoreML integration you’ll
    see
    @dokun24

    View full-size slide

  100. Lumina
    • Gratuitous plugging!
    • A full service camera framework
    • QR codes, bar codes, live photos, the list
    goes on…
    • …and the easiest CoreML integration you’ll
    see
    @dokun24

    View full-size slide

  101. Live Demo Of Lumina
    @dokun24

    View full-size slide

  102. Synchronizing Asynchronously Streaming Data
    @dokun24

    View full-size slide

  103. Synchronizing Asynchronously Streaming Data
    • AVCaptureVideoDataOutput -> video frames
    @dokun24

    View full-size slide

  104. Synchronizing Asynchronously Streaming Data
    • AVCaptureVideoDataOutput -> video frames
    • AVCaptureMetadataOutput -> QR codes, barcodes, faces
    @dokun24

    View full-size slide

  105. Synchronizing Asynchronously Streaming Data
    • AVCaptureVideoDataOutput -> video frames
    • AVCaptureMetadataOutput -> QR codes, barcodes, faces
    • AVCapturePhotoOutput -> still photos
    @dokun24

    View full-size slide

  106. Synchronizing Asynchronously Streaming Data
    • AVCaptureVideoDataOutput -> video frames
    • AVCaptureMetadataOutput -> QR codes, barcodes, faces
    • AVCapturePhotoOutput -> still photos
    • AVCaptureDepthDataOutput -> depth data maps
    @dokun24

    View full-size slide

  107. Synchronizing Asynchronously Streaming Data
    • AVCaptureVideoDataOutput -> video frames
    • AVCaptureMetadataOutput -> QR codes, barcodes, faces
    • AVCapturePhotoOutput -> still photos
    • AVCaptureDepthDataOutput -> depth data maps
    • You see the problem here, right?
    @dokun24

    View full-size slide

  108. Synchronizing Asynchronously Streaming Data
    @dokun24

    View full-size slide

  109. Synchronizing Asynchronously Streaming Data
    AVCaptureDataOutputSynchronizer // iOS 11.0 +
    @dokun24

    View full-size slide

  110. Synchronizing Asynchronously Streaming Data
    AVCaptureDataOutputSynchronizer // iOS 11.0 +
    init(dataOutputs: [AVCaptureOutput])
    @dokun24

    View full-size slide

  111. Synchronizing Asynchronously Streaming Data
    AVCaptureDataOutputSynchronizer // iOS 11.0 +
    init(dataOutputs: [AVCaptureOutput])
    var delegate: AVCaptureDataOutputSynchronizerDelegate
    @dokun24

    View full-size slide

  112. Synchronizing Asynchronously Streaming Data
    AVCaptureDataOutputSynchronizer // iOS 11.0 +
    init(dataOutputs: [AVCaptureOutput])
    var delegate: AVCaptureDataOutputSynchronizerDelegate
    var delegateCallbackQueue: DispatchQueue?
    @dokun24

    View full-size slide

  113. Synchronizing Asynchronously Streaming Data
    @dokun24

    View full-size slide

  114. Synchronizing Asynchronously Streaming Data
    AVCaptureDataOutputSynchronizerDelegate
    @dokun24

    View full-size slide

  115. Synchronizing Asynchronously Streaming Data
    AVCaptureDataOutputSynchronizerDelegate
    func dataOutputSynchronizer(_ synchronizer:
    AVCaptureDataOutputSynchronizer, didOutput synchronizedDataCollection:
    AVCaptureSynchronizedDataCollection) {
    let collection = synchronizedDataCollection
    }
    @dokun24

    View full-size slide

  116. Synchronizing Asynchronously Streaming Data
    @dokun24

    View full-size slide

  117. Synchronizing Asynchronously Streaming Data
    AVCaptureSynchronizedDataCollection // iOS 11.0 +
    @dokun24

    View full-size slide

  118. Synchronizing Asynchronously Streaming Data
    AVCaptureSynchronizedDataCollection // iOS 11.0 +
    func synchronizedData(for captureOutput: AVCaptureOutput) ->
    AVCaptureSynchronizedData?
    @dokun24

    View full-size slide

  119. Synchronizing Asynchronously Streaming Data
    AVCaptureSynchronizedDataCollection // iOS 11.0 +
    func synchronizedData(for captureOutput: AVCaptureOutput) ->
    AVCaptureSynchronizedData?
    @dokun24
    var count: Int

    View full-size slide

  120. Questions?
    @dokun24

    View full-size slide