AVFoundation, iOS 11 & Lumina

0f552f8411335ac247e390aa03790990?s=47 David Okun
November 07, 2017

AVFoundation, iOS 11 & Lumina

Come learn about what’s new in iOS 11 with AVFoundation, as David Okun walks us through new ways to capture still images, live photos, and depth maps via his open source framework, Lumina.

0f552f8411335ac247e390aa03790990?s=128

David Okun

November 07, 2017
Tweet

Transcript

  1. AVFoundation, iOS 11 & Lumina David Okun @dokun24

  2. WEIRD SWIFT ATX @dokun24

  3. WEIRD SWIFT ATX •Meets every Thursday @dokun24

  4. WEIRD SWIFT ATX •Meets every Thursday •6-10 pm @dokun24

  5. WEIRD SWIFT ATX •Meets every Thursday •6-10 pm •Been at

    BennU coffee so far but… @dokun24
  6. WEIRD SWIFT ATX •Meets every Thursday •6-10 pm •Been at

    BennU coffee so far but… •…next week is at Galvanize!!! @dokun24
  7. What We’ll Cover @dokun24

  8. What We’ll Cover • A history of AVFoundation’s input and

    output @dokun24
  9. What We’ll Cover • A history of AVFoundation’s input and

    output • What changed in iOS 10 @dokun24
  10. What We’ll Cover • A history of AVFoundation’s input and

    output • What changed in iOS 10 • New methods in iOS 11 @dokun24
  11. What We’ll Cover • A history of AVFoundation’s input and

    output • What changed in iOS 10 • New methods in iOS 11 • Capturing Depth Data @dokun24
  12. What We’ll Cover • A history of AVFoundation’s input and

    output • What changed in iOS 10 • New methods in iOS 11 • Capturing Depth Data • Lumina @dokun24
  13. What We’ll Cover • A history of AVFoundation’s input and

    output • What changed in iOS 10 • New methods in iOS 11 • Capturing Depth Data • Lumina • Synchronizing Asynchronously Streaming Data @dokun24
  14. What We Won’t Cover @dokun24

  15. What We Won’t Cover • High Efficiency Image File Format

    (HEIF) @dokun24
  16. What We Won’t Cover • High Efficiency Image File Format

    (HEIF) • High Efficiency Video Coding (HEVC) @dokun24
  17. What We Won’t Cover • High Efficiency Image File Format

    (HEIF) • High Efficiency Video Coding (HEVC) • AVCaptureDepthDataOutput @dokun24
  18. Reference Material @dokun24

  19. Reference Material • WWDC 2016, Session 501 @dokun24

  20. Reference Material • WWDC 2016, Session 501 • WWDC 2017,

    Session 507 @dokun24
  21. Reference Material • WWDC 2016, Session 501 • WWDC 2017,

    Session 507 • https://github.com/dokun1/lumina @dokun24
  22. @dokun24

  23. AVCaptureDevice @dokun24

  24. AVCaptureDevice AVCaptureDeviceInput @dokun24

  25. AVCaptureDevice AVCaptureDeviceInput AVCaptureSession @dokun24

  26. AVCaptureDevice AVCaptureDeviceInput AVCaptureConnection AVCaptureSession AVCapturePhotoOutput @dokun24

  27. AVCaptureDevice AVCaptureDeviceInput AVCaptureConnection AVCaptureConnection AVCaptureSession AVCapturePhotoOutput AVCaptureMovieFileOutput @dokun24

  28. AVCaptureDevice AVCaptureDeviceInput AVCaptureConnection AVCaptureConnection AVCaptureConnection AVCaptureSession AVCapturePhotoOutput AVCaptureMovieFileOutput AVCaptureMetadataOutput @dokun24

  29. AVCaptureDevice AVCaptureDeviceInput AVCaptureConnection AVCaptureConnection AVCaptureConnection AVCaptureConnection AVCaptureSession AVCapturePhotoOutput AVCaptureMovieFileOutput AVCaptureMetadataOutput

    AVCaptureVideoDataOutput @dokun24
  30. AVCaptureDevice AVCaptureDeviceInput AVCaptureConnection AVCaptureConnection AVCaptureConnection AVCaptureConnection AVCaptureSession AVCapturePhotoOutput AVCaptureMovieFileOutput AVCaptureMetadataOutput

    AVCaptureVideoDataOutput AVCaptureVideoPreviewLayer @dokun24
  31. Before iOS 10 @dokun24

  32. Before iOS 10 AVCaptureStillImageOutput @dokun24

  33. Before iOS 10 AVCaptureStillImageOutput func captureStillImageAsynchronously(from connection: AVCaptureConnection, completionHandler handler:

    @escaping (CMSampleBuffer?, Error?) -> Swift.Void) @dokun24
  34. Since iOS 10 @dokun24

  35. Since iOS 10 AVCapturePhotoOutput @dokun24

  36. Since iOS 10 AVCapturePhotoOutput var settings = AVCapturePhotoSettings() self.photoOutput.capturePhoto(with: settings,

    delegate: self) @dokun24
  37. Since iOS 10 @dokun24

  38. Since iOS 10 AVCapturePhotoSettings @dokun24

  39. Since iOS 10 AVCapturePhotoSettings var uniqueID: Int64 // generated on

    init @dokun24
  40. Since iOS 10 AVCapturePhotoSettings var uniqueID: Int64 // generated on

    init var flashMode: AVCaptureDevice.FlashMode @dokun24
  41. Since iOS 10 AVCapturePhotoSettings var uniqueID: Int64 // generated on

    init var flashMode: AVCaptureDevice.FlashMode var isAutoStillImageStabilizationEnabled: Bool @dokun24
  42. Since iOS 10 AVCapturePhotoSettings var uniqueID: Int64 // generated on

    init var flashMode: AVCaptureDevice.FlashMode var isAutoStillImageStabilizationEnabled: Bool var isHighResolutionPhotoEnabled: Bool @dokun24
  43. Since iOS 10 AVCapturePhotoSettings var uniqueID: Int64 // generated on

    init var flashMode: AVCaptureDevice.FlashMode var isAutoStillImageStabilizationEnabled: Bool var isHighResolutionPhotoEnabled: Bool var isAutoDualCameraFusionEnabled: Bool // iOS 10.2+ @dokun24
  44. Since iOS 10 @dokun24

  45. Since iOS 10 AVCapturePhotoOutputDelegate @dokun24

  46. Since iOS 10 AVCapturePhotoOutputDelegate func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings:

    AVCaptureResolvedPhotoSettings) @dokun24
  47. Since iOS 10 AVCapturePhotoOutputDelegate func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings:

    AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, willCapturePhotoFor resolvedSettings: AVCaptureResolvedPhotoSettings) @dokun24
  48. Since iOS 10 AVCapturePhotoOutputDelegate func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings:

    AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, willCapturePhotoFor resolvedSettings: AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, didCapturePhotoFor resolvedSettings: AVCaptureResolvedPhotoSettings) @dokun24
  49. Since iOS 10 AVCapturePhotoOutputDelegate func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photoSampleBuffer:

    CMSampleBuffer?, previewPhoto previewPhotoSampleBuffer: CMSampleBuffer?, resolvedSettings: AVCaptureResolvedPhotoSettings, bracketSettings: AVCaptureBracketedStillImageSettings?, error: Error?) func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings: AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, willCapturePhotoFor resolvedSettings: AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, didCapturePhotoFor resolvedSettings: AVCaptureResolvedPhotoSettings) @dokun24
  50. Since iOS 10 AVCapturePhotoOutputDelegate func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photoSampleBuffer:

    CMSampleBuffer?, previewPhoto previewPhotoSampleBuffer: CMSampleBuffer?, resolvedSettings: AVCaptureResolvedPhotoSettings, bracketSettings: AVCaptureBracketedStillImageSettings?, error: Error?) func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings: AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, willCapturePhotoFor resolvedSettings: AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, didCapturePhotoFor resolvedSettings: AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, didFinishCaptureFor resolvedSettings: AVCaptureResolvedPhotoSettings, error: Error?) @dokun24
  51. Since iOS 10 @dokun24

  52. Since iOS 10 AVCaptureResolvedPhotoSettings @dokun24

  53. Since iOS 10 AVCaptureResolvedPhotoSettings var uniqueID: Int64 @dokun24

  54. Since iOS 10 AVCaptureResolvedPhotoSettings var uniqueID: Int64 var photoDimensions: CMVideoDimensions

    @dokun24
  55. Since iOS 10 AVCaptureResolvedPhotoSettings var uniqueID: Int64 var photoDimensions: CMVideoDimensions

    var livePhotoMovieDimensions: CMVideoDimensions @dokun24
  56. Since iOS 10 AVCaptureResolvedPhotoSettings var uniqueID: Int64 var photoDimensions: CMVideoDimensions

    var livePhotoMovieDimensions: CMVideoDimensions var isFlashEnabled: Bool @dokun24
  57. Since iOS 10 AVCaptureResolvedPhotoSettings var uniqueID: Int64 var photoDimensions: CMVideoDimensions

    var livePhotoMovieDimensions: CMVideoDimensions var isFlashEnabled: Bool var isStillImageStabilizationEnabled: Bool @dokun24
  58. Since iOS 10 AVCaptureResolvedPhotoSettings var uniqueID: Int64 var photoDimensions: CMVideoDimensions

    var livePhotoMovieDimensions: CMVideoDimensions var isFlashEnabled: Bool var isStillImageStabilizationEnabled: Bool var isDualCameraFusionEnabled: Bool // iOS 10.2+ @dokun24
  59. New in iOS 11 @dokun24

  60. New in iOS 11 AVCapturePhotoOutputDelegate @dokun24

  61. New in iOS 11 AVCapturePhotoOutputDelegate func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto

    photo: AVCapturePhoto, error: Error?) @dokun24
  62. New in iOS 11 @dokun24

  63. New in iOS 11 AVCapturePhoto @dokun24

  64. New in iOS 11 AVCapturePhoto var pixelBuffer: CVPixelBuffer? @dokun24

  65. New in iOS 11 AVCapturePhoto var pixelBuffer: CVPixelBuffer? var previewPixelBuffer:

    CVPixelBuffer? @dokun24
  66. New in iOS 11 AVCapturePhoto var pixelBuffer: CVPixelBuffer? var previewPixelBuffer:

    CVPixelBuffer? var metadata: [String : Any] @dokun24
  67. New in iOS 11 AVCapturePhoto var pixelBuffer: CVPixelBuffer? var previewPixelBuffer:

    CVPixelBuffer? var metadata: [String : Any] var resolvedSettings: AVCaptureResolvedPhotoSettings @dokun24
  68. New in iOS 11 AVCapturePhoto var pixelBuffer: CVPixelBuffer? var previewPixelBuffer:

    CVPixelBuffer? var metadata: [String : Any] var resolvedSettings: AVCaptureResolvedPhotoSettings var photoCount: Int @dokun24
  69. New in iOS 11 AVCapturePhoto var pixelBuffer: CVPixelBuffer? var previewPixelBuffer:

    CVPixelBuffer? var metadata: [String : Any] var resolvedSettings: AVCaptureResolvedPhotoSettings var photoCount: Int var depthData: AVDepthData? @dokun24
  70. New in iOS 11 AVCapturePhoto @dokun24

  71. New in iOS 11 AVCapturePhoto @dokun24

  72. New in iOS 11 AVCapturePhoto func cgImageRepresentation() -> Unmanaged<CGImage>? @dokun24

  73. New in iOS 11 AVCapturePhoto func cgImageRepresentation() -> Unmanaged<CGImage>? func

    takeRetainedValue() -> Instance @dokun24
  74. New in iOS 11 AVCapturePhoto func cgImageRepresentation() -> Unmanaged<CGImage>? func

    takeRetainedValue() -> Instance func takeUnretainedValue() -> Instance @dokun24
  75. Back to iOS 10 @dokun24

  76. Back to iOS 10 AVCapturePhotoOutputDelegate -> CMSampleBuffer -> UIImage @dokun24

  77. Back to iOS 10 AVCapturePhotoOutputDelegate -> CMSampleBuffer -> UIImage guard

    let imageData = AVCapturePhotoOutput.jpegPhotoDataRepresentation(forJPEGSampleBuffer: sampleBuffer, previewPhotoSampleBuffer: nil) else { return nil } guard let dataProvider = CGDataProvider(data: imageData as CFData) else { return nil } guard let cgImage = CGImage(jpegDataProviderSource: dataProvider, decode: nil, shouldInterpolate: true, intent: CGColorRenderingIntent.defaultIntent) else { return nil } return UIImage(cgImage: cgImage) @dokun24
  78. New in iOS 11 @dokun24

  79. New in iOS 11 AVCapturePhoto -> UIImage @dokun24

  80. New in iOS 11 AVCapturePhoto -> UIImage guard let cgImage

    = photo.cgImageRepresentation() else { return nil } return UIImage(cgImage: cgImage.takeUnretainedValue()) @dokun24
  81. Capturing Depth Data @dokun24

  82. Capturing Depth Data @dokun24

  83. Capturing Depth Data @dokun24

  84. New in iOS 11 AVCapturePhoto @dokun24 var pixelBuffer: CVPixelBuffer? var

    previewPixelBuffer: CVPixelBuffer? var metadata: [String : Any] var resolvedSettings: AVCaptureResolvedPhotoSettings var photoCount: Int var depthData: AVDepthData?
  85. New in iOS 11 AVCapturePhoto @dokun24 var pixelBuffer: CVPixelBuffer? var

    previewPixelBuffer: CVPixelBuffer? var metadata: [String : Any] var resolvedSettings: AVCaptureResolvedPhotoSettings var photoCount: Int var depthData: AVDepthData?
  86. Capturing Depth Data @dokun24

  87. Capturing Depth Data AVDepthData @dokun24

  88. Capturing Depth Data AVDepthData var depthDataAccuracy: AVDepthData.Accuracy @dokun24

  89. Capturing Depth Data AVDepthData var depthDataAccuracy: AVDepthData.Accuracy var isDepthDataFiltered: Bool

    @dokun24
  90. Capturing Depth Data AVDepthData var depthDataAccuracy: AVDepthData.Accuracy var isDepthDataFiltered: Bool

    var depthDataQuality: AVDepthData.Quality @dokun24
  91. Capturing Depth Data AVDepthData var depthDataAccuracy: AVDepthData.Accuracy var isDepthDataFiltered: Bool

    var depthDataQuality: AVDepthData.Quality var cameraCalibrationData: AVCameraCalibrationData? @dokun24
  92. Capturing Depth Data AVDepthData var depthDataAccuracy: AVDepthData.Accuracy var isDepthDataFiltered: Bool

    var depthDataQuality: AVDepthData.Quality var cameraCalibrationData: AVCameraCalibrationData? var depthDataMap: CVPixelBuffer @dokun24
  93. Capturing Depth Data @dokun24

  94. Capturing Depth Data // in your capture session configuration if

    self.photoOutput.isDepthDataDeliverySupported { self.photoOutput.isDepthDataDeliveryEnabled = true } @dokun24
  95. Capturing Depth Data // in your capture session configuration if

    self.photoOutput.isDepthDataDeliverySupported { self.photoOutput.isDepthDataDeliveryEnabled = true } // in your preparation for capturing a still image with AVCapturePhotoSettings if self.photoOutput.isDepthDataDeliverySupported { settings.isDepthDataDeliveryEnabled = true } @dokun24
  96. Lumina @dokun24

  97. Lumina • Gratuitous plugging! @dokun24

  98. Lumina • Gratuitous plugging! • A full service camera framework

    @dokun24
  99. Lumina • Gratuitous plugging! • A full service camera framework

    • QR codes, bar codes, live photos, the list goes on… @dokun24
  100. Lumina • Gratuitous plugging! • A full service camera framework

    • QR codes, bar codes, live photos, the list goes on… • …and the easiest CoreML integration you’ll see @dokun24
  101. Lumina • Gratuitous plugging! • A full service camera framework

    • QR codes, bar codes, live photos, the list goes on… • …and the easiest CoreML integration you’ll see @dokun24
  102. Live Demo Of Lumina @dokun24

  103. Synchronizing Asynchronously Streaming Data @dokun24

  104. Synchronizing Asynchronously Streaming Data • AVCaptureVideoDataOutput -> video frames @dokun24

  105. Synchronizing Asynchronously Streaming Data • AVCaptureVideoDataOutput -> video frames •

    AVCaptureMetadataOutput -> QR codes, barcodes, faces @dokun24
  106. Synchronizing Asynchronously Streaming Data • AVCaptureVideoDataOutput -> video frames •

    AVCaptureMetadataOutput -> QR codes, barcodes, faces • AVCapturePhotoOutput -> still photos @dokun24
  107. Synchronizing Asynchronously Streaming Data • AVCaptureVideoDataOutput -> video frames •

    AVCaptureMetadataOutput -> QR codes, barcodes, faces • AVCapturePhotoOutput -> still photos • AVCaptureDepthDataOutput -> depth data maps @dokun24
  108. Synchronizing Asynchronously Streaming Data • AVCaptureVideoDataOutput -> video frames •

    AVCaptureMetadataOutput -> QR codes, barcodes, faces • AVCapturePhotoOutput -> still photos • AVCaptureDepthDataOutput -> depth data maps • You see the problem here, right? @dokun24
  109. Synchronizing Asynchronously Streaming Data @dokun24

  110. Synchronizing Asynchronously Streaming Data AVCaptureDataOutputSynchronizer // iOS 11.0 + @dokun24

  111. Synchronizing Asynchronously Streaming Data AVCaptureDataOutputSynchronizer // iOS 11.0 + init(dataOutputs:

    [AVCaptureOutput]) @dokun24
  112. Synchronizing Asynchronously Streaming Data AVCaptureDataOutputSynchronizer // iOS 11.0 + init(dataOutputs:

    [AVCaptureOutput]) var delegate: AVCaptureDataOutputSynchronizerDelegate @dokun24
  113. Synchronizing Asynchronously Streaming Data AVCaptureDataOutputSynchronizer // iOS 11.0 + init(dataOutputs:

    [AVCaptureOutput]) var delegate: AVCaptureDataOutputSynchronizerDelegate var delegateCallbackQueue: DispatchQueue? @dokun24
  114. Synchronizing Asynchronously Streaming Data @dokun24

  115. Synchronizing Asynchronously Streaming Data AVCaptureDataOutputSynchronizerDelegate @dokun24

  116. Synchronizing Asynchronously Streaming Data AVCaptureDataOutputSynchronizerDelegate func dataOutputSynchronizer(_ synchronizer: AVCaptureDataOutputSynchronizer, didOutput

    synchronizedDataCollection: AVCaptureSynchronizedDataCollection) { let collection = synchronizedDataCollection } @dokun24
  117. Synchronizing Asynchronously Streaming Data @dokun24

  118. Synchronizing Asynchronously Streaming Data AVCaptureSynchronizedDataCollection // iOS 11.0 + @dokun24

  119. Synchronizing Asynchronously Streaming Data AVCaptureSynchronizedDataCollection // iOS 11.0 + func

    synchronizedData(for captureOutput: AVCaptureOutput) -> AVCaptureSynchronizedData? @dokun24
  120. Synchronizing Asynchronously Streaming Data AVCaptureSynchronizedDataCollection // iOS 11.0 + func

    synchronizedData(for captureOutput: AVCaptureOutput) -> AVCaptureSynchronizedData? @dokun24 var count: Int
  121. Questions? @dokun24