AVFoundation, iOS 11 & Lumina

Come learn about what’s new in iOS 11 with AVFoundation, as David Okun walks us through new ways to capture still images, live photos, and depth maps via his open source framework, Lumina.

Transcript

1. AVFoundation, iOS 11 & Lumina
   David Okun
   @dokun24
   

   View Slide

2. WEIRD SWIFT ATX
   @dokun24
   

   View Slide

3. WEIRD SWIFT ATX
   •Meets every Thursday
   @dokun24
   

   View Slide

4. WEIRD SWIFT ATX
   •Meets every Thursday
   •6-10 pm
   @dokun24
   

   View Slide

5. WEIRD SWIFT ATX
   •Meets every Thursday
   •6-10 pm
   •Been at BennU coffee so far but…
   @dokun24
   

   View Slide

6. WEIRD SWIFT ATX
   •Meets every Thursday
   •6-10 pm
   •Been at BennU coffee so far but…
   •…next week is at Galvanize!!!
   @dokun24
   

   View Slide

7. What We’ll Cover
   @dokun24
   

   View Slide

8. What We’ll Cover
   • A history of AVFoundation’s input and output
   @dokun24
   

   View Slide

9. What We’ll Cover
   • A history of AVFoundation’s input and output
   • What changed in iOS 10
   @dokun24
   

   View Slide

10. What We’ll Cover
    • A history of AVFoundation’s input and output
    • What changed in iOS 10
    • New methods in iOS 11
    @dokun24
    

    View Slide

11. What We’ll Cover
    • A history of AVFoundation’s input and output
    • What changed in iOS 10
    • New methods in iOS 11
    • Capturing Depth Data
    @dokun24
    

    View Slide

12. What We’ll Cover
    • A history of AVFoundation’s input and output
    • What changed in iOS 10
    • New methods in iOS 11
    • Capturing Depth Data
    • Lumina
    @dokun24
    

    View Slide

13. What We’ll Cover
    • A history of AVFoundation’s input and output
    • What changed in iOS 10
    • New methods in iOS 11
    • Capturing Depth Data
    • Lumina
    • Synchronizing Asynchronously Streaming Data
    @dokun24
    

    View Slide

14. What We Won’t Cover
    @dokun24
    

    View Slide

15. What We Won’t Cover
    • High Efficiency Image File Format (HEIF)
    @dokun24
    

    View Slide

16. What We Won’t Cover
    • High Efficiency Image File Format (HEIF)
    • High Efficiency Video Coding (HEVC)
    @dokun24
    

    View Slide

17. What We Won’t Cover
    • High Efficiency Image File Format (HEIF)
    • High Efficiency Video Coding (HEVC)
    • AVCaptureDepthDataOutput
    @dokun24
    

    View Slide

18. Reference Material
    @dokun24
    

    View Slide

19. Reference Material
    • WWDC 2016, Session 501
    @dokun24
    

    View Slide

20. Reference Material
    • WWDC 2016, Session 501
    • WWDC 2017, Session 507
    @dokun24
    

    View Slide

21. Reference Material
    • WWDC 2016, Session 501
    • WWDC 2017, Session 507
    • https://github.com/dokun1/lumina
    @dokun24
    

    View Slide

22. @dokun24
    

    View Slide

23. AVCaptureDevice
    @dokun24
    

    View Slide

24. AVCaptureDevice
    AVCaptureDeviceInput
    @dokun24
    

    View Slide

25. AVCaptureDevice
    AVCaptureDeviceInput
    AVCaptureSession
    @dokun24
    

    View Slide

26. AVCaptureDevice
    AVCaptureDeviceInput
    AVCaptureConnection
    AVCaptureSession
    AVCapturePhotoOutput
    @dokun24
    

    View Slide

27. AVCaptureDevice
    AVCaptureDeviceInput
    AVCaptureConnection
    AVCaptureConnection
    AVCaptureSession
    AVCapturePhotoOutput
    AVCaptureMovieFileOutput
    @dokun24
    

    View Slide

28. AVCaptureDevice
    AVCaptureDeviceInput
    AVCaptureConnection
    AVCaptureConnection
    AVCaptureConnection
    AVCaptureSession
    AVCapturePhotoOutput
    AVCaptureMovieFileOutput
    AVCaptureMetadataOutput
    @dokun24
    

    View Slide

29. AVCaptureDevice
    AVCaptureDeviceInput
    AVCaptureConnection
    AVCaptureConnection
    AVCaptureConnection
    AVCaptureConnection
    AVCaptureSession
    AVCapturePhotoOutput
    AVCaptureMovieFileOutput
    AVCaptureMetadataOutput
    AVCaptureVideoDataOutput
    @dokun24
    

    View Slide

30. AVCaptureDevice
    AVCaptureDeviceInput
    AVCaptureConnection
    AVCaptureConnection
    AVCaptureConnection
    AVCaptureConnection
    AVCaptureSession
    AVCapturePhotoOutput
    AVCaptureMovieFileOutput
    AVCaptureMetadataOutput
    AVCaptureVideoDataOutput
    AVCaptureVideoPreviewLayer
    @dokun24
    

    View Slide

31. Before iOS 10
    @dokun24
    

    View Slide

32. Before iOS 10
    AVCaptureStillImageOutput
    @dokun24
    

    View Slide

33. Before iOS 10
    AVCaptureStillImageOutput
    func captureStillImageAsynchronously(from connection:
    AVCaptureConnection, completionHandler handler: @escaping
    (CMSampleBuffer?, Error?) -> Swift.Void)
    @dokun24
    

    View Slide

34. Since iOS 10
    @dokun24
    

    View Slide

35. Since iOS 10
    AVCapturePhotoOutput
    @dokun24
    

    View Slide

36. Since iOS 10
    AVCapturePhotoOutput
    var settings = AVCapturePhotoSettings()
    self.photoOutput.capturePhoto(with: settings, delegate: self)
    @dokun24
    

    View Slide

37. Since iOS 10
    @dokun24
    

    View Slide

38. Since iOS 10
    AVCapturePhotoSettings
    @dokun24
    

    View Slide

39. Since iOS 10
    AVCapturePhotoSettings
    var uniqueID: Int64 // generated on init
    @dokun24
    

    View Slide

40. Since iOS 10
    AVCapturePhotoSettings
    var uniqueID: Int64 // generated on init
    var flashMode: AVCaptureDevice.FlashMode
    @dokun24
    

    View Slide

41. Since iOS 10
    AVCapturePhotoSettings
    var uniqueID: Int64 // generated on init
    var flashMode: AVCaptureDevice.FlashMode
    var isAutoStillImageStabilizationEnabled: Bool
    @dokun24
    

    View Slide

42. Since iOS 10
    AVCapturePhotoSettings
    var uniqueID: Int64 // generated on init
    var flashMode: AVCaptureDevice.FlashMode
    var isAutoStillImageStabilizationEnabled: Bool
    var isHighResolutionPhotoEnabled: Bool
    @dokun24
    

    View Slide

43. Since iOS 10
    AVCapturePhotoSettings
    var uniqueID: Int64 // generated on init
    var flashMode: AVCaptureDevice.FlashMode
    var isAutoStillImageStabilizationEnabled: Bool
    var isHighResolutionPhotoEnabled: Bool
    var isAutoDualCameraFusionEnabled: Bool // iOS 10.2+
    @dokun24
    

    View Slide

44. Since iOS 10
    @dokun24
    

    View Slide

45. Since iOS 10
    AVCapturePhotoOutputDelegate
    @dokun24
    

    View Slide

46. Since iOS 10
    AVCapturePhotoOutputDelegate
    func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    @dokun24
    

    View Slide

47. Since iOS 10
    AVCapturePhotoOutputDelegate
    func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    func photoOutput(_ output: AVCapturePhotoOutput, willCapturePhotoFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    @dokun24
    

    View Slide

48. Since iOS 10
    AVCapturePhotoOutputDelegate
    func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    func photoOutput(_ output: AVCapturePhotoOutput, willCapturePhotoFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    func photoOutput(_ output: AVCapturePhotoOutput, didCapturePhotoFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    @dokun24
    

    View Slide

49. Since iOS 10
    AVCapturePhotoOutputDelegate
    func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photoSampleBuffer:
    CMSampleBuffer?, previewPhoto previewPhotoSampleBuffer: CMSampleBuffer?, resolvedSettings:
    AVCaptureResolvedPhotoSettings, bracketSettings: AVCaptureBracketedStillImageSettings?, error: Error?)
    func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    func photoOutput(_ output: AVCapturePhotoOutput, willCapturePhotoFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    func photoOutput(_ output: AVCapturePhotoOutput, didCapturePhotoFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    @dokun24
    

    View Slide

50. Since iOS 10
    AVCapturePhotoOutputDelegate
    func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photoSampleBuffer:
    CMSampleBuffer?, previewPhoto previewPhotoSampleBuffer: CMSampleBuffer?, resolvedSettings:
    AVCaptureResolvedPhotoSettings, bracketSettings: AVCaptureBracketedStillImageSettings?, error: Error?)
    func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    func photoOutput(_ output: AVCapturePhotoOutput, willCapturePhotoFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    func photoOutput(_ output: AVCapturePhotoOutput, didCapturePhotoFor resolvedSettings:
    AVCaptureResolvedPhotoSettings)
    func photoOutput(_ output: AVCapturePhotoOutput, didFinishCaptureFor resolvedSettings:
    AVCaptureResolvedPhotoSettings, error: Error?)
    @dokun24
    

    View Slide

51. Since iOS 10
    @dokun24
    

    View Slide

52. Since iOS 10
    AVCaptureResolvedPhotoSettings
    @dokun24
    

    View Slide

53. Since iOS 10
    AVCaptureResolvedPhotoSettings
    var uniqueID: Int64
    @dokun24
    

    View Slide

54. Since iOS 10
    AVCaptureResolvedPhotoSettings
    var uniqueID: Int64
    var photoDimensions: CMVideoDimensions
    @dokun24
    

    View Slide

55. Since iOS 10
    AVCaptureResolvedPhotoSettings
    var uniqueID: Int64
    var photoDimensions: CMVideoDimensions
    var livePhotoMovieDimensions: CMVideoDimensions
    @dokun24
    

    View Slide

56. Since iOS 10
    AVCaptureResolvedPhotoSettings
    var uniqueID: Int64
    var photoDimensions: CMVideoDimensions
    var livePhotoMovieDimensions: CMVideoDimensions
    var isFlashEnabled: Bool
    @dokun24
    

    View Slide

57. Since iOS 10
    AVCaptureResolvedPhotoSettings
    var uniqueID: Int64
    var photoDimensions: CMVideoDimensions
    var livePhotoMovieDimensions: CMVideoDimensions
    var isFlashEnabled: Bool
    var isStillImageStabilizationEnabled: Bool
    @dokun24
    

    View Slide

58. Since iOS 10
    AVCaptureResolvedPhotoSettings
    var uniqueID: Int64
    var photoDimensions: CMVideoDimensions
    var livePhotoMovieDimensions: CMVideoDimensions
    var isFlashEnabled: Bool
    var isStillImageStabilizationEnabled: Bool
    var isDualCameraFusionEnabled: Bool // iOS 10.2+
    @dokun24
    

    View Slide

59. New in iOS 11
    @dokun24
    

    View Slide

60. New in iOS 11
    AVCapturePhotoOutputDelegate
    @dokun24
    

    View Slide

61. New in iOS 11
    AVCapturePhotoOutputDelegate
    func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto
    photo: AVCapturePhoto, error: Error?)
    @dokun24
    

    View Slide

62. New in iOS 11
    @dokun24
    

    View Slide

63. New in iOS 11
    AVCapturePhoto
    @dokun24
    

    View Slide

64. New in iOS 11
    AVCapturePhoto
    var pixelBuffer: CVPixelBuffer?
    @dokun24
    

    View Slide

65. New in iOS 11
    AVCapturePhoto
    var pixelBuffer: CVPixelBuffer?
    var previewPixelBuffer: CVPixelBuffer?
    @dokun24
    

    View Slide

66. New in iOS 11
    AVCapturePhoto
    var pixelBuffer: CVPixelBuffer?
    var previewPixelBuffer: CVPixelBuffer?
    var metadata: [String : Any]
    @dokun24
    

    View Slide

67. New in iOS 11
    AVCapturePhoto
    var pixelBuffer: CVPixelBuffer?
    var previewPixelBuffer: CVPixelBuffer?
    var metadata: [String : Any]
    var resolvedSettings: AVCaptureResolvedPhotoSettings
    @dokun24
    

    View Slide

68. New in iOS 11
    AVCapturePhoto
    var pixelBuffer: CVPixelBuffer?
    var previewPixelBuffer: CVPixelBuffer?
    var metadata: [String : Any]
    var resolvedSettings: AVCaptureResolvedPhotoSettings
    var photoCount: Int
    @dokun24
    

    View Slide

69. New in iOS 11
    AVCapturePhoto
    var pixelBuffer: CVPixelBuffer?
    var previewPixelBuffer: CVPixelBuffer?
    var metadata: [String : Any]
    var resolvedSettings: AVCaptureResolvedPhotoSettings
    var photoCount: Int
    var depthData: AVDepthData?
    @dokun24
    

    View Slide

70. New in iOS 11
    AVCapturePhoto
    @dokun24
    

    View Slide

71. New in iOS 11
    AVCapturePhoto
    @dokun24
    

    View Slide

72. New in iOS 11
    AVCapturePhoto
    func cgImageRepresentation() -> Unmanaged?
    @dokun24
    

    View Slide

73. New in iOS 11
    AVCapturePhoto
    func cgImageRepresentation() -> Unmanaged?
    func takeRetainedValue() -> Instance
    @dokun24
    

    View Slide

74. New in iOS 11
    AVCapturePhoto
    func cgImageRepresentation() -> Unmanaged?
    func takeRetainedValue() -> Instance
    func takeUnretainedValue() -> Instance
    @dokun24
    

    View Slide

75. Back to iOS 10
    @dokun24
    

    View Slide

76. Back to iOS 10
    AVCapturePhotoOutputDelegate -> CMSampleBuffer -> UIImage
    @dokun24
    

    View Slide

77. Back to iOS 10
    AVCapturePhotoOutputDelegate -> CMSampleBuffer -> UIImage
    guard let imageData = AVCapturePhotoOutput.jpegPhotoDataRepresentation(forJPEGSampleBuffer:
    sampleBuffer, previewPhotoSampleBuffer: nil) else {
    return nil
    }
    guard let dataProvider = CGDataProvider(data: imageData as CFData) else {
    return nil
    }
    guard let cgImage = CGImage(jpegDataProviderSource: dataProvider, decode: nil,
    shouldInterpolate: true, intent: CGColorRenderingIntent.defaultIntent) else {
    return nil
    }
    return UIImage(cgImage: cgImage)
    @dokun24
    

    View Slide

78. New in iOS 11
    @dokun24
    

    View Slide

79. New in iOS 11
    AVCapturePhoto -> UIImage
    @dokun24
    

    View Slide

80. New in iOS 11
    AVCapturePhoto -> UIImage
    guard let cgImage = photo.cgImageRepresentation() else {
    return nil
    }
    return UIImage(cgImage: cgImage.takeUnretainedValue())
    @dokun24
    

    View Slide

81. Capturing Depth Data
    @dokun24
    

    View Slide

82. Capturing Depth Data
    @dokun24
    

    View Slide

83. Capturing Depth Data
    @dokun24
    

    View Slide

84. New in iOS 11
    AVCapturePhoto
    @dokun24
    var pixelBuffer: CVPixelBuffer?
    var previewPixelBuffer: CVPixelBuffer?
    var metadata: [String : Any]
    var resolvedSettings: AVCaptureResolvedPhotoSettings
    var photoCount: Int
    var depthData: AVDepthData?
    

    View Slide

85. New in iOS 11
    AVCapturePhoto
    @dokun24
    var pixelBuffer: CVPixelBuffer?
    var previewPixelBuffer: CVPixelBuffer?
    var metadata: [String : Any]
    var resolvedSettings: AVCaptureResolvedPhotoSettings
    var photoCount: Int
    var depthData: AVDepthData?
    

    View Slide

86. Capturing Depth Data
    @dokun24
    

    View Slide

87. Capturing Depth Data
    AVDepthData
    @dokun24
    

    View Slide

88. Capturing Depth Data
    AVDepthData
    var depthDataAccuracy: AVDepthData.Accuracy
    @dokun24
    

    View Slide

89. Capturing Depth Data
    AVDepthData
    var depthDataAccuracy: AVDepthData.Accuracy
    var isDepthDataFiltered: Bool
    @dokun24
    

    View Slide

90. Capturing Depth Data
    AVDepthData
    var depthDataAccuracy: AVDepthData.Accuracy
    var isDepthDataFiltered: Bool
    var depthDataQuality: AVDepthData.Quality
    @dokun24
    

    View Slide

91. Capturing Depth Data
    AVDepthData
    var depthDataAccuracy: AVDepthData.Accuracy
    var isDepthDataFiltered: Bool
    var depthDataQuality: AVDepthData.Quality
    var cameraCalibrationData: AVCameraCalibrationData?
    @dokun24
    

    View Slide

92. Capturing Depth Data
    AVDepthData
    var depthDataAccuracy: AVDepthData.Accuracy
    var isDepthDataFiltered: Bool
    var depthDataQuality: AVDepthData.Quality
    var cameraCalibrationData: AVCameraCalibrationData?
    var depthDataMap: CVPixelBuffer
    @dokun24
    

    View Slide

93. Capturing Depth Data
    @dokun24
    

    View Slide

94. Capturing Depth Data
    // in your capture session configuration
    if self.photoOutput.isDepthDataDeliverySupported {
    self.photoOutput.isDepthDataDeliveryEnabled = true
    }
    @dokun24
    

    View Slide

95. Capturing Depth Data
    // in your capture session configuration
    if self.photoOutput.isDepthDataDeliverySupported {
    self.photoOutput.isDepthDataDeliveryEnabled = true
    }
    // in your preparation for capturing a still image with AVCapturePhotoSettings
    if self.photoOutput.isDepthDataDeliverySupported {
    settings.isDepthDataDeliveryEnabled = true
    }
    @dokun24
    

    View Slide

96. Lumina
    @dokun24
    

    View Slide

97. Lumina
    • Gratuitous plugging!
    @dokun24
    

    View Slide

98. Lumina
    • Gratuitous plugging!
    • A full service camera framework
    @dokun24
    

    View Slide

99. Lumina
    • Gratuitous plugging!
    • A full service camera framework
    • QR codes, bar codes, live photos, the list
    goes on…
    @dokun24
    

    View Slide

100. Lumina
     • Gratuitous plugging!
     • A full service camera framework
     • QR codes, bar codes, live photos, the list
     goes on…
     • …and the easiest CoreML integration you’ll
     see
     @dokun24
     

     View Slide

101. Lumina
     • Gratuitous plugging!
     • A full service camera framework
     • QR codes, bar codes, live photos, the list
     goes on…
     • …and the easiest CoreML integration you’ll
     see
     @dokun24
     

     View Slide

102. Live Demo Of Lumina
     @dokun24
     

     View Slide

103. Synchronizing Asynchronously Streaming Data
     @dokun24
     

     View Slide

104. Synchronizing Asynchronously Streaming Data
     • AVCaptureVideoDataOutput -> video frames
     @dokun24
     

     View Slide

105. Synchronizing Asynchronously Streaming Data
     • AVCaptureVideoDataOutput -> video frames
     • AVCaptureMetadataOutput -> QR codes, barcodes, faces
     @dokun24
     

     View Slide

106. Synchronizing Asynchronously Streaming Data
     • AVCaptureVideoDataOutput -> video frames
     • AVCaptureMetadataOutput -> QR codes, barcodes, faces
     • AVCapturePhotoOutput -> still photos
     @dokun24
     

     View Slide

107. Synchronizing Asynchronously Streaming Data
     • AVCaptureVideoDataOutput -> video frames
     • AVCaptureMetadataOutput -> QR codes, barcodes, faces
     • AVCapturePhotoOutput -> still photos
     • AVCaptureDepthDataOutput -> depth data maps
     @dokun24
     

     View Slide

108. Synchronizing Asynchronously Streaming Data
     • AVCaptureVideoDataOutput -> video frames
     • AVCaptureMetadataOutput -> QR codes, barcodes, faces
     • AVCapturePhotoOutput -> still photos
     • AVCaptureDepthDataOutput -> depth data maps
     • You see the problem here, right?
     @dokun24
     

     View Slide

109. Synchronizing Asynchronously Streaming Data
     @dokun24
     

     View Slide

110. Synchronizing Asynchronously Streaming Data
     AVCaptureDataOutputSynchronizer // iOS 11.0 +
     @dokun24
     

     View Slide

111. Synchronizing Asynchronously Streaming Data
     AVCaptureDataOutputSynchronizer // iOS 11.0 +
     init(dataOutputs: [AVCaptureOutput])
     @dokun24
     

     View Slide

112. Synchronizing Asynchronously Streaming Data
     AVCaptureDataOutputSynchronizer // iOS 11.0 +
     init(dataOutputs: [AVCaptureOutput])
     var delegate: AVCaptureDataOutputSynchronizerDelegate
     @dokun24
     

     View Slide

113. Synchronizing Asynchronously Streaming Data
     AVCaptureDataOutputSynchronizer // iOS 11.0 +
     init(dataOutputs: [AVCaptureOutput])
     var delegate: AVCaptureDataOutputSynchronizerDelegate
     var delegateCallbackQueue: DispatchQueue?
     @dokun24
     

     View Slide

114. Synchronizing Asynchronously Streaming Data
     @dokun24
     

     View Slide

115. Synchronizing Asynchronously Streaming Data
     AVCaptureDataOutputSynchronizerDelegate
     @dokun24
     

     View Slide

116. Synchronizing Asynchronously Streaming Data
     AVCaptureDataOutputSynchronizerDelegate
     func dataOutputSynchronizer(_ synchronizer:
     AVCaptureDataOutputSynchronizer, didOutput synchronizedDataCollection:
     AVCaptureSynchronizedDataCollection) {
     let collection = synchronizedDataCollection
     }
     @dokun24
     

     View Slide

117. Synchronizing Asynchronously Streaming Data
     @dokun24
     

     View Slide

118. Synchronizing Asynchronously Streaming Data
     AVCaptureSynchronizedDataCollection // iOS 11.0 +
     @dokun24
     

     View Slide

119. Synchronizing Asynchronously Streaming Data
     AVCaptureSynchronizedDataCollection // iOS 11.0 +
     func synchronizedData(for captureOutput: AVCaptureOutput) ->
     AVCaptureSynchronizedData?
     @dokun24
     

     View Slide

120. Synchronizing Asynchronously Streaming Data
     AVCaptureSynchronizedDataCollection // iOS 11.0 +
     func synchronizedData(for captureOutput: AVCaptureOutput) ->
     AVCaptureSynchronizedData?
     @dokun24
     var count: Int
     

     View Slide

121. Questions?
     @dokun24
     

     View Slide