AVFoundation, iOS 11 & Lumina

Come learn about what’s new in iOS 11 with AVFoundation, as David Okun walks us through new ways to capture still images, live photos, and depth maps via his open source framework, Lumina.

Transcript

1. AVFoundation, iOS 11 & Lumina David Okun @dokun24
2. WEIRD SWIFT ATX @dokun24
3. WEIRD SWIFT ATX •Meets every Thursday @dokun24
4. WEIRD SWIFT ATX •Meets every Thursday •6-10 pm @dokun24
5. WEIRD SWIFT ATX •Meets every Thursday •6-10 pm •Been at BennU coffee so far but… @dokun24
6. WEIRD SWIFT ATX •Meets every Thursday •6-10 pm •Been at BennU coffee so far but… •…next week is at Galvanize!!! @dokun24
7. What We’ll Cover @dokun24
8. What We’ll Cover • A history of AVFoundation’s input and output @dokun24
9. What We’ll Cover • A history of AVFoundation’s input and output • What changed in iOS 10 @dokun24
10. What We’ll Cover • A history of AVFoundation’s input and output • What changed in iOS 10 • New methods in iOS 11 @dokun24
11. What We’ll Cover • A history of AVFoundation’s input and output • What changed in iOS 10 • New methods in iOS 11 • Capturing Depth Data @dokun24
12. What We’ll Cover • A history of AVFoundation’s input and output • What changed in iOS 10 • New methods in iOS 11 • Capturing Depth Data • Lumina @dokun24
13. What We’ll Cover • A history of AVFoundation’s input and output • What changed in iOS 10 • New methods in iOS 11 • Capturing Depth Data • Lumina • Synchronizing Asynchronously Streaming Data @dokun24
14. What We Won’t Cover @dokun24
15. What We Won’t Cover • High Efficiency Image File Format (HEIF) @dokun24
16. What We Won’t Cover • High Efficiency Image File Format (HEIF) • High Efficiency Video Coding (HEVC) @dokun24
17. What We Won’t Cover • High Efficiency Image File Format (HEIF) • High Efficiency Video Coding (HEVC) • AVCaptureDepthDataOutput @dokun24
18. Reference Material @dokun24
19. Reference Material • WWDC 2016, Session 501 @dokun24
20. Reference Material • WWDC 2016, Session 501 • WWDC 2017, Session 507 @dokun24
21. Reference Material • WWDC 2016, Session 501 • WWDC 2017, Session 507 • https://github.com/dokun1/lumina @dokun24
22. @dokun24
23. AVCaptureDevice @dokun24
24. AVCaptureDevice AVCaptureDeviceInput @dokun24
25. AVCaptureDevice AVCaptureDeviceInput AVCaptureSession @dokun24
26. AVCaptureDevice AVCaptureDeviceInput AVCaptureConnection AVCaptureSession AVCapturePhotoOutput @dokun24
27. AVCaptureDevice AVCaptureDeviceInput AVCaptureConnection AVCaptureConnection AVCaptureSession AVCapturePhotoOutput AVCaptureMovieFileOutput @dokun24
28. AVCaptureDevice AVCaptureDeviceInput AVCaptureConnection AVCaptureConnection AVCaptureConnection AVCaptureSession AVCapturePhotoOutput AVCaptureMovieFileOutput AVCaptureMetadataOutput @dokun24
29. AVCaptureDevice AVCaptureDeviceInput AVCaptureConnection AVCaptureConnection AVCaptureConnection AVCaptureConnection AVCaptureSession AVCapturePhotoOutput AVCaptureMovieFileOutput AVCaptureMetadataOutput AVCaptureVideoDataOutput @dokun24
30. AVCaptureDevice AVCaptureDeviceInput AVCaptureConnection AVCaptureConnection AVCaptureConnection AVCaptureConnection AVCaptureSession AVCapturePhotoOutput AVCaptureMovieFileOutput AVCaptureMetadataOutput AVCaptureVideoDataOutput AVCaptureVideoPreviewLayer @dokun24
31. Before iOS 10 @dokun24
32. Before iOS 10 AVCaptureStillImageOutput @dokun24
33. Before iOS 10 AVCaptureStillImageOutput func captureStillImageAsynchronously(from connection: AVCaptureConnection, completionHandler handler: @escaping (CMSampleBuffer?, Error?) -> Swift.Void) @dokun24
34. Since iOS 10 @dokun24
35. Since iOS 10 AVCapturePhotoOutput @dokun24
36. Since iOS 10 AVCapturePhotoOutput var settings = AVCapturePhotoSettings() self.photoOutput.capturePhoto(with: settings, delegate: self) @dokun24
37. Since iOS 10 @dokun24
38. Since iOS 10 AVCapturePhotoSettings @dokun24
39. Since iOS 10 AVCapturePhotoSettings var uniqueID: Int64 // generated on init @dokun24
40. Since iOS 10 AVCapturePhotoSettings var uniqueID: Int64 // generated on init var flashMode: AVCaptureDevice.FlashMode @dokun24
41. Since iOS 10 AVCapturePhotoSettings var uniqueID: Int64 // generated on init var flashMode: AVCaptureDevice.FlashMode var isAutoStillImageStabilizationEnabled: Bool @dokun24
42. Since iOS 10 AVCapturePhotoSettings var uniqueID: Int64 // generated on init var flashMode: AVCaptureDevice.FlashMode var isAutoStillImageStabilizationEnabled: Bool var isHighResolutionPhotoEnabled: Bool @dokun24
43. Since iOS 10 AVCapturePhotoSettings var uniqueID: Int64 // generated on init var flashMode: AVCaptureDevice.FlashMode var isAutoStillImageStabilizationEnabled: Bool var isHighResolutionPhotoEnabled: Bool var isAutoDualCameraFusionEnabled: Bool // iOS 10.2+ @dokun24
44. Since iOS 10 @dokun24
45. Since iOS 10 AVCapturePhotoOutputDelegate @dokun24
46. Since iOS 10 AVCapturePhotoOutputDelegate func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings: AVCaptureResolvedPhotoSettings) @dokun24
47. Since iOS 10 AVCapturePhotoOutputDelegate func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings: AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, willCapturePhotoFor resolvedSettings: AVCaptureResolvedPhotoSettings) @dokun24
48. Since iOS 10 AVCapturePhotoOutputDelegate func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings: AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, willCapturePhotoFor resolvedSettings: AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, didCapturePhotoFor resolvedSettings: AVCaptureResolvedPhotoSettings) @dokun24
49. Since iOS 10 AVCapturePhotoOutputDelegate func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photoSampleBuffer: CMSampleBuffer?, previewPhoto previewPhotoSampleBuffer: CMSampleBuffer?, resolvedSettings: AVCaptureResolvedPhotoSettings, bracketSettings: AVCaptureBracketedStillImageSettings?, error: Error?) func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings: AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, willCapturePhotoFor resolvedSettings: AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, didCapturePhotoFor resolvedSettings: AVCaptureResolvedPhotoSettings) @dokun24
50. Since iOS 10 AVCapturePhotoOutputDelegate func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photoSampleBuffer: CMSampleBuffer?, previewPhoto previewPhotoSampleBuffer: CMSampleBuffer?, resolvedSettings: AVCaptureResolvedPhotoSettings, bracketSettings: AVCaptureBracketedStillImageSettings?, error: Error?) func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings: AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, willCapturePhotoFor resolvedSettings: AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, didCapturePhotoFor resolvedSettings: AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, didFinishCaptureFor resolvedSettings: AVCaptureResolvedPhotoSettings, error: Error?) @dokun24
51. Since iOS 10 @dokun24
52. Since iOS 10 AVCaptureResolvedPhotoSettings @dokun24
53. Since iOS 10 AVCaptureResolvedPhotoSettings var uniqueID: Int64 @dokun24
54. Since iOS 10 AVCaptureResolvedPhotoSettings var uniqueID: Int64 var photoDimensions: CMVideoDimensions @dokun24
55. Since iOS 10 AVCaptureResolvedPhotoSettings var uniqueID: Int64 var photoDimensions: CMVideoDimensions var livePhotoMovieDimensions: CMVideoDimensions @dokun24
56. Since iOS 10 AVCaptureResolvedPhotoSettings var uniqueID: Int64 var photoDimensions: CMVideoDimensions var livePhotoMovieDimensions: CMVideoDimensions var isFlashEnabled: Bool @dokun24
57. Since iOS 10 AVCaptureResolvedPhotoSettings var uniqueID: Int64 var photoDimensions: CMVideoDimensions var livePhotoMovieDimensions: CMVideoDimensions var isFlashEnabled: Bool var isStillImageStabilizationEnabled: Bool @dokun24
58. Since iOS 10 AVCaptureResolvedPhotoSettings var uniqueID: Int64 var photoDimensions: CMVideoDimensions var livePhotoMovieDimensions: CMVideoDimensions var isFlashEnabled: Bool var isStillImageStabilizationEnabled: Bool var isDualCameraFusionEnabled: Bool // iOS 10.2+ @dokun24
59. New in iOS 11 @dokun24
60. New in iOS 11 AVCapturePhotoOutputDelegate @dokun24
61. New in iOS 11 AVCapturePhotoOutputDelegate func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) @dokun24
62. New in iOS 11 @dokun24
63. New in iOS 11 AVCapturePhoto @dokun24
64. New in iOS 11 AVCapturePhoto var pixelBuffer: CVPixelBuffer? @dokun24
65. New in iOS 11 AVCapturePhoto var pixelBuffer: CVPixelBuffer? var previewPixelBuffer: CVPixelBuffer? @dokun24
66. New in iOS 11 AVCapturePhoto var pixelBuffer: CVPixelBuffer? var previewPixelBuffer: CVPixelBuffer? var metadata: [String : Any] @dokun24
67. New in iOS 11 AVCapturePhoto var pixelBuffer: CVPixelBuffer? var previewPixelBuffer: CVPixelBuffer? var metadata: [String : Any] var resolvedSettings: AVCaptureResolvedPhotoSettings @dokun24
68. New in iOS 11 AVCapturePhoto var pixelBuffer: CVPixelBuffer? var previewPixelBuffer: CVPixelBuffer? var metadata: [String : Any] var resolvedSettings: AVCaptureResolvedPhotoSettings var photoCount: Int @dokun24
69. New in iOS 11 AVCapturePhoto var pixelBuffer: CVPixelBuffer? var previewPixelBuffer: CVPixelBuffer? var metadata: [String : Any] var resolvedSettings: AVCaptureResolvedPhotoSettings var photoCount: Int var depthData: AVDepthData? @dokun24
70. New in iOS 11 AVCapturePhoto @dokun24
71. New in iOS 11 AVCapturePhoto @dokun24
72. New in iOS 11 AVCapturePhoto func cgImageRepresentation() -> Unmanaged<CGImage>? @dokun24
73. New in iOS 11 AVCapturePhoto func cgImageRepresentation() -> Unmanaged<CGImage>? func takeRetainedValue() -> Instance @dokun24
74. New in iOS 11 AVCapturePhoto func cgImageRepresentation() -> Unmanaged<CGImage>? func takeRetainedValue() -> Instance func takeUnretainedValue() -> Instance @dokun24
75. Back to iOS 10 @dokun24
76. Back to iOS 10 AVCapturePhotoOutputDelegate -> CMSampleBuffer -> UIImage @dokun24
77. Back to iOS 10 AVCapturePhotoOutputDelegate -> CMSampleBuffer -> UIImage guard let imageData = AVCapturePhotoOutput.jpegPhotoDataRepresentation(forJPEGSampleBuffer: sampleBuffer, previewPhotoSampleBuffer: nil) else { return nil } guard let dataProvider = CGDataProvider(data: imageData as CFData) else { return nil } guard let cgImage = CGImage(jpegDataProviderSource: dataProvider, decode: nil, shouldInterpolate: true, intent: CGColorRenderingIntent.defaultIntent) else { return nil } return UIImage(cgImage: cgImage) @dokun24
78. New in iOS 11 @dokun24
79. New in iOS 11 AVCapturePhoto -> UIImage @dokun24
80. New in iOS 11 AVCapturePhoto -> UIImage guard let cgImage = photo.cgImageRepresentation() else { return nil } return UIImage(cgImage: cgImage.takeUnretainedValue()) @dokun24
81. Capturing Depth Data @dokun24
82. Capturing Depth Data @dokun24
83. Capturing Depth Data @dokun24
84. New in iOS 11 AVCapturePhoto @dokun24 var pixelBuffer: CVPixelBuffer? var previewPixelBuffer: CVPixelBuffer? var metadata: [String : Any] var resolvedSettings: AVCaptureResolvedPhotoSettings var photoCount: Int var depthData: AVDepthData?
85. New in iOS 11 AVCapturePhoto @dokun24 var pixelBuffer: CVPixelBuffer? var previewPixelBuffer: CVPixelBuffer? var metadata: [String : Any] var resolvedSettings: AVCaptureResolvedPhotoSettings var photoCount: Int var depthData: AVDepthData?
86. Capturing Depth Data @dokun24
87. Capturing Depth Data AVDepthData @dokun24
88. Capturing Depth Data AVDepthData var depthDataAccuracy: AVDepthData.Accuracy @dokun24
89. Capturing Depth Data AVDepthData var depthDataAccuracy: AVDepthData.Accuracy var isDepthDataFiltered: Bool @dokun24
90. Capturing Depth Data AVDepthData var depthDataAccuracy: AVDepthData.Accuracy var isDepthDataFiltered: Bool var depthDataQuality: AVDepthData.Quality @dokun24
91. Capturing Depth Data AVDepthData var depthDataAccuracy: AVDepthData.Accuracy var isDepthDataFiltered: Bool var depthDataQuality: AVDepthData.Quality var cameraCalibrationData: AVCameraCalibrationData? @dokun24
92. Capturing Depth Data AVDepthData var depthDataAccuracy: AVDepthData.Accuracy var isDepthDataFiltered: Bool var depthDataQuality: AVDepthData.Quality var cameraCalibrationData: AVCameraCalibrationData? var depthDataMap: CVPixelBuffer @dokun24
93. Capturing Depth Data @dokun24
94. Capturing Depth Data // in your capture session configuration if self.photoOutput.isDepthDataDeliverySupported { self.photoOutput.isDepthDataDeliveryEnabled = true } @dokun24
95. Capturing Depth Data // in your capture session configuration if self.photoOutput.isDepthDataDeliverySupported { self.photoOutput.isDepthDataDeliveryEnabled = true } // in your preparation for capturing a still image with AVCapturePhotoSettings if self.photoOutput.isDepthDataDeliverySupported { settings.isDepthDataDeliveryEnabled = true } @dokun24
96. Lumina @dokun24
97. Lumina • Gratuitous plugging! @dokun24
98. Lumina • Gratuitous plugging! • A full service camera framework @dokun24
99. Lumina • Gratuitous plugging! • A full service camera framework • QR codes, bar codes, live photos, the list goes on… @dokun24
100. Lumina • Gratuitous plugging! • A full service camera framework • QR codes, bar codes, live photos, the list goes on… • …and the easiest CoreML integration you’ll see @dokun24
101. Lumina • Gratuitous plugging! • A full service camera framework • QR codes, bar codes, live photos, the list goes on… • …and the easiest CoreML integration you’ll see @dokun24
102. Live Demo Of Lumina @dokun24
103. Synchronizing Asynchronously Streaming Data @dokun24
104. Synchronizing Asynchronously Streaming Data • AVCaptureVideoDataOutput -> video frames @dokun24
105. Synchronizing Asynchronously Streaming Data • AVCaptureVideoDataOutput -> video frames • AVCaptureMetadataOutput -> QR codes, barcodes, faces @dokun24
106. Synchronizing Asynchronously Streaming Data • AVCaptureVideoDataOutput -> video frames • AVCaptureMetadataOutput -> QR codes, barcodes, faces • AVCapturePhotoOutput -> still photos @dokun24
107. Synchronizing Asynchronously Streaming Data • AVCaptureVideoDataOutput -> video frames • AVCaptureMetadataOutput -> QR codes, barcodes, faces • AVCapturePhotoOutput -> still photos • AVCaptureDepthDataOutput -> depth data maps @dokun24
108. Synchronizing Asynchronously Streaming Data • AVCaptureVideoDataOutput -> video frames • AVCaptureMetadataOutput -> QR codes, barcodes, faces • AVCapturePhotoOutput -> still photos • AVCaptureDepthDataOutput -> depth data maps • You see the problem here, right? @dokun24
109. Synchronizing Asynchronously Streaming Data @dokun24
110. Synchronizing Asynchronously Streaming Data AVCaptureDataOutputSynchronizer // iOS 11.0 + @dokun24
111. Synchronizing Asynchronously Streaming Data AVCaptureDataOutputSynchronizer // iOS 11.0 + init(dataOutputs: [AVCaptureOutput]) @dokun24
112. Synchronizing Asynchronously Streaming Data AVCaptureDataOutputSynchronizer // iOS 11.0 + init(dataOutputs: [AVCaptureOutput]) var delegate: AVCaptureDataOutputSynchronizerDelegate @dokun24
113. Synchronizing Asynchronously Streaming Data AVCaptureDataOutputSynchronizer // iOS 11.0 + init(dataOutputs: [AVCaptureOutput]) var delegate: AVCaptureDataOutputSynchronizerDelegate var delegateCallbackQueue: DispatchQueue? @dokun24
114. Synchronizing Asynchronously Streaming Data @dokun24
115. Synchronizing Asynchronously Streaming Data AVCaptureDataOutputSynchronizerDelegate @dokun24
116. Synchronizing Asynchronously Streaming Data AVCaptureDataOutputSynchronizerDelegate func dataOutputSynchronizer(_ synchronizer: AVCaptureDataOutputSynchronizer, didOutput synchronizedDataCollection: AVCaptureSynchronizedDataCollection) { let collection = synchronizedDataCollection } @dokun24
117. Synchronizing Asynchronously Streaming Data @dokun24
118. Synchronizing Asynchronously Streaming Data AVCaptureSynchronizedDataCollection // iOS 11.0 + @dokun24
119. Synchronizing Asynchronously Streaming Data AVCaptureSynchronizedDataCollection // iOS 11.0 + func synchronizedData(for captureOutput: AVCaptureOutput) -> AVCaptureSynchronizedData? @dokun24
120. Synchronizing Asynchronously Streaming Data AVCaptureSynchronizedDataCollection // iOS 11.0 + func synchronizedData(for captureOutput: AVCaptureOutput) -> AVCaptureSynchronizedData? @dokun24 var count: Int
121. Questions? @dokun24