Share
Embed
AVFoundation, iOS 11 & Lumina
Come learn about what’s new in iOS 11 with AVFoundation, as David Okun walks us through new ways to capture still images, live photos, and depth maps via his open source framework, Lumina.
David Okun
November 07, 2017
Transcript
121 slides
-
WEIRD SWIFT ATX •Meets every Thursday •6-10 pm •Been at
BennU coffee so far but… @dokun24 -
WEIRD SWIFT ATX •Meets every Thursday •6-10 pm •Been at
BennU coffee so far but… •…next week is at Galvanize!!! @dokun24 -
What We’ll Cover • A history of AVFoundation’s input and
output @dokun24 -
What We’ll Cover • A history of AVFoundation’s input and
output • What changed in iOS 10 @dokun24 -
What We’ll Cover • A history of AVFoundation’s input and
output • What changed in iOS 10 • New methods in iOS 11 @dokun24 -
What We’ll Cover • A history of AVFoundation’s input and
output • What changed in iOS 10 • New methods in iOS 11 • Capturing Depth Data @dokun24 -
What We’ll Cover • A history of AVFoundation’s input and
output • What changed in iOS 10 • New methods in iOS 11 • Capturing Depth Data • Lumina @dokun24 -
What We’ll Cover • A history of AVFoundation’s input and
output • What changed in iOS 10 • New methods in iOS 11 • Capturing Depth Data • Lumina • Synchronizing Asynchronously Streaming Data @dokun24 -
What We Won’t Cover • High Efficiency Image File Format
(HEIF) @dokun24 -
What We Won’t Cover • High Efficiency Image File Format
(HEIF) • High Efficiency Video Coding (HEVC) @dokun24 -
What We Won’t Cover • High Efficiency Image File Format
(HEIF) • High Efficiency Video Coding (HEVC) • AVCaptureDepthDataOutput @dokun24 -
Reference Material • WWDC 2016, Session 501 • WWDC 2017,
Session 507 @dokun24 -
Reference Material • WWDC 2016, Session 501 • WWDC 2017,
Session 507 • https://github.com/dokun1/lumina @dokun24 -
AVCaptureDevice AVCaptureDeviceInput AVCaptureConnection AVCaptureConnection AVCaptureConnection AVCaptureConnection AVCaptureSession AVCapturePhotoOutput AVCaptureMovieFileOutput AVCaptureMetadataOutput
AVCaptureVideoDataOutput AVCaptureVideoPreviewLayer @dokun24 -
Before iOS 10 AVCaptureStillImageOutput func captureStillImageAsynchronously(from connection: AVCaptureConnection, completionHandler handler:
@escaping (CMSampleBuffer?, Error?) -> Swift.Void) @dokun24 -
Since iOS 10 AVCapturePhotoSettings var uniqueID: Int64 // generated on
init var flashMode: AVCaptureDevice.FlashMode @dokun24 -
Since iOS 10 AVCapturePhotoSettings var uniqueID: Int64 // generated on
init var flashMode: AVCaptureDevice.FlashMode var isAutoStillImageStabilizationEnabled: Bool @dokun24 -
Since iOS 10 AVCapturePhotoSettings var uniqueID: Int64 // generated on
init var flashMode: AVCaptureDevice.FlashMode var isAutoStillImageStabilizationEnabled: Bool var isHighResolutionPhotoEnabled: Bool @dokun24 -
Since iOS 10 AVCapturePhotoSettings var uniqueID: Int64 // generated on
init var flashMode: AVCaptureDevice.FlashMode var isAutoStillImageStabilizationEnabled: Bool var isHighResolutionPhotoEnabled: Bool var isAutoDualCameraFusionEnabled: Bool // iOS 10.2+ @dokun24 -
Since iOS 10 AVCapturePhotoOutputDelegate func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings:
AVCaptureResolvedPhotoSettings) @dokun24 -
Since iOS 10 AVCapturePhotoOutputDelegate func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings:
AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, willCapturePhotoFor resolvedSettings: AVCaptureResolvedPhotoSettings) @dokun24 -
Since iOS 10 AVCapturePhotoOutputDelegate func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings:
AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, willCapturePhotoFor resolvedSettings: AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, didCapturePhotoFor resolvedSettings: AVCaptureResolvedPhotoSettings) @dokun24 -
Since iOS 10 AVCapturePhotoOutputDelegate func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photoSampleBuffer:
CMSampleBuffer?, previewPhoto previewPhotoSampleBuffer: CMSampleBuffer?, resolvedSettings: AVCaptureResolvedPhotoSettings, bracketSettings: AVCaptureBracketedStillImageSettings?, error: Error?) func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings: AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, willCapturePhotoFor resolvedSettings: AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, didCapturePhotoFor resolvedSettings: AVCaptureResolvedPhotoSettings) @dokun24 -
Since iOS 10 AVCapturePhotoOutputDelegate func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photoSampleBuffer:
CMSampleBuffer?, previewPhoto previewPhotoSampleBuffer: CMSampleBuffer?, resolvedSettings: AVCaptureResolvedPhotoSettings, bracketSettings: AVCaptureBracketedStillImageSettings?, error: Error?) func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings: AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, willCapturePhotoFor resolvedSettings: AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, didCapturePhotoFor resolvedSettings: AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, didFinishCaptureFor resolvedSettings: AVCaptureResolvedPhotoSettings, error: Error?) @dokun24 -
Since iOS 10 AVCaptureResolvedPhotoSettings var uniqueID: Int64 var photoDimensions: CMVideoDimensions
var livePhotoMovieDimensions: CMVideoDimensions @dokun24 -
Since iOS 10 AVCaptureResolvedPhotoSettings var uniqueID: Int64 var photoDimensions: CMVideoDimensions
var livePhotoMovieDimensions: CMVideoDimensions var isFlashEnabled: Bool @dokun24 -
Since iOS 10 AVCaptureResolvedPhotoSettings var uniqueID: Int64 var photoDimensions: CMVideoDimensions
var livePhotoMovieDimensions: CMVideoDimensions var isFlashEnabled: Bool var isStillImageStabilizationEnabled: Bool @dokun24 -
Since iOS 10 AVCaptureResolvedPhotoSettings var uniqueID: Int64 var photoDimensions: CMVideoDimensions
var livePhotoMovieDimensions: CMVideoDimensions var isFlashEnabled: Bool var isStillImageStabilizationEnabled: Bool var isDualCameraFusionEnabled: Bool // iOS 10.2+ @dokun24 -
New in iOS 11 AVCapturePhotoOutputDelegate func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto
photo: AVCapturePhoto, error: Error?) @dokun24 -
New in iOS 11 AVCapturePhoto var pixelBuffer: CVPixelBuffer? var previewPixelBuffer:
CVPixelBuffer? @dokun24 -
New in iOS 11 AVCapturePhoto var pixelBuffer: CVPixelBuffer? var previewPixelBuffer:
CVPixelBuffer? var metadata: [String : Any] @dokun24 -
New in iOS 11 AVCapturePhoto var pixelBuffer: CVPixelBuffer? var previewPixelBuffer:
CVPixelBuffer? var metadata: [String : Any] var resolvedSettings: AVCaptureResolvedPhotoSettings @dokun24 -
New in iOS 11 AVCapturePhoto var pixelBuffer: CVPixelBuffer? var previewPixelBuffer:
CVPixelBuffer? var metadata: [String : Any] var resolvedSettings: AVCaptureResolvedPhotoSettings var photoCount: Int @dokun24 -
New in iOS 11 AVCapturePhoto var pixelBuffer: CVPixelBuffer? var previewPixelBuffer:
CVPixelBuffer? var metadata: [String : Any] var resolvedSettings: AVCaptureResolvedPhotoSettings var photoCount: Int var depthData: AVDepthData? @dokun24 -
New in iOS 11 AVCapturePhoto func cgImageRepresentation() -> Unmanaged<CGImage>? func
takeRetainedValue() -> Instance @dokun24 -
New in iOS 11 AVCapturePhoto func cgImageRepresentation() -> Unmanaged<CGImage>? func
takeRetainedValue() -> Instance func takeUnretainedValue() -> Instance @dokun24 -
Back to iOS 10 AVCapturePhotoOutputDelegate -> CMSampleBuffer -> UIImage guard
let imageData = AVCapturePhotoOutput.jpegPhotoDataRepresentation(forJPEGSampleBuffer: sampleBuffer, previewPhotoSampleBuffer: nil) else { return nil } guard let dataProvider = CGDataProvider(data: imageData as CFData) else { return nil } guard let cgImage = CGImage(jpegDataProviderSource: dataProvider, decode: nil, shouldInterpolate: true, intent: CGColorRenderingIntent.defaultIntent) else { return nil } return UIImage(cgImage: cgImage) @dokun24 -
New in iOS 11 AVCapturePhoto -> UIImage guard let cgImage
= photo.cgImageRepresentation() else { return nil } return UIImage(cgImage: cgImage.takeUnretainedValue()) @dokun24 -
New in iOS 11 AVCapturePhoto @dokun24 var pixelBuffer: CVPixelBuffer? var
previewPixelBuffer: CVPixelBuffer? var metadata: [String : Any] var resolvedSettings: AVCaptureResolvedPhotoSettings var photoCount: Int var depthData: AVDepthData? -
New in iOS 11 AVCapturePhoto @dokun24 var pixelBuffer: CVPixelBuffer? var
previewPixelBuffer: CVPixelBuffer? var metadata: [String : Any] var resolvedSettings: AVCaptureResolvedPhotoSettings var photoCount: Int var depthData: AVDepthData? -
Capturing Depth Data AVDepthData var depthDataAccuracy: AVDepthData.Accuracy var isDepthDataFiltered: Bool
var depthDataQuality: AVDepthData.Quality @dokun24 -
Capturing Depth Data AVDepthData var depthDataAccuracy: AVDepthData.Accuracy var isDepthDataFiltered: Bool
var depthDataQuality: AVDepthData.Quality var cameraCalibrationData: AVCameraCalibrationData? @dokun24 -
Capturing Depth Data AVDepthData var depthDataAccuracy: AVDepthData.Accuracy var isDepthDataFiltered: Bool
var depthDataQuality: AVDepthData.Quality var cameraCalibrationData: AVCameraCalibrationData? var depthDataMap: CVPixelBuffer @dokun24 -
Capturing Depth Data // in your capture session configuration if
self.photoOutput.isDepthDataDeliverySupported { self.photoOutput.isDepthDataDeliveryEnabled = true } @dokun24 -
Capturing Depth Data // in your capture session configuration if
self.photoOutput.isDepthDataDeliverySupported { self.photoOutput.isDepthDataDeliveryEnabled = true } // in your preparation for capturing a still image with AVCapturePhotoSettings if self.photoOutput.isDepthDataDeliverySupported { settings.isDepthDataDeliveryEnabled = true } @dokun24 -
Lumina • Gratuitous plugging! • A full service camera framework
• QR codes, bar codes, live photos, the list goes on… @dokun24 -
Lumina • Gratuitous plugging! • A full service camera framework
• QR codes, bar codes, live photos, the list goes on… • …and the easiest CoreML integration you’ll see @dokun24 -
Lumina • Gratuitous plugging! • A full service camera framework
• QR codes, bar codes, live photos, the list goes on… • …and the easiest CoreML integration you’ll see @dokun24 -
Synchronizing Asynchronously Streaming Data • AVCaptureVideoDataOutput -> video frames •
AVCaptureMetadataOutput -> QR codes, barcodes, faces @dokun24 -
Synchronizing Asynchronously Streaming Data • AVCaptureVideoDataOutput -> video frames •
AVCaptureMetadataOutput -> QR codes, barcodes, faces • AVCapturePhotoOutput -> still photos @dokun24 -
Synchronizing Asynchronously Streaming Data • AVCaptureVideoDataOutput -> video frames •
AVCaptureMetadataOutput -> QR codes, barcodes, faces • AVCapturePhotoOutput -> still photos • AVCaptureDepthDataOutput -> depth data maps @dokun24 -
Synchronizing Asynchronously Streaming Data • AVCaptureVideoDataOutput -> video frames •
AVCaptureMetadataOutput -> QR codes, barcodes, faces • AVCapturePhotoOutput -> still photos • AVCaptureDepthDataOutput -> depth data maps • You see the problem here, right? @dokun24 -
Synchronizing Asynchronously Streaming Data AVCaptureDataOutputSynchronizer // iOS 11.0 + init(dataOutputs:
[AVCaptureOutput]) @dokun24 -
Synchronizing Asynchronously Streaming Data AVCaptureDataOutputSynchronizer // iOS 11.0 + init(dataOutputs:
[AVCaptureOutput]) var delegate: AVCaptureDataOutputSynchronizerDelegate @dokun24 -
Synchronizing Asynchronously Streaming Data AVCaptureDataOutputSynchronizer // iOS 11.0 + init(dataOutputs:
[AVCaptureOutput]) var delegate: AVCaptureDataOutputSynchronizerDelegate var delegateCallbackQueue: DispatchQueue? @dokun24 -
Synchronizing Asynchronously Streaming Data AVCaptureDataOutputSynchronizerDelegate func dataOutputSynchronizer(_ synchronizer: AVCaptureDataOutputSynchronizer, didOutput
synchronizedDataCollection: AVCaptureSynchronizedDataCollection) { let collection = synchronizedDataCollection } @dokun24 -
Synchronizing Asynchronously Streaming Data AVCaptureSynchronizedDataCollection // iOS 11.0 + func
synchronizedData(for captureOutput: AVCaptureOutput) -> AVCaptureSynchronizedData? @dokun24 -
Synchronizing Asynchronously Streaming Data AVCaptureSynchronizedDataCollection // iOS 11.0 + func
synchronizedData(for captureOutput: AVCaptureOutput) -> AVCaptureSynchronizedData? @dokun24 var count: Int