macOSで自分のカメラを作ってみよう - Core Media IO Extensions

Transcript

1. NBD04Ͱࣗ෼ͷΧϝϥΛ࡞ͬͯΈΑ͏
   
   $PSF
   .FEJB
   *0
   &YUFOTJPOT ෰෦
   ஐ J04%$
    !TINEFWFMPQ
2. NBD04ͷԾ૝Χϝϥ࣮૷ʹ͍ͭͯ࿩͠·͢
3. ʮ൒೥લͷࣗ෼͕ٽ͍ͯتͿࢿྉʯ
   😂
4. ෰෦
   ஐ $ZCFS"HFOU
   
   
   
   $ZCFS
   "*
   1SPEVDUJPOT
   
   
   
   Y3Ϊϧυ
   Ϧʔμʔ
   
   
   
   "3
   /FYU
   &YQFSU ࠷ۙͷڵຯ
   ɹWJTJPO04ɺ7JTJPO
   1SPɺ
   ɹ6OSFBM
   &OHJOF
   ɺ/%*
   ɹJ1IPOFͷө૾ϫʔΫϑϩʔ׆༻
5. IUUQTHJUIVCDPNTBUPTIJWJTJPO04@%BZT ༨ஊ
   WJTJPO04
   %BZT
   νϟϨϯδ΍Γ·ͨ͠
6. ݱ࣮֦ுͷΤϞ͞
7. Ծ૝Χϝϥ͸
   ɹݱ࣮֦ு͍ͯ͠Δ͠ɹ
   ɹΧϝϥػೳΛιϑτ΢ΣΞͰ࣮ݱ͍ͯ͠Δ͠
   ɹϋʔυ΢ΣΞͱ࿈ಈ΋Ͱ͖Δ
   ͷͰ޷͖
8. ·ͣײँΛड़΂·͢ʂ IUUQTHJUIVCDPNOPQQFGPYXPMGDJ fi MUFSDBN
9. ·ͣײँΛड़΂·͢ʂ IUUQTRJJUBDPNGV[JLJJUFNTDBDBFBE
10. ໨࣍ NBD04ͷԾ૝Χϝϥʹ͍ͭͯ
    γϯϓϧͳ࣮૷Λ࡞੒͠ಈ͔ͯ͠ΈΔ
    Χϝϥө૾දࣔͱө૾΁ͷ߹੒
    ͞Βʹػೳ֦ு
    σϞ
11. ໨࣍ NBD04ͷԾ૝Χϝϥʹ͍ͭͯ
    γϯϓϧͳ࣮૷Λ࡞੒͠ಈ͔ͯ͠ΈΔ
    Χϝϥө૾දࣔͱө૾΁ͷ߹੒
    ͞Βʹػೳ֦ு
    σϞ
12. NBD04ͷԾ૝Χϝϥʹ͍ͭͯ
13. NBD04ͷԾ૝Χϝϥʹ͍ͭͯ CFGPSF
    NBD04
     %FWJDF
    "CTUSBDUJPO
    -BZFS
    %"-
    1MVH*OT
14. NBD04ͷԾ૝Χϝϥʹ͍ͭͯ IUUQTTQFBLFSEFDLDPNTBUPTIJNBDPTKJBYJBOHLBNFSBUFSPUVQVLBNVTIJ[IVBOHGBOHGBUPTPGBMTFYJBO
15. NBD04ͷԾ૝Χϝϥʹ͍ͭͯ IUUQTEFWFMPQFSBQQMFDPNWJEFPTQMBZXXED
16. NBD04ͷԾ૝Χϝϥʹ͍ͭͯ Ґஔ֬ೝ༻ΦϒδΣΫτΛஔ͍ͯΈΔ IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPODPSFNFEJBJP
17. NBD04ͷԾ૝Χϝϥʹ͍ͭͯ Ґஔ֬ೝ༻ΦϒδΣΫτΛஔ͍ͯΈΔ IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPODPSFNFEJBJP
18. NBD04ͷԾ૝Χϝϥʹ͍ͭͯ NBD04
    ʙ $PSF
    .FEJB
    *0
    &YUFOTJPOT
19. NBD04ͷԾ૝Χϝϥʹ͍ͭͯ IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPODPSFNFEJBJPDSFBUJOH@B@DBNFSB@FYUFOTJPO@XJUI@DPSF@NFEJB@J@P
20. )PTU
    BQQ &YUFOTJPO
21. IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPODPSFNFEJBJPDSFBUJOH@B@DBNFSB@FYUFOTJPO@XJUI@DPSF@NFEJB@J@P
22. IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPODPSFNFEJBJPDSFBUJOH@B@DBNFSB@FYUFOTJPO@XJUI@DPSF@NFEJB@J@P
23. ໨࣍ NBD04ͷԾ૝Χϝϥʹ͍ͭͯ
    γϯϓϧͳ࣮૷Λ࡞੒͠ಈ͔ͯ͠ΈΔ
    Χϝϥө૾දࣔͱө૾΁ͷ߹੒
    ͞Βʹػೳ֦ு
    σϞ
24. ໨࣍ NBD04ͷԾ૝Χϝϥʹ͍ͭͯ
    γϯϓϧͳ࣮૷Λ࡞੒͠ಈ͔ͯ͠ΈΔ
    Χϝϥө૾දࣔͱө૾΁ͷ߹੒
    ͞Βʹػೳ֦ு
    σϞ
25. γϯϓϧͳ࣮૷Λ࡞੒͠ಈ͔ͯ͠ΈΔ
26. ϓϩδΣΫτ৽ن࡞੒
    4ZTUFN
    &YUFOTJPOͱ"QQ
    (SPVQT௥Ճ
    $BNFSB
    &YUFOTJPO௥Ճͱ"QQ
    (SPVQTઃఆ
    *OTUBMMͱ6OJOTUBMMϘλϯ௥Ճ
    Ϗϧυ͠"QQMJDBUJPO഑ԼʹBQQΛ഑ஔ
    ࣮ߦͱηΩϡϦςΟڐՄ γϯϓϧͳ࣮૷Λ࡞੒͠ಈ͔ͯ͠ΈΔ
39. import SwiftUI import SystemExtensions struct ContentView: View { let extensionID: String = "tokyo.shmdevelopment.MyCreativeCamera.Extension" var body: some View { HStack { Button { let activationRequest = OSSystemExtensionRequest.activationRequest( forExtensionWithIdentifier: extensionID, queue: .main) OSSystemExtensionManager.shared.submitRequest(activationRequest) } label: { Text("Install") } Button { let deactivationRequest = OSSystemExtensionRequest.deactivationRequest( forExtensionWithIdentifier: extensionID, queue: .main) OSSystemExtensionManager.shared.submitRequest(deactivationRequest) } label: { Text("Uninstall") } } .padding() } } $POUFOU7JFXTXJGU
40. import SwiftUI import SystemExtensions struct ContentView: View { let extensionID: String = "tokyo.shmdevelopment.MyCreativeCamera.Extension" var body: some View { HStack { Button { let activationRequest = OSSystemExtensionRequest.activationRequest( forExtensionWithIdentifier: extensionID, queue: .main) OSSystemExtensionManager.shared.submitRequest(activationRequest) } label: { Text("Install") } Button { let deactivationRequest = OSSystemExtensionRequest.deactivationRequest( forExtensionWithIdentifier: extensionID, queue: .main) OSSystemExtensionManager.shared.submitRequest(deactivationRequest) } label: { Text("Uninstall") } } .padding() } } $POUFOU7JFXTXJGU
46. 🎉
47. ॳճҎ߱ͷߋ৽࡞ۀ͸গ͠࡞๏͕͋Γ·͢ γϯϓϧͳ࣮૷Λ࡞੒͠ಈ͔ͯ͠ΈΔ
49. ໨࣍ NBD04ͷԾ૝Χϝϥʹ͍ͭͯ
    γϯϓϧͳ࣮૷Λ࡞੒͠ಈ͔ͯ͠ΈΔ
    Χϝϥө૾දࣔͱө૾΁ͷ߹੒
    ͞Βʹػೳ֦ு
    σϞ
50. ໨࣍ NBD04ͷԾ૝Χϝϥʹ͍ͭͯ
    γϯϓϧͳ࣮૷Λ࡞੒͠ಈ͔ͯ͠ΈΔ
    Χϝϥө૾දࣔͱө૾΁ͷ߹੒
    ͞Βʹػೳ֦ு
    σϞ
51. Χϝϥө૾දࣔͱө૾΁ͷ߹੒
52. Χϝϥө૾දࣔ
53. &YUFOTJPOଆ
58. import AVFoundation import CoreImage import CoreImage.CIFilterBuiltins &YUFOTJPO1SPWJEFSTXJGU
59. extension AVCaptureDevice.DiscoverySession { static func faceTimeDevice() -> AVCaptureDevice { let discoverySession = AVCaptureDevice.DiscoverySession( deviceTypes: [.builtInWideAngleCamera], mediaType: .video, position: .unspecified ) let devices = discoverySession.devices let device = devices.filter({ $0.manufacturer == "Apple Inc." && $0.modelID.hasPrefix("FaceTime ")}).first! return device } } &YUFOTJPO1SPWJEFSTXJGU
60. let input: AVCaptureDeviceInput = { let device = AVCaptureDevice.DiscoverySession.faceTimeDevice() return try! AVCaptureDeviceInput(device: device) }() let output: AVCaptureVideoDataOutput = { let output = AVCaptureVideoDataOutput() output.videoSettings = [ kCVPixelBufferPixelFormatTypeKey as String : kCVPixelFormatType_32BGRA ] return output }() lazy var session: AVCaptureSession = { var session = AVCaptureSession() session.addInput(input) output.setSampleBufferDelegate(self, queue: .main) session.addOutput(output) return session }() &YUFOTJPO1SPWJEFSTXJGU
61. let input: AVCaptureDeviceInput = { let device = AVCaptureDevice.DiscoverySession.faceTimeDevice() return try! AVCaptureDeviceInput(device: device) }() let output: AVCaptureVideoDataOutput = { let output = AVCaptureVideoDataOutput() output.videoSettings = [ kCVPixelBufferPixelFormatTypeKey as String : kCVPixelFormatType_32BGRA ] return output }() lazy var session: AVCaptureSession = { var session = AVCaptureSession() session.addInput(input) output.setSampleBufferDelegate(self, queue: .main) session.addOutput(output) return session }() &YUFOTJPO1SPWJEFSTXJGU
62. func startStreaming() { guard let _ = _bufferPool else { return } _streamingCounter += 1 _timer = DispatchSource.makeTimerSource(flags: .strict, queue: _timerQueue) _timer!.schedule(deadline: .now(), repeating: 1.0 / Double(kFrameRate), leeway: .seconds(0)) _timer!.setEventHandler { var err: OSStatus = 0 let now = CMClockGetTime(CMClockGetHostTimeClock()) var pixelBuffer: CVPixelBuffer? err = CVPixelBufferPoolCreatePixelBufferWithAuxAttributes(kCFAllocatorDefault, self._bufferPool, self._bufferAuxAttributes, &pixelBuffer) if err != 0 { os_log(.error, "out of pixel buffers \(err)") } if let pixelBuffer = pixelBuffer { CVPixelBufferLockBaseAddress(pixelBuffer, []) var bufferPtr = CVPixelBufferGetBaseAddress(pixelBuffer)! let width = CVPixelBufferGetWidth(pixelBuffer) let height = CVPixelBufferGetHeight(pixelBuffer) let rowBytes = CVPixelBufferGetBytesPerRow(pixelBuffer) memset(bufferPtr, 0, rowBytes * height) let whiteStripeStartRow = self._whiteStripeStartRow if self._whiteStripeIsAscending { self._whiteStripeStartRow = whiteStripeStartRow - 1 self._whiteStripeIsAscending = self._whiteStripeStartRow > 0 } else { self._whiteStripeStartRow = whiteStripeStartRow + 1 self._whiteStripeIsAscending = self._whiteStripeStartRow >= (height - kWhiteStripeHeight) } bufferPtr += rowBytes * Int(whiteStripeStartRow) for _ in 0..<kWhiteStripeHeight { for _ in 0..<width { var white: UInt32 = 0xFFFFFFFF memcpy(bufferPtr, &white, MemoryLayout.size(ofValue: white)) bufferPtr += MemoryLayout.size(ofValue: white) } } CVPixelBufferUnlockBaseAddress(pixelBuffer, []) var sbuf: CMSampleBuffer! var timingInfo = CMSampleTimingInfo() timingInfo.presentationTimeStamp = CMClockGetTime(CMClockGetHostTimeClock()) err = CMSampleBufferCreateForImageBuffer(allocator: kCFAllocatorDefault, imageBuffer: pixelBuffer, dataReady: true, makeDataReadyCallback: nil, refcon: nil, formatDescription: self._videoDescription, sampleTiming: &timingInfo, sampleBufferOut: &sbuf) if err == 0 { self._streamSource.stream.send(sbuf, discontinuity: [], hostTimeInNanoseconds: UInt64(timingInfo.presentationTimeStamp.seconds * Double(NSEC_PER_SEC))) } os_log(.info, "video time \(timingInfo.presentationTimeStamp.seconds) now \(now.seconds) err \(err)") } } _timer!.setCancelHandler { } _timer!.resume() } &YUFOTJPO1SPWJEFSTXJGU DMBTT
    &YUFOTJPO%FWJDF4PVSDF
63. func startStreaming() { guard let _ = _bufferPool else { return } _streamingCounter += 1 _timer = DispatchSource.makeTimerSource(flags: .strict, queue: _timerQueue) _timer!.schedule(deadline: .now(), repeating: 1.0 / Double(kFrameRate), leeway: .seconds(0)) _timer!.setEventHandler { var err: OSStatus = 0 let now = CMClockGetTime(CMClockGetHostTimeClock()) var pixelBuffer: CVPixelBuffer? err = CVPixelBufferPoolCreatePixelBufferWithAuxAttributes(kCFAllocatorDefault, self._bufferPool, self._bufferAuxAttributes, &pixelBuffer) if err != 0 { os_log(.error, "out of pixel buffers \(err)") } if let pixelBuffer = pixelBuffer { CVPixelBufferLockBaseAddress(pixelBuffer, []) var bufferPtr = CVPixelBufferGetBaseAddress(pixelBuffer)! let width = CVPixelBufferGetWidth(pixelBuffer) let height = CVPixelBufferGetHeight(pixelBuffer) let rowBytes = CVPixelBufferGetBytesPerRow(pixelBuffer) memset(bufferPtr, 0, rowBytes * height) let whiteStripeStartRow = self._whiteStripeStartRow if self._whiteStripeIsAscending { self._whiteStripeStartRow = whiteStripeStartRow - 1 self._whiteStripeIsAscending = self._whiteStripeStartRow > 0 } else { self._whiteStripeStartRow = whiteStripeStartRow + 1 self._whiteStripeIsAscending = self._whiteStripeStartRow >= (height - kWhiteStripeHeight) } bufferPtr += rowBytes * Int(whiteStripeStartRow) for _ in 0..<kWhiteStripeHeight { for _ in 0..<width { var white: UInt32 = 0xFFFFFFFF memcpy(bufferPtr, &white, MemoryLayout.size(ofValue: white)) bufferPtr += MemoryLayout.size(ofValue: white) } } CVPixelBufferUnlockBaseAddress(pixelBuffer, []) var sbuf: CMSampleBuffer! var timingInfo = CMSampleTimingInfo() timingInfo.presentationTimeStamp = CMClockGetTime(CMClockGetHostTimeClock()) err = CMSampleBufferCreateForImageBuffer(allocator: kCFAllocatorDefault, imageBuffer: pixelBuffer, dataReady: true, makeDataReadyCallback: nil, refcon: nil, formatDescription: self._videoDescription, sampleTiming: &timingInfo, sampleBufferOut: &sbuf) if err == 0 { self._streamSource.stream.send(sbuf, discontinuity: [], hostTimeInNanoseconds: UInt64(timingInfo.presentationTimeStamp.seconds * Double(NSEC_PER_SEC))) } os_log(.info, "video time \(timingInfo.presentationTimeStamp.seconds) now \(now.seconds) err \(err)") } } _timer!.setCancelHandler { } _timer!.resume() } &YUFOTJPO1SPWJEFSTXJGU DMBTT
    &YUFOTJPO%FWJDF4PVSDF
64. func startStreaming() { guard let _ = _bufferPool else { return } _streamingCounter += 1 _timer = DispatchSource.makeTimerSource(flags: .strict, queue: _timerQueue) _timer!.schedule(deadline: .now(), repeating: 1.0 / Double(kFrameRate), leeway: .seconds(0)) _timer!.setEventHandler { var err: OSStatus = 0 let now = CMClockGetTime(CMClockGetHostTimeClock()) var pixelBuffer: CVPixelBuffer? err = CVPixelBufferPoolCreatePixelBufferWithAuxAttributes(kCFAllocatorDefault, self._bufferPool, self._bufferAuxAttributes, &pixelBuffer) if err != 0 { os_log(.error, "out of pixel buffers \(err)") } if let pixelBuffer = pixelBuffer { CVPixelBufferLockBaseAddress(pixelBuffer, []) var bufferPtr = CVPixelBufferGetBaseAddress(pixelBuffer)! let width = CVPixelBufferGetWidth(pixelBuffer) let height = CVPixelBufferGetHeight(pixelBuffer) let rowBytes = CVPixelBufferGetBytesPerRow(pixelBuffer) memset(bufferPtr, 0, rowBytes * height) let whiteStripeStartRow = self._whiteStripeStartRow if self._whiteStripeIsAscending { self._whiteStripeStartRow = whiteStripeStartRow - 1 self._whiteStripeIsAscending = self._whiteStripeStartRow > 0 } else { self._whiteStripeStartRow = whiteStripeStartRow + 1 self._whiteStripeIsAscending = self._whiteStripeStartRow >= (height - kWhiteStripeHeight) } bufferPtr += rowBytes * Int(whiteStripeStartRow) for _ in 0..<kWhiteStripeHeight { for _ in 0..<width { var white: UInt32 = 0xFFFFFFFF memcpy(bufferPtr, &white, MemoryLayout.size(ofValue: white)) bufferPtr += MemoryLayout.size(ofValue: white) } } CVPixelBufferUnlockBaseAddress(pixelBuffer, []) var sbuf: CMSampleBuffer! var timingInfo = CMSampleTimingInfo() timingInfo.presentationTimeStamp = CMClockGetTime(CMClockGetHostTimeClock()) err = CMSampleBufferCreateForImageBuffer(allocator: kCFAllocatorDefault, imageBuffer: pixelBuffer, dataReady: true, makeDataReadyCallback: nil, refcon: nil, formatDescription: self._videoDescription, sampleTiming: &timingInfo, sampleBufferOut: &sbuf) if err == 0 { self._streamSource.stream.send(sbuf, discontinuity: [], hostTimeInNanoseconds: UInt64(timingInfo.presentationTimeStamp.seconds * Double(NSEC_PER_SEC))) } os_log(.info, "video time \(timingInfo.presentationTimeStamp.seconds) now \(now.seconds) err \(err)") } } _timer!.setCancelHandler { } _timer!.resume() } &YUFOTJPO1SPWJEFSTXJGU DMBTT
    &YUFOTJPO%FWJDF4PVSDF
65. self._streamSource.stream.send(sbuf, discontinuity: [], hostTimeInNanoseconds: UInt64(timingInfo.presentationTimeStamp.seconds * Double(NSEC_PER_SEC))) &YUFOTJPO1SPWJEFSTXJGU
66. func startStreaming() { guard let _ = _bufferPool else { return } _streamingCounter += 1 _timer = DispatchSource.makeTimerSource(flags: .strict, queue: _timerQueue) _timer!.schedule(deadline: .now(), repeating: 1.0 / Double(kFrameRate), leeway: .seconds(0)) _timer!.setEventHandler { var err: OSStatus = 0 let now = CMClockGetTime(CMClockGetHostTimeClock()) var pixelBuffer: CVPixelBuffer? err = CVPixelBufferPoolCreatePixelBufferWithAuxAttributes(kCFAllocatorDefault, self._bufferPool, self._bufferAuxAttributes, &pixelBuffer) if err != 0 { os_log(.error, "out of pixel buffers \(err)") } if let pixelBuffer = pixelBuffer { CVPixelBufferLockBaseAddress(pixelBuffer, []) var bufferPtr = CVPixelBufferGetBaseAddress(pixelBuffer)! let width = CVPixelBufferGetWidth(pixelBuffer) let height = CVPixelBufferGetHeight(pixelBuffer) let rowBytes = CVPixelBufferGetBytesPerRow(pixelBuffer) memset(bufferPtr, 0, rowBytes * height) let whiteStripeStartRow = self._whiteStripeStartRow if self._whiteStripeIsAscending { self._whiteStripeStartRow = whiteStripeStartRow - 1 self._whiteStripeIsAscending = self._whiteStripeStartRow > 0 } else { self._whiteStripeStartRow = whiteStripeStartRow + 1 self._whiteStripeIsAscending = self._whiteStripeStartRow >= (height - kWhiteStripeHeight) } bufferPtr += rowBytes * Int(whiteStripeStartRow) for _ in 0..<kWhiteStripeHeight { for _ in 0..<width { var white: UInt32 = 0xFFFFFFFF memcpy(bufferPtr, &white, MemoryLayout.size(ofValue: white)) bufferPtr += MemoryLayout.size(ofValue: white) } } CVPixelBufferUnlockBaseAddress(pixelBuffer, []) var sbuf: CMSampleBuffer! var timingInfo = CMSampleTimingInfo() timingInfo.presentationTimeStamp = CMClockGetTime(CMClockGetHostTimeClock()) err = CMSampleBufferCreateForImageBuffer(allocator: kCFAllocatorDefault, imageBuffer: pixelBuffer, dataReady: true, makeDataReadyCallback: nil, refcon: nil, formatDescription: self._videoDescription, sampleTiming: &timingInfo, sampleBufferOut: &sbuf) if err == 0 { self._streamSource.stream.send(sbuf, discontinuity: [], hostTimeInNanoseconds: UInt64(timingInfo.presentationTimeStamp.seconds * Double(NSEC_PER_SEC))) } os_log(.info, "video time \(timingInfo.presentationTimeStamp.seconds) now \(now.seconds) err \(err)") } } _timer!.setCancelHandler { } _timer!.resume() } &YUFOTJPO1SPWJEFSTXJGU DMBTT
    &YUFOTJPO%FWJDF4PVSDF
67. func startStreaming() { session.startRunning() } &YUFOTJPO1SPWJEFSTXJGU DMBTT
    &YUFOTJPO%FWJDF4PVSDF
68. extension ExtensionDeviceSource: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { let inputImage = CIImage(cvImageBuffer: sampleBuffer.imageBuffer!) let filter = CIFilter.sepiaTone() filter.inputImage = inputImage ciContext.render(filter.outputImage!, to: sampleBuffer.imageBuffer!) _streamSource.stream.send(sampleBuffer, discontinuity: .time, hostTimeInNanoseconds: 0) } func captureOutput(_ output: AVCaptureOutput, didDrop sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { _streamSource.stream.send(sampleBuffer, discontinuity: .sampleDropped, hostTimeInNanoseconds: 0) } } &YUFOTJPO1SPWJEFSTXJGU
69. extension ExtensionDeviceSource: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { let inputImage = CIImage(cvImageBuffer: sampleBuffer.imageBuffer!) let filter = CIFilter.sepiaTone() filter.inputImage = inputImage ciContext.render(filter.outputImage!, to: sampleBuffer.imageBuffer!) _streamSource.stream.send(sampleBuffer, discontinuity: .time, hostTimeInNanoseconds: 0) } func captureOutput(_ output: AVCaptureOutput, didDrop sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { _streamSource.stream.send(sampleBuffer, discontinuity: .sampleDropped, hostTimeInNanoseconds: 0) } } &YUFOTJPO1SPWJEFSTXJGU
70. Χϝϥө૾Λදࣔ͢Δ
72. 🎉
73. Χϝϥө૾ʹը૾Λಈతʹ߹੒͢Δ
74. ϗετΞϓϦɺ&YUFOTJPO
    ڞ௨෦෼
75. static func generate(text: String, size: CGFloat = 100, imageSize: NSSize = NSSize(width: 1920, height: 1080)) -> CIImage? { let font = NSFont(name: "HiraginoSans-W9", size: size) ?? NSFont.systemFont(ofSize: size) let image = NSImage(size: imageSize, flipped: false) { (rect) -> Bool in let paragraphStyle = NSMutableParagraphStyle() paragraphStyle.alignment = .center paragraphStyle.lineBreakMode = .byCharWrapping let numberOfLines: CGFloat = CGFloat(text.split(separator: "\n").count) let rectangle = NSRect(x: 0, y: imageSize.height - font.lineHeight() * numberOfLines, width: imageSize.width, height: font.lineHeight() * numberOfLines) let textAttributes = [ .strokeColor: NSColor.black, .foregroundColor: NSColor.white, .strokeWidth: -1, .font: font, .paragraphStyle: paragraphStyle ] as [NSAttributedString.Key : Any] (text as NSString).draw(in: rectangle, withAttributes: textAttributes) return true } return image.ciImage } 6UJMJUZTXJGU
76. &YUFOTJPOଆ
77. private var textImage: CIImage? // දࣔ֬ೝͷͨΊॳظॲཧ࣌ʹҰ౓͚ͩը૾࡞੒ textImage = Utility.generate(text: "Hello Ծ૝Χϝϥʂ") &YUFOTJPO1SPWJEFSTXJGU
78. private let filterComposite = CIFilter(name: "CISourceOverCompositing") private func compose(bgImage: CIImage, overlayImage: CIImage?) -> CIImage? { guard let filterComposite = filterComposite, let overlayImage = overlayImage else { return bgImage } filterComposite.setValue(overlayImage, forKeyPath: kCIInputImageKey) filterComposite.setValue(bgImage, forKeyPath: kCIInputBackgroundImageKey) return filterComposite.outputImage } &YUFOTJPO1SPWJEFSTXJGU
79. &YUFOTJPO1SPWJEFSTXJGU func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { let inputImage = CIImage(cvImageBuffer: sampleBuffer.imageBuffer!) let filter = CIFilter.sepiaTone() filter.inputImage = inputImage ciContext.render(filter.outputImage!, to: sampleBuffer.imageBuffer!) _streamSource.stream.send(sampleBuffer, discontinuity: .time, hostTimeInNanoseconds: 0) }
80. func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { let inputImage = CIImage(cvImageBuffer: sampleBuffer.imageBuffer!) if let compositedImage = compose(bgImage: inputImage, overlayImage: textImage) { ciContext.render(compositedImage, to: sampleBuffer.imageBuffer!) } _streamSource.stream.send(sampleBuffer, discontinuity: .time, hostTimeInNanoseconds: 0) } &YUFOTJPO1SPWJEFSTXJGU
82. 🎉
83. ໨࣍ NBD04ͷԾ૝Χϝϥʹ͍ͭͯ
    γϯϓϧͳ࣮૷Λ࡞੒͠ಈ͔ͯ͠ΈΔ
    Χϝϥө૾දࣔͱө૾΁ͷ߹੒
    ͞Βʹػೳ֦ு
    σϞ
84. ໨࣍ NBD04ͷԾ૝Χϝϥʹ͍ͭͯ
    γϯϓϧͳ࣮૷Λ࡞੒͠ಈ͔ͯ͠ΈΔ
    Χϝϥө૾දࣔͱө૾΁ͷ߹੒
    ͞Βʹػೳ֦ு
    σϞ
85. ͞Βʹػೳ֦ு
86. ϗετΞϓϦͱ࿈ಈͤ͞Δ
87. )PTU
    BQQ &YUFOTJPO
88. )PTU
    BQQ &YUFOTJPO ಉ͡"QQ
    (SPVQͷ
    6TFS%FGBVMUT
89. ࠓճ͸ϥΠϒϥϦ
    %FGBVMUT
    Λ௥Ճ͠࢖༻ IUUQTHJUIVCDPNTJOESFTPSIVT%FGBVMUT
95. ϗετΞϓϦɺ&YUFOTJPO
    ڞ௨෦෼
96. import Defaults extension Defaults.Keys { public static let isBypass = Key<Bool>("isBypass", default: false, suite: defaultsSuite) public static let message = Key<String>("message", default: "Hello World", suite: defaultsSuite) } %FGBVMUT ,FZTTXJGU
97. ϗετΞϓϦଆ
98. import SwiftUI import SystemExtensions import Defaults struct ContentView: View { @Default(.message) var message var body: some View { VStack { HStack { Button { let activationRequest = OSSystemExtensionRequest.activationRequest( forExtensionWithIdentifier: extensionID, queue: .main) OSSystemExtensionManager.shared.submitRequest(activationRequest) } label: { Text("Install") } Button { let deactivationRequest = OSSystemExtensionRequest.deactivationRequest( forExtensionWithIdentifier: extensionID, queue: .main) OSSystemExtensionManager.shared.submitRequest(deactivationRequest) } label: { Text("Uninstall") } Defaults.Toggle("Bypass", key: .isBypass) } .padding() HStack { TextEditor(text: $message) .font(.system(size: 16)) .border(Color.gray, width: 1) } .padding() } } } $POUFOU7JFXTXJGU
99. import SwiftUI import SystemExtensions import Defaults struct ContentView: View { @Default(.message) var message var body: some View { VStack { HStack { Button { let activationRequest = OSSystemExtensionRequest.activationRequest( forExtensionWithIdentifier: extensionID, queue: .main) OSSystemExtensionManager.shared.submitRequest(activationRequest) } label: { Text("Install") } Button { let deactivationRequest = OSSystemExtensionRequest.deactivationRequest( forExtensionWithIdentifier: extensionID, queue: .main) OSSystemExtensionManager.shared.submitRequest(deactivationRequest) } label: { Text("Uninstall") } Defaults.Toggle("Bypass", key: .isBypass) } .padding() HStack { TextEditor(text: $message) .font(.system(size: 16)) .border(Color.gray, width: 1) } .padding() } } } $POUFOU7JFXTXJGU
100. &YUFOTJPOଆ
101. private func observeSettings() { Task { for await isBypass in Defaults.updates(.isBypass) { self.isBypass = isBypass } } Task { for await message in Defaults.updates(.message) { textImage = Utility.generate(text: message) } } } &YUFOTJPO1SPWJEFSTXJGU
102. private func observeSettings() { Task { for await isBypass in Defaults.updates(.isBypass) { self.isBypass = isBypass } } Task { for await message in Defaults.updates(.message) { textImage = Utility.generate(text: message) } } } &YUFOTJPO1SPWJEFSTXJGU
103. func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { if isBypass { _streamSource.stream.send(sampleBuffer, discontinuity: .time, hostTimeInNanoseconds: 0) return } let inputImage = CIImage(cvImageBuffer: sampleBuffer.imageBuffer!) if let compositedImage = compose(bgImage: inputImage, overlayImage: textImage) { ciContext.render(compositedImage, to: sampleBuffer.imageBuffer!) } _streamSource.stream.send(sampleBuffer, discontinuity: .time, hostTimeInNanoseconds: 0) } &YUFOTJPO1SPWJEFSTXJGU
104. func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { if isBypass { _streamSource.stream.send(sampleBuffer, discontinuity: .time, hostTimeInNanoseconds: 0) return } let inputImage = CIImage(cvImageBuffer: sampleBuffer.imageBuffer!) if let compositedImage = compose(bgImage: inputImage, overlayImage: textImage) { ciContext.render(compositedImage, to: sampleBuffer.imageBuffer!) } _streamSource.stream.send(sampleBuffer, discontinuity: .time, hostTimeInNanoseconds: 0) } &YUFOTJPO1SPWJEFSTXJGU
105. ςΩετೖྗΛ൓ө
107. 🎉
108. ͞Βʹػೳ֦ு
109. Ի੠ೝࣝ
110. ϗετΞϓϦଆ
115. class ViewModel: NSObject, ObservableObject, SpeechRecognizerDelegate { @Default(.message) var message: String let speechRecognizer = SpeechRecognizer() func setUpSpeechRecognizer() { speechRecognizer.setup(delegate: self) } func startSpeechRecognize() { guard !speechRecognizer.started else { return } speechRecognizer.start() } func stopSpeechRecognize() { speechRecognizer.stop() } func speechTextUpdate(value: String) { message = value } } $POUFOU7JFXTXJGU
116. import AVFoundation import Speech public class SpeechRecognizer { (...) public func setup(delegate: SpeechRecognizerDelegate) { self.delegate = delegate (...) recognizer = SFSpeechRecognizer(locale: Locale(identifier: "ja-JP")) } (...) private func startRecognitionTask() { recognitionTask = recognizer.recognitionTask(with: recognitionRequest) { result, error in var isFinal = false if let foundResult = result { self.previousResult = foundResult isFinal = foundResult.isFinal print(foundResult.bestTranscription.formattedString) self.delegate?.speechTextUpdate(value: foundResult.bestTranscription.formattedString) } if error != nil { print("\(error!.localizedDescription)") self.stop() } if isFinal { print("FINAL RESULT reached") self.stop() } } } } 4QFFDI3FDPHOJ[FSTXJGU
     Ұ෦ൈਮ
117. import AVFoundation import Speech public class SpeechRecognizer { (...) public func setup(delegate: SpeechRecognizerDelegate) { self.delegate = delegate (...) recognizer = SFSpeechRecognizer(locale: Locale(identifier: "ja-JP")) } (...) private func startRecognitionTask() { recognitionTask = recognizer.recognitionTask(with: recognitionRequest) { result, error in var isFinal = false if let foundResult = result { self.previousResult = foundResult isFinal = foundResult.isFinal print(foundResult.bestTranscription.formattedString) self.delegate?.speechTextUpdate(value: foundResult.bestTranscription.formattedString) } if error != nil { print("\(error!.localizedDescription)") self.stop() } if isFinal { print("FINAL RESULT reached") self.stop() } } } } 4QFFDI3FDPHOJ[FSTXJGU
     Ұ෦ൈਮ
119. 🎉
120. ໨࣍ NBD04ͷԾ૝Χϝϥʹ͍ͭͯ
     γϯϓϧͳ࣮૷Λ࡞੒͠ಈ͔ͯ͠ΈΔ
     Χϝϥө૾දࣔͱө૾΁ͷ߹੒
     ͞Βʹػೳ֦ு
     σϞ
121. ໨࣍ NBD04ͷԾ૝Χϝϥʹ͍ͭͯ
     γϯϓϧͳ࣮૷Λ࡞੒͠ಈ͔ͯ͠ΈΔ
     Χϝϥө૾දࣔͱө૾΁ͷ߹੒
     ͞Βʹػೳ֦ு
     σϞ
122. σϞ
123. ࠔͬͨ࣌͸ ΞϓϦߋ৽࣌͸Ұ౓ফͯ͠ೖΕ௚͠ਪ঑
     04࠶ىಈ͢Δͱղܾ͢Δ͜ͱ΋ଟ͍
     "QQ
     4BOECPYͷνΣοΫ๨Εͯ·ͤΜ͔
     JOGPQMJTU΁ͷ௥ه๨Εͯ·ͤΜ͔
124. ·ͱΊ NBD04ͷԾ૝Χϝϥʹ͍ͭͯ
     ɹɹ%"-
     ˠ
     $PSF
     .FEJB
     *0
     &YUFOTJPOT
     γϯϓϧͳ࣮૷Λ࡞੒͠ಈ͔ͯ͠ΈΔ
     ɹɹ$BNFSB
     &YUFOTJPO௥Ճͱ࡞๏
     Χϝϥө૾දࣔͱө૾΁ͷ߹੒
     ɹɹ෺ཧΧϝϥͱͷ࿈ಈɺιϑτ΢ΣΞͰͷՃ޻
     ͞Βʹػೳ֦ு
     ɹɹίϯτϩʔϧΞϓϦͱ࿈ಈɺԻ੠ೝࣝ
125. (JWF
     "XBZ ࠓճͷσϞ࣮૷Λ(JU)VCʹΞοϓ͠·ͨ͠ IUUQTHJUIVCDPNTBUPTIJ.Z$SFBUJWF$BNFSB