macOS仮想カメラ「テロップカム」 実装方法とその先

1a1d418bdf51cbf8fce1317f6c80a907?s=47 satoshi0212
September 20, 2020

macOS仮想カメラ「テロップカム」 実装方法とその先

1a1d418bdf51cbf8fce1317f6c80a907?s=128

satoshi0212

September 20, 2020
Tweet

Transcript

  1. NBD04Ծ૝ΧϝϥʮςϩοϓΧϜʯ ࣮૷ํ๏ͱͦͷઌ J04%$+BQBO  ෰෦ஐ!TINEFWFMPQ

  2. Ծ૝Χϝϥ 4OBQ$BNFSB NNINN

  3. ͜Ε͸ָ͍͠ʂ

  4. ϓϩάϥϚͳΒҰ౓͸

  5. ࣗ࡞ͨ͘͠ͳΓ·͢ΑͶʂ

  6. ྑ͍͓஌Βͤ

  7. ͳΜͱ

  8. 4XJGUͷΈͰ࣮૷Մೳʂ

  9. 5XJUUFSͰ(8όʔνϟϧΧϝϥ࡞੒νϟϨϯδͰ࡞੒աఔπΠʔτ͍ͯ͠·͢

  10. None
  11. None
  12. ߏ੒ཁૉ

  13. ߏ੒ཁૉ ɹ$PSF.FEJB*0%"-1MVHJO ɹίϯτϩʔϥΞϓϦ

  14. $PSF.FEJB*0%"- 1MVHJO ίϯτϩʔϥΞϓϦ /41BTUFCPBSE -JCSBSZ$PSF.FEJB*01MVH*OT%"- Ծ૝Χϝϥͷ࣮ମ 6*Λ࣋ͨͳ͍1MVHJOʹ ஋Λ౉͢ΞϓϦ

  15. ϓϩδΣΫτ࡞੒ͱΠϯλʔϑΣΠε࣮૷ $PSF.FEJB*0%"-1MVHJO

  16. None
  17. None
  18. None
  19. ࣗ෼Ͱࢦఆ͢Δ*%஋ɻ66*%ͳͲઃఆ

  20. ΤϯτϦʔϙΠϯτGVODUJPO໊

  21. Ծ૝Χϝϥ1MVH*Oݻఆ஋ ઌड़ͷ66*%஋

  22. import Foundation import CoreMediaIO @_cdecl("VirtualCameraSampleMain") func VirtualCameraSampleMain(allocator: CFAllocator, requestedTypeUUID: CFUUID)

    -> CMIOHardwarePlugInRef { return pluginRef } Main.swift
  23. import Foundation import CoreMediaIO @_cdecl("VirtualCameraSampleMain") func VirtualCameraSampleMain(allocator: CFAllocator, requestedTypeUUID: CFUUID)

    -> CMIOHardwarePlugInRef { return pluginRef } Main.swift ΤϯτϦʔϙΠϯτ ΠϯλʔϑΣΠεͷࢀরΛฦ٫
  24. let pluginRef: CMIOHardwarePlugInRef = { let interfacePtr = UnsafeMutablePointer<CMIOHardwarePlugInInterface>.allocate(capacity: 1)

    interfacePtr.pointee = createPluginInterface() let pluginRef = CMIOHardwarePlugInRef.allocate(capacity: 1) pluginRef.pointee = interfacePtr return pluginRef }() PluginInterface.swift (ൈਮ)
  25. let pluginRef: CMIOHardwarePlugInRef = { let interfacePtr = UnsafeMutablePointer<CMIOHardwarePlugInInterface>.allocate(capacity: 1)

    interfacePtr.pointee = createPluginInterface() let pluginRef = CMIOHardwarePlugInRef.allocate(capacity: 1) pluginRef.pointee = interfacePtr return pluginRef }() PluginInterface.swift (ൈਮ) ΠϯλʔϑΣΠεࢀরΛฦ٫
  26. private func createPluginInterface() -> CMIOHardwarePlugInInterface { return CMIOHardwarePlugInInterface( _reserved: nil,

    QueryInterface: QueryInterface, AddRef: AddRef, Release: Release, Initialize: Initialize, InitializeWithObjectID: InitializeWithObjectID, Teardown: Teardown, ObjectShow: ObjectShow, ObjectHasProperty: ObjectHasProperty, ObjectIsPropertySettable: ObjectIsPropertySettable, ObjectGetPropertyDataSize: ObjectGetPropertyDataSize, ObjectGetPropertyData: ObjectGetPropertyData, ObjectSetPropertyData: ObjectSetPropertyData, DeviceSuspend: DeviceSuspend, DeviceResume: DeviceResume, DeviceStartStream: DeviceStartStream, DeviceStopStream: DeviceStopStream, DeviceProcessAVCCommand: DeviceProcessAVCCommand, DeviceProcessRS422Command: DeviceProcessRS422Command, StreamCopyBufferQueue: StreamCopyBufferQueue, StreamDeckPlay: StreamDeckPlay, StreamDeckStop: StreamDeckStop, StreamDeckJog: StreamDeckJog, StreamDeckCueTo: StreamDeckCueTo) } PluginInterface.swift (ൈਮ)
  27. private func createPluginInterface() -> CMIOHardwarePlugInInterface { return CMIOHardwarePlugInInterface( _reserved: nil,

    QueryInterface: QueryInterface, AddRef: AddRef, Release: Release, Initialize: Initialize, InitializeWithObjectID: InitializeWithObjectID, Teardown: Teardown, ObjectShow: ObjectShow, ObjectHasProperty: ObjectHasProperty, ObjectIsPropertySettable: ObjectIsPropertySettable, ObjectGetPropertyDataSize: ObjectGetPropertyDataSize, ObjectGetPropertyData: ObjectGetPropertyData, ObjectSetPropertyData: ObjectSetPropertyData, DeviceSuspend: DeviceSuspend, DeviceResume: DeviceResume, DeviceStartStream: DeviceStartStream, DeviceStopStream: DeviceStopStream, DeviceProcessAVCCommand: DeviceProcessAVCCommand, DeviceProcessRS422Command: DeviceProcessRS422Command, StreamCopyBufferQueue: StreamCopyBufferQueue, StreamDeckPlay: StreamDeckPlay, StreamDeckStop: StreamDeckStop, StreamDeckJog: StreamDeckJog, StreamDeckCueTo: StreamDeckCueTo) } PluginInterface.swift (ൈਮ) ΠϯλʔϑΣΠε͕ظ଴͢Δ΋ͷΛฦ٫
  28. PluginInterface.swift (ൈਮ) ৄࡉ͸ιʔεࢀর

  29. Χϝϥө૾औಘͱग़ྗ

  30. func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, ɹɹɹɹɹɹɹɹɹɹɹ ɹɹfrom connection:

    AVCaptureConnection) { if output == cameraCapture.output { guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return } let cameraImage = CIImage(cvImageBuffer: imageBuffer) let compositedImage = compose(bgImage: cameraImage, overlayImage: self.textImage) var pixelBuffer: CVPixelBuffer? _ = CVPixelBufferCreate( kCFAllocatorDefault, Int(compositedImage.extent.size.width), Int(compositedImage.extent.height), kCVPixelFormatType_32BGRA, self.CVPixelBufferCreateOptions as CFDictionary, &pixelBuffer ) if let pixelBuffer = pixelBuffer { context.render(compositedImage, to: pixelBuffer) delegate?.videoComposer(self, didComposeImageBuffer: pixelBuffer) } } } VideoComposer.swift (ൈਮ)
  31. func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, ɹɹɹɹɹɹɹɹɹɹɹ ɹɹfrom connection:

    AVCaptureConnection) { if output == cameraCapture.output { guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return } let cameraImage = CIImage(cvImageBuffer: imageBuffer) let compositedImage = compose(bgImage: cameraImage, overlayImage: self.textImage) var pixelBuffer: CVPixelBuffer? _ = CVPixelBufferCreate( kCFAllocatorDefault, Int(compositedImage.extent.size.width), Int(compositedImage.extent.height), kCVPixelFormatType_32BGRA, self.CVPixelBufferCreateOptions as CFDictionary, &pixelBuffer ) if let pixelBuffer = pixelBuffer { context.render(compositedImage, to: pixelBuffer) delegate?.videoComposer(self, didComposeImageBuffer: pixelBuffer) } } } VideoComposer.swift (ൈਮ) ผ్ੜ੒ͨ͠ΦʔόʔϨΠςΩετը૾ͱ߹੒ EFMFHBUFܦ༝ͰQJYFM#V⒎FSΛ౉͢
  32. private func enqueueBuffer() { (தུ) var sampleBufferUnmanaged: Unmanaged<CMSampleBuffer>? = nil

    error = CMIOSampleBufferCreateForImageBuffer( kCFAllocatorDefault, pixelBuffer, formatDescription, &timing, sequenceNumber, UInt32(kCMIOSampleBufferNoDiscontinuities), &sampleBufferUnmanaged ) guard error == noErr else { log("CMIOSampleBufferCreateForImageBuffer Error: \(error)") return } CMSimpleQueueEnqueue(queue, element: sampleBufferUnmanaged!.toOpaque()) queueAlteredProc?(objectID, sampleBufferUnmanaged!.toOpaque(), queueAlteredRefCon) sequenceNumber += 1 } Stream.swift (ൈਮ)
  33. private func enqueueBuffer() { (தུ) var sampleBufferUnmanaged: Unmanaged<CMSampleBuffer>? = nil

    error = CMIOSampleBufferCreateForImageBuffer( kCFAllocatorDefault, pixelBuffer, formatDescription, &timing, sequenceNumber, UInt32(kCMIOSampleBufferNoDiscontinuities), &sampleBufferUnmanaged ) guard error == noErr else { log("CMIOSampleBufferCreateForImageBuffer Error: \(error)") return } CMSimpleQueueEnqueue(queue, element: sampleBufferUnmanaged!.toOpaque()) queueAlteredProc?(objectID, sampleBufferUnmanaged!.toOpaque(), queueAlteredRefCon) sequenceNumber += 1 } Stream.swift (ൈਮ) QJYFM#V⒎FS͔Βੜ੒ͨ͠ $.4BNQMF#V⒎FSΛRVFVFʹ௥Ճ
  34. QMVHJOϑΝΠϧͷ഑ஔ

  35. None
  36. -JCSBSZ$PSF.FEJB*01MVH*OT%"-

  37. දࣔ͞Εͨʂ

  38. ίϯτϩʔϥΞϓϦ

  39. $PSF.FEJB*0%"- 1MVHJO ίϯτϩʔϥΞϓϦ /41BTUFCPBSE -JCSBSZ$PSF.FEJB*01MVH*OT%"- Ծ૝Χϝϥͷ࣮ମ 6*Λ࣋ͨͳ͍1MVHJOʹ ஋Λ౉͢ΞϓϦ

  40. 1BTUFCPBSEʹ஋Λஔ͘໾ׂ

  41. None
  42. /41BTUFCPBSEͰσʔλڞ༗

  43. extension NSPasteboard.Name { static let main = NSPasteboard.Name(Config.mainAppBundleIdentifier) } extension

    NSPasteboard.PasteboardType { static let plain = NSPasteboard.PasteboardType(rawValue: "public.utf8-plain-text") } class SettingsPasteboard { static let shared = SettingsPasteboard() open var settings = [String: Any]() open func current() -> [String: Any] { let pasteboard = NSPasteboard(name: .main) if let element = pasteboard.pasteboardItems?.last, let str = element.string(forType: .plain), let data = str.data(using: .utf8) { do { let json = try JSONSerialization.jsonObject(with: data, options : .allowFragments) as! [String: Any] settings = json } catch { print("can't convert json") } } return settings } open func update() { let jsonStr = stringify(json: settings) let pasteboard = NSPasteboard(name: .main) pasteboard.declareTypes([.string], owner: nil) pasteboard.setString(jsonStr, forType: .string) } ... } SettingsPasteboard.swift (ൈਮ)
  44. extension NSPasteboard.Name { static let main = NSPasteboard.Name(Config.mainAppBundleIdentifier) } extension

    NSPasteboard.PasteboardType { static let plain = NSPasteboard.PasteboardType(rawValue: "public.utf8-plain-text") } class SettingsPasteboard { static let shared = SettingsPasteboard() open var settings = [String: Any]() open func current() -> [String: Any] { let pasteboard = NSPasteboard(name: .main) if let element = pasteboard.pasteboardItems?.last, let str = element.string(forType: .plain), let data = str.data(using: .utf8) { do { let json = try JSONSerialization.jsonObject(with: data, options : .allowFragments) as! [String: Any] settings = json } catch { print("can't convert json") } } return settings } open func update() { let jsonStr = stringify(json: settings) let pasteboard = NSPasteboard(name: .main) pasteboard.declareTypes([.string], owner: nil) pasteboard.setString(jsonStr, forType: .string) } ... } SettingsPasteboard.swift (ൈਮ)
  45. $PSF.FEJB*0%"- 1MVHJO ίϯτϩʔϥΞϓϦ /41BTUFCPBSE -JCSBSZ$PSF.FEJB*01MVH*OT%"- Ծ૝Χϝϥͷ࣮ମ 6*Λ࣋ͨͳ͍1MVHJOʹ ஋Λ౉͢ΞϓϦ

  46. ׬੒

  47. ίϯτϩʔϥΞϓϦ͔Β஋Λૹ৴

  48. None
  49. None
  50. None
  51. ࣮૷ํ๏ͱʮͦͷઌʯ

  52. 'BDFUJNFDBNFSB BVEJPJOQVU TIBEFSF⒎FDU "1* LFZCPBSEJOQVU ,FZOPUF FUD ֎෦ Ծ૝Χϝϥ ೖग़ྗՃ޻ػߏ

    ݱ࣮֦ுͱͯ͠ͷԾ૝Χϝϥ ֤छೖྗ
  53. ϚΠϯυϚοϓ

  54. δΣωϨΠςΟϒදݱ

  55. ελϯϓϦΞΫγϣϯ

  56. Ի੠ೝࣝͱ຋༁

  57. ݱ࣮֦ுͱͯ͠ͷԾ૝Χϝϥ

  58. ࢀߟ IUUQTEFWFMPQFSBQQMFDPNMJCSBSZBSDIJWFTBNQMFDPEF$PSF.FEJB*0*OUSPEVDUJPO*OUSPIUNM IUUQTHJUIVCDPNKPIOCPJMFTDPSFNFEJBJPEBMNJOJNBMFYBNQMF IUUQTHJUIVCDPNTFBODIBT4JNQMF%"-1MVHJO IUUQTTQFBLFSEFDLDPNLJTIJLBXBLBUTVNJWJSUVBMXFCDBNFSBXP[VPSPV

  59. 5XJUUFSͰ࠷৽৘ใൃ৴த !TINEFWFMPQ Ծ૝Χϝϥ࣮૷αϯϓϧ IUUQTHJUIVCDPNTBUPTIJ7JSUVBM$BNFSB4BNQMF