Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Virtual Web Cameraを作ろう

Virtual Web Cameraを作ろう

Virtual Web Cameraを作ろう/Let's Build Virtual Webcam for macOS
CoreMediaIO Device Abstraction Layer Plug-In

Kishikawa Katsumi

April 27, 2020
Tweet

More Decks by Kishikawa Katsumi

Other Decks in Programming

Transcript

  1. Kishikawa Katsumi
    Virtual Web CameraΛ࡞Ζ͏
    CoreMediaIO Device Abstraction Layer Plug-In

    View full-size slide

  2. Snap Camera Package Installer

    View full-size slide

  3. Snap Camera Package Installer

    View full-size slide

  4. Snap Camera Package Installer

    View full-size slide

  5. CoreMediaIO DAL Plug-In

    View full-size slide

  6. Sample Code
    • https://developer.apple.com/library/archive/samplecode/CoreMediaIO/Introduction/Intro.html

    • https://github.com/lvsti/CoreMediaIO-DAL-Example

    • https://github.com/johnboiles/obs-mac-virtualcam

    • https://github.com/johnboiles/coremediaio-dal-minimal-example

    • https://github.com/seanchas116/SimpleDALPlugin

    • https://github.com/kishikawakatsumi/VirtualCameraComposer-Example

    View full-size slide

  7. Apple's Sample Code

    View full-size slide

  8. johnboiles/obs-mac-virtualcam

    View full-size slide

  9. johnboiles/coremediaio-dal-minimal-example

    View full-size slide

  10. seanchas116/SimpleDALPlugin

    View full-size slide

  11. Virtual Camera
    CoreMediaIO DAL Plug-In

    View full-size slide

  12. CoreMediaIO DAL Plug-In

    View full-size slide

  13. CoreMediaIO DAL Plug-In
    ݻఆͷIDɻCoreMediaIOͷϔομʹॻ͍ͯ͋Δɻ

    View full-size slide

  14. CoreMediaIO DAL Plug-In
    ೚ҙͷIDɻuuidgenͳͲͰద౰ʹ࡞Δɻ

    ॳظԽ࣌ʹ౉͞ΕΔɻ

    View full-size slide

  15. CoreMediaIO DAL Plug-In
    ΤϯτϦʔϙΠϯτɻ

    ಉ໊͡લͷؔ਺Λ༻ҙ͢Δɻ

    View full-size slide

  16. CoreMediaIO DAL Plug-In
    #import
    #import "PlugInInterface.h"
    #import "Logging.h"
    //! PlugInMain is the entrypoint for the plugin
    extern "C" {
    void* PlugInMain(CFAllocatorRef allocator, CFUUIDRef requestedTypeUUID) {
    DLogFunc(@"");
    if (!CFEqual(requestedTypeUUID, kCMIOHardwarePlugInTypeID)) {
    return 0;
    }
    return PlugInRef();
    }
    }

    View full-size slide

  17. CoreMediaIO DAL Plug-In
    #import
    #import "PlugInInterface.h"
    #import "Logging.h"
    //! PlugInMain is the entrypoint for the plugin
    extern "C" {
    void* PlugInMain(CFAllocatorRef allocator, CFUUIDRef requestedTypeUUID) {
    DLogFunc(@"");
    if (!CFEqual(requestedTypeUUID, kCMIOHardwarePlugInTypeID)) {
    return 0;
    }
    return PlugInRef();
    }
    }
    C++Ͱॻ͘ͳΒϚϯάϦϯά͞Εͳ͍Α͏ʹ͢Δɻ

    View full-size slide

  18. CoreMediaIO DAL Plug-In
    import Foundation
    import CoreMediaIO
    @_cdecl("simpleDALPluginMain")
    func simpleDALPluginMain(allocator: CFAllocator,
    requestedTypeUUID: CFUUID) -> CMIOHardwarePlugInRef {
    NSLog("simpleDALPluginMain")
    return pluginRef
    } SwiftͰ΋ಉ͡ɻ

    View full-size slide

  19. CoreMediaIO DAL Plug-In
    PlugIn *plugIn = [PlugIn SharedPlugIn];
    plugIn.objectId = objectID;
    Device *device = [[Device alloc] init];
    CMIOObjectID deviceId;
    error = CMIOObjectCreate(PlugInRef(), kCMIOObjectSystemObject, kCMIODeviceClassID, &deviceId);
    ...
    Stream *stream = [[Stream alloc] init];
    CMIOObjectID streamId;
    error = CMIOObjectCreate(PlugInRef(), deviceId, kCMIOStreamClassID, &streamId);
    ...
    stream.objectId = streamId;
    [[ObjectStore SharedObjectStore] setObject:stream forObjectId:streamId];
    device.streamId = streamId;
    // Tell the system about the Device
    error = CMIOObjectsPublishedAndDied(PlugInRef(), kCMIOObjectSystemObject, 1, &deviceId, 0, 0);
    ...
    // Tell the system about the Stream
    error = CMIOObjectsPublishedAndDied(PlugInRef(), deviceId, 1, &streamId, 0, 0);
    ...
    ΠχγϟϥΠζ͕ݺͼग़͞ΕͨΒ

    σόΠε΍ετϦʔϜΛొ࿥͢Δɻ

    ͜ΕͰΧϝϥσόΠεͱͯ͠બ୒

    Ͱ͖ΔΑ͏ʹͳΔɻ

    View full-size slide

  20. CoreMediaIO DAL Plug-In
    CMSampleBufferRef buffer;
    err = CMIOSampleBufferCreateForImageBuffer(
    kCFAllocatorDefault,
    pixelBuffer,
    format,
    &timing,
    self.sequenceNumber,
    kCMIOSampleBufferNoDiscontinuities,
    &buffer
    );
    if (err != noErr) {
    DLog(@"CMIOSampleBufferCreateForImageBuffer err %d", err);
    }
    CMSimpleQueueEnqueue(self.queue, buffer);
    // Inform the clients that the queue has been altered
    if (self.alteredProc != NULL) {
    (self.alteredProc)(self.objectId, buffer, self.alteredRefCon);
    }
    ͋ͱ͸αϯϓϧόοϑΝΛ

    Ͳ͏ʹ͔ͯ͠࡞ͬͯ

    Ωϡʔʹ௥Ճ͍͚ͯͩ͘͠ɻ

    View full-size slide

  21. Development Tips

    View full-size slide

  22. Development Tips
    UnknownͷAVCaptureDeviceͱͯ͠ݟ͑Δ
    import Cocoa
    import AVFoundation
    class ViewController: NSViewController, AVCaptureVideoDataOutputSampleBufferDelegate {
    private let session = AVCaptureSession()
    override func viewDidLoad() {
    super.viewDidLoad()
    view.wantsLayer = true
    AVCaptureDevice.DiscoverySession(deviceTypes: [.externalUnknown], mediaType: .video, position: .unspecified).devices.forEach {
    do {
    let input = try AVCaptureDeviceInput(device: $0) // ଞʹ΋ϓϥάΠϯ͕͋Δ৔߹͸ద౰ʹௐ੔͢Δ
    if session.canAddInput(input) {
    session.addInput(input)
    }
    } catch {
    print(error)
    }
    }
    let previewLayer = AVCaptureVideoPreviewLayer()
    previewLayer.autoresizingMask = [.layerWidthSizable, .layerHeightSizable]
    previewLayer.session = session
    if let layer = view.layer {
    previewLayer.frame = layer.bounds
    layer.addSublayer(previewLayer)
    }
    session.startRunning()
    }
    }
    σόοά΍ಈ࡞֬ೝΛ؆୯ʹ͢Δʹ͸ɾɾɾ

    View full-size slide

  23. Development Tips
    UnknownͷAVCaptureDeviceͱͯ͠ݟ͑Δ
    import Cocoa
    import AVFoundation
    class ViewController: NSViewController, AVCaptureVideoDataOutputSampleBufferDelegate {
    private let session = AVCaptureSession()
    override func viewDidLoad() {
    super.viewDidLoad()
    view.wantsLayer = true
    AVCaptureDevice.DiscoverySession(deviceTypes: [.externalUnknown], mediaType: .video, position: .unspecified).devices.forEach {
    do {
    let input = try AVCaptureDeviceInput(device: $0) // ଞʹ΋ϓϥάΠϯ͕͋Δ৔߹͸ద౰ʹௐ੔͢Δ
    if session.canAddInput(input) {
    session.addInput(input)
    }
    } catch {
    print(error)
    }
    }
    let previewLayer = AVCaptureVideoPreviewLayer()
    previewLayer.autoresizingMask = [.layerWidthSizable, .layerHeightSizable]
    previewLayer.session = session
    if let layer = view.layer {
    previewLayer.frame = layer.bounds
    layer.addSublayer(previewLayer)
    }
    session.startRunning()
    }
    }

    View full-size slide

  24. Development Tips
    UnknownͷAVCaptureDeviceͱͯ͠ݟ͑Δ
    import Cocoa
    import AVFoundation
    class ViewController: NSViewController, AVCaptureVideoDataOutputSampleBufferDelegate {
    private let session = AVCaptureSession()
    override func viewDidLoad() {
    super.viewDidLoad()
    view.wantsLayer = true
    AVCaptureDevice.DiscoverySession(deviceTypes: [.externalUnknown], mediaType: .video, position: .unspecified).devices.forEach {
    do {
    let input = try AVCaptureDeviceInput(device: $0) // ଞʹ΋ϓϥάΠϯ͕͋Δ৔߹͸ద౰ʹௐ੔͢Δ
    if session.canAddInput(input) {
    session.addInput(input)
    }
    } catch {
    print(error)
    }
    }
    let previewLayer = AVCaptureVideoPreviewLayer()
    previewLayer.autoresizingMask = [.layerWidthSizable, .layerHeightSizable]
    previewLayer.session = session
    if let layer = view.layer {
    previewLayer.frame = layer.bounds
    layer.addSublayer(previewLayer)
    }
    session.startRunning()
    }
    }
    ϓϥάΠϯ΍ΤΫεςϯγϣϯͱ͍ͬͨιϑτ΢ΣΞ͸

    ͱʹ͔͘σόοά͕େม͕ͩɺ͜Ε͚ͩͰಉ͡ϓϩηεͰಈ͘ͷͰ

    ϩάͳͲ͕શ෦ݟ͑ΔɻPreviewLayerͰಈ࡞΋֬ೝͰ͖Δɻ

    ͨͩผͷΞϓϦͰಈ͔͢৔߹ͱڍಈ͕ҟͳΔ͜ͱ͕͋ΔͷͰద౓ʹ֬ೝ͢Δ

    View full-size slide

  25. Development Tips
    αϯϓϧόοϑΝΛCIImageʹͯ͠ҐஔΛม͑ͯ߹੒
    func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection:
    AVCaptureConnection) {
    if output == self.cameraCapture.output {
    guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }
    let cameraImage = CIImage(cvImageBuffer: imageBuffer)
    guard let screenImageBuffer = lastScreenImageBuffer else { return }
    let screenImage = CIImage(cvImageBuffer: screenImageBuffer)
    let translatedCameraImage = cameraImage.transformed(by: CGAffineTransform(translationX:
    screenImage.extent.width - cameraImage.extent.width, y: 0))
    let compositedImage = translatedCameraImage.composited(over: screenImage)
    ... αϯϓϧόοϑΝ͸ͱʹ͔͘CIImageʹͯ͠͠·͏ͱɺૢ࡞͕؆୯ɻ

    ߹੒΍֦େॖখɾҠಈ΋CIImageͷϝιουΛݺͿ͚ͩɻ

    ͜Ε͸΢Πϯυ΢ͷΩϟϓνϟͱΧϝϥͷө૾Λ߹੒͍ͯ͠Δɻ

    View full-size slide

  26. Development Tips
    αϯϓϧόοϑΝΛCIImageʹͯ͠ςΩετΛΦʔόʔϨΠ
    let windowImage = CGWindowListCreateImage(.null, .optionIncludingWindow, CGWindowID(windowID),
    [.bestResolution, .boundsIgnoreFraming]) {
    let ciImage = CIImage(cgImage: windowImage)
    if let text = self.settings["text"] as? String, !text.isEmpty {
    let bitmapImageRep = NSBitmapImageRep(ciImage: ciImage)
    let g = NSGraphicsContext(bitmapImageRep: bitmapImageRep)
    NSGraphicsContext.saveGraphicsState()
    NSGraphicsContext.current = g
    (text as NSString).draw(at: NSPoint(x: 200, y: ciImage.extent.height - 400), withAttributes: [.fon
    NSFont.boldSystemFont(ofSize: 400), .foregroundColor: NSColor.black])
    NSGraphicsContext.restoreGraphicsState()
    αϯϓϧόοϑΝʹςΩετΛࡌͤΔͷ΋

    CIImageʹͯ͠draw͢Ε͹OKɻ

    View full-size slide

  27. Development Tips
    εΫϦʔϯશମͷΩϟϓνϟ
    class ScreenCapture: NSObject {
    private let session = AVCaptureSession()
    let output = AVCaptureVideoDataOutput()
    override init() {
    session.sessionPreset = .high
    if let input = AVCaptureScreenInput(displayID: CGMainDisplayID()) {
    if session.canAddInput(input) {
    session.addInput(input)
    if session.canAddOutput(output) {
    session.addOutput(output)
    }
    }
    }
    }
    func startRunning() {
    session.startRunning()
    }
    func stopRunning() {
    session.stopRunning()
    }
    }
    εΫϦʔϯશମͷΩϟϓνϟʹ͸

    AVCaptureScreenInput͕༻ҙ͞Ε͍ͯΔɻ

    View full-size slide

  28. Development Tips
    ΢Πϯυ΢ͷҰཡ
    if let windowList = CGWindowListCopyWindowInfo([.optionOnScreenOnly, .excludeDesktopElements],
    kCGNullWindowID) {
    arrayController.content = (windowList as NSArray).filter{ (entry) -> Bool in
    if let entry = entry as? NSDictionary, let sharingState = entry[kCGWindowSharingState]
    as? Int, sharingState != CGWindowSharingType.none.rawValue {
    return true
    }
    return false
    }
    ΢Πϯυ΢͝ͱͷεΫϦʔϯγϣοτΛࡱΔʹ͸

    ·ͣ΢Πϯυ΢ͷҰཡ͔ΒIDΛऔಘͯ͠ɺ

    View full-size slide

  29. Development Tips
    ΢Πϯυ΢ͷΩϟϓνϟ
    if let windowID = self.settings["windowID"] as? Int,
    let windowImage = CGWindowListCreateImage(.null, .optionIncludingWindow,
    CGWindowID(windowID), [.bestResolution, .boundsIgnoreFraming]) {
    let ciImage = CIImage(cgImage: windowImage)
    ઐ༻ͷؔ਺ʹ΢Πϯυ΢IDΛ౉͢ɻ

    View full-size slide

  30. Development Tips
    Son of GrabʢGrabͱ͍͏Screenshot.appͷલ਎ͷΑ͏ͳΞϓϦͷαϯϓϧʣ
    https://developer.apple.com/library/archive/samplecode/SonOfGrab/
    Introduction/Intro.html#//apple_ref/doc/uid/DTS10004490-Intro-
    DontLinkElementID_2
    ΢Πϯυ΢͝ͱͷεΫϦʔϯγϣοτ͸

    ͜ͷαϯϓϧίʔυ͕ศརɻ

    XIBʹΤϥʔ͕ग़ΔͷͰ։͍ͯߋ৽͕ඞཁʢࣗಈʣɻ

    View full-size slide

  31. Development Tips
    σʔλͷड͚౉͠ʢϓϩηεؒ௨৴ʣ
    • File I/O

    • Distributed Notification

    • XPC Service

    • HTTPS (Maybe)

    • Unix IPC (Maybe)
    ઃఆΛม͑ͨΓ֎͔ΒσʔλΛ౉͍ͨ͠ͱ͍͏ͱ͖ɺ

    ϓϩηεؒ௨৴͕࢖͑Δʢ͸ͣʣɻ

    ͨͩFileܦ༝΍XPC Service͸ಈ͖·ͤΜͰͨ͠ɻ

    ΋ͱ΋ͱͷAppleͷαϯϓϧͰ͸Mach PortΛ࢖ͬͨ

    ϓϩηεؒ௨৴Λ͍ͯ͠ΔͷͰɺগͳ͘ͱ΋

    ͦΕ͸࢖͑Δʢ͸ͣʣ

    View full-size slide