Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Virtual Web Cameraを作ろう

Virtual Web Cameraを作ろう

Virtual Web Cameraを作ろう/Let's Build Virtual Webcam for macOS
CoreMediaIO Device Abstraction Layer Plug-In

Kishikawa Katsumi

April 27, 2020
Tweet

More Decks by Kishikawa Katsumi

Other Decks in Programming

Transcript

  1. Kishikawa Katsumi
    Virtual Web CameraΛ࡞Ζ͏
    CoreMediaIO Device Abstraction Layer Plug-In

    View Slide

  2. Snap Camera

    View Slide

  3. View Slide

  4. Snap Camera Package Installer

    View Slide

  5. Snap Camera Package Installer

    View Slide

  6. Snap Camera Package Installer

    View Slide

  7. View Slide

  8. View Slide

  9. CoreMediaIO DAL Plug-In

    View Slide

  10. View Slide

  11. Sample Code
    • https://developer.apple.com/library/archive/samplecode/CoreMediaIO/Introduction/Intro.html

    • https://github.com/lvsti/CoreMediaIO-DAL-Example

    • https://github.com/johnboiles/obs-mac-virtualcam

    • https://github.com/johnboiles/coremediaio-dal-minimal-example

    • https://github.com/seanchas116/SimpleDALPlugin

    • https://github.com/kishikawakatsumi/VirtualCameraComposer-Example

    View Slide

  12. Apple's Sample Code

    View Slide

  13. View Slide

  14. View Slide

  15. johnboiles/obs-mac-virtualcam

    View Slide

  16. johnboiles/coremediaio-dal-minimal-example

    View Slide

  17. seanchas116/SimpleDALPlugin

    View Slide

  18. Virtual Camera
    CoreMediaIO DAL Plug-In

    View Slide

  19. CoreMediaIO DAL Plug-In

    View Slide

  20. CoreMediaIO DAL Plug-In
    ݻఆͷIDɻCoreMediaIOͷϔομʹॻ͍ͯ͋Δɻ

    View Slide

  21. CoreMediaIO DAL Plug-In
    ೚ҙͷIDɻuuidgenͳͲͰద౰ʹ࡞Δɻ

    ॳظԽ࣌ʹ౉͞ΕΔɻ

    View Slide

  22. CoreMediaIO DAL Plug-In
    ΤϯτϦʔϙΠϯτɻ

    ಉ໊͡લͷؔ਺Λ༻ҙ͢Δɻ

    View Slide

  23. CoreMediaIO DAL Plug-In
    #import
    #import "PlugInInterface.h"
    #import "Logging.h"
    //! PlugInMain is the entrypoint for the plugin
    extern "C" {
    void* PlugInMain(CFAllocatorRef allocator, CFUUIDRef requestedTypeUUID) {
    DLogFunc(@"");
    if (!CFEqual(requestedTypeUUID, kCMIOHardwarePlugInTypeID)) {
    return 0;
    }
    return PlugInRef();
    }
    }

    View Slide

  24. CoreMediaIO DAL Plug-In
    #import
    #import "PlugInInterface.h"
    #import "Logging.h"
    //! PlugInMain is the entrypoint for the plugin
    extern "C" {
    void* PlugInMain(CFAllocatorRef allocator, CFUUIDRef requestedTypeUUID) {
    DLogFunc(@"");
    if (!CFEqual(requestedTypeUUID, kCMIOHardwarePlugInTypeID)) {
    return 0;
    }
    return PlugInRef();
    }
    }
    C++Ͱॻ͘ͳΒϚϯάϦϯά͞Εͳ͍Α͏ʹ͢Δɻ

    View Slide

  25. CoreMediaIO DAL Plug-In
    import Foundation
    import CoreMediaIO
    @_cdecl("simpleDALPluginMain")
    func simpleDALPluginMain(allocator: CFAllocator,
    requestedTypeUUID: CFUUID) -> CMIOHardwarePlugInRef {
    NSLog("simpleDALPluginMain")
    return pluginRef
    } SwiftͰ΋ಉ͡ɻ

    View Slide

  26. CoreMediaIO DAL Plug-In
    PlugIn *plugIn = [PlugIn SharedPlugIn];
    plugIn.objectId = objectID;
    Device *device = [[Device alloc] init];
    CMIOObjectID deviceId;
    error = CMIOObjectCreate(PlugInRef(), kCMIOObjectSystemObject, kCMIODeviceClassID, &deviceId);
    ...
    Stream *stream = [[Stream alloc] init];
    CMIOObjectID streamId;
    error = CMIOObjectCreate(PlugInRef(), deviceId, kCMIOStreamClassID, &streamId);
    ...
    stream.objectId = streamId;
    [[ObjectStore SharedObjectStore] setObject:stream forObjectId:streamId];
    device.streamId = streamId;
    // Tell the system about the Device
    error = CMIOObjectsPublishedAndDied(PlugInRef(), kCMIOObjectSystemObject, 1, &deviceId, 0, 0);
    ...
    // Tell the system about the Stream
    error = CMIOObjectsPublishedAndDied(PlugInRef(), deviceId, 1, &streamId, 0, 0);
    ...
    ΠχγϟϥΠζ͕ݺͼग़͞ΕͨΒ

    σόΠε΍ετϦʔϜΛొ࿥͢Δɻ

    ͜ΕͰΧϝϥσόΠεͱͯ͠બ୒

    Ͱ͖ΔΑ͏ʹͳΔɻ

    View Slide

  27. CoreMediaIO DAL Plug-In
    CMSampleBufferRef buffer;
    err = CMIOSampleBufferCreateForImageBuffer(
    kCFAllocatorDefault,
    pixelBuffer,
    format,
    &timing,
    self.sequenceNumber,
    kCMIOSampleBufferNoDiscontinuities,
    &buffer
    );
    if (err != noErr) {
    DLog(@"CMIOSampleBufferCreateForImageBuffer err %d", err);
    }
    CMSimpleQueueEnqueue(self.queue, buffer);
    // Inform the clients that the queue has been altered
    if (self.alteredProc != NULL) {
    (self.alteredProc)(self.objectId, buffer, self.alteredRefCon);
    }
    ͋ͱ͸αϯϓϧόοϑΝΛ

    Ͳ͏ʹ͔ͯ͠࡞ͬͯ

    Ωϡʔʹ௥Ճ͍͚ͯͩ͘͠ɻ

    View Slide

  28. Development Tips

    View Slide

  29. Development Tips
    UnknownͷAVCaptureDeviceͱͯ͠ݟ͑Δ
    import Cocoa
    import AVFoundation
    class ViewController: NSViewController, AVCaptureVideoDataOutputSampleBufferDelegate {
    private let session = AVCaptureSession()
    override func viewDidLoad() {
    super.viewDidLoad()
    view.wantsLayer = true
    AVCaptureDevice.DiscoverySession(deviceTypes: [.externalUnknown], mediaType: .video, position: .unspecified).devices.forEach {
    do {
    let input = try AVCaptureDeviceInput(device: $0) // ଞʹ΋ϓϥάΠϯ͕͋Δ৔߹͸ద౰ʹௐ੔͢Δ
    if session.canAddInput(input) {
    session.addInput(input)
    }
    } catch {
    print(error)
    }
    }
    let previewLayer = AVCaptureVideoPreviewLayer()
    previewLayer.autoresizingMask = [.layerWidthSizable, .layerHeightSizable]
    previewLayer.session = session
    if let layer = view.layer {
    previewLayer.frame = layer.bounds
    layer.addSublayer(previewLayer)
    }
    session.startRunning()
    }
    }
    σόοά΍ಈ࡞֬ೝΛ؆୯ʹ͢Δʹ͸ɾɾɾ

    View Slide

  30. Development Tips
    UnknownͷAVCaptureDeviceͱͯ͠ݟ͑Δ
    import Cocoa
    import AVFoundation
    class ViewController: NSViewController, AVCaptureVideoDataOutputSampleBufferDelegate {
    private let session = AVCaptureSession()
    override func viewDidLoad() {
    super.viewDidLoad()
    view.wantsLayer = true
    AVCaptureDevice.DiscoverySession(deviceTypes: [.externalUnknown], mediaType: .video, position: .unspecified).devices.forEach {
    do {
    let input = try AVCaptureDeviceInput(device: $0) // ଞʹ΋ϓϥάΠϯ͕͋Δ৔߹͸ద౰ʹௐ੔͢Δ
    if session.canAddInput(input) {
    session.addInput(input)
    }
    } catch {
    print(error)
    }
    }
    let previewLayer = AVCaptureVideoPreviewLayer()
    previewLayer.autoresizingMask = [.layerWidthSizable, .layerHeightSizable]
    previewLayer.session = session
    if let layer = view.layer {
    previewLayer.frame = layer.bounds
    layer.addSublayer(previewLayer)
    }
    session.startRunning()
    }
    }

    View Slide

  31. Development Tips
    UnknownͷAVCaptureDeviceͱͯ͠ݟ͑Δ
    import Cocoa
    import AVFoundation
    class ViewController: NSViewController, AVCaptureVideoDataOutputSampleBufferDelegate {
    private let session = AVCaptureSession()
    override func viewDidLoad() {
    super.viewDidLoad()
    view.wantsLayer = true
    AVCaptureDevice.DiscoverySession(deviceTypes: [.externalUnknown], mediaType: .video, position: .unspecified).devices.forEach {
    do {
    let input = try AVCaptureDeviceInput(device: $0) // ଞʹ΋ϓϥάΠϯ͕͋Δ৔߹͸ద౰ʹௐ੔͢Δ
    if session.canAddInput(input) {
    session.addInput(input)
    }
    } catch {
    print(error)
    }
    }
    let previewLayer = AVCaptureVideoPreviewLayer()
    previewLayer.autoresizingMask = [.layerWidthSizable, .layerHeightSizable]
    previewLayer.session = session
    if let layer = view.layer {
    previewLayer.frame = layer.bounds
    layer.addSublayer(previewLayer)
    }
    session.startRunning()
    }
    }
    ϓϥάΠϯ΍ΤΫεςϯγϣϯͱ͍ͬͨιϑτ΢ΣΞ͸

    ͱʹ͔͘σόοά͕େม͕ͩɺ͜Ε͚ͩͰಉ͡ϓϩηεͰಈ͘ͷͰ

    ϩάͳͲ͕શ෦ݟ͑ΔɻPreviewLayerͰಈ࡞΋֬ೝͰ͖Δɻ

    ͨͩผͷΞϓϦͰಈ͔͢৔߹ͱڍಈ͕ҟͳΔ͜ͱ͕͋ΔͷͰద౓ʹ֬ೝ͢Δ

    View Slide

  32. Development Tips
    αϯϓϧόοϑΝΛCIImageʹͯ͠ҐஔΛม͑ͯ߹੒
    func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection:
    AVCaptureConnection) {
    if output == self.cameraCapture.output {
    guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }
    let cameraImage = CIImage(cvImageBuffer: imageBuffer)
    guard let screenImageBuffer = lastScreenImageBuffer else { return }
    let screenImage = CIImage(cvImageBuffer: screenImageBuffer)
    let translatedCameraImage = cameraImage.transformed(by: CGAffineTransform(translationX:
    screenImage.extent.width - cameraImage.extent.width, y: 0))
    let compositedImage = translatedCameraImage.composited(over: screenImage)
    ... αϯϓϧόοϑΝ͸ͱʹ͔͘CIImageʹͯ͠͠·͏ͱɺૢ࡞͕؆୯ɻ

    ߹੒΍֦େॖখɾҠಈ΋CIImageͷϝιουΛݺͿ͚ͩɻ

    ͜Ε͸΢Πϯυ΢ͷΩϟϓνϟͱΧϝϥͷө૾Λ߹੒͍ͯ͠Δɻ

    View Slide

  33. Development Tips
    αϯϓϧόοϑΝΛCIImageʹͯ͠ςΩετΛΦʔόʔϨΠ
    let windowImage = CGWindowListCreateImage(.null, .optionIncludingWindow, CGWindowID(windowID),
    [.bestResolution, .boundsIgnoreFraming]) {
    let ciImage = CIImage(cgImage: windowImage)
    if let text = self.settings["text"] as? String, !text.isEmpty {
    let bitmapImageRep = NSBitmapImageRep(ciImage: ciImage)
    let g = NSGraphicsContext(bitmapImageRep: bitmapImageRep)
    NSGraphicsContext.saveGraphicsState()
    NSGraphicsContext.current = g
    (text as NSString).draw(at: NSPoint(x: 200, y: ciImage.extent.height - 400), withAttributes: [.fon
    NSFont.boldSystemFont(ofSize: 400), .foregroundColor: NSColor.black])
    NSGraphicsContext.restoreGraphicsState()
    αϯϓϧόοϑΝʹςΩετΛࡌͤΔͷ΋

    CIImageʹͯ͠draw͢Ε͹OKɻ

    View Slide

  34. Development Tips
    εΫϦʔϯશମͷΩϟϓνϟ
    class ScreenCapture: NSObject {
    private let session = AVCaptureSession()
    let output = AVCaptureVideoDataOutput()
    override init() {
    session.sessionPreset = .high
    if let input = AVCaptureScreenInput(displayID: CGMainDisplayID()) {
    if session.canAddInput(input) {
    session.addInput(input)
    if session.canAddOutput(output) {
    session.addOutput(output)
    }
    }
    }
    }
    func startRunning() {
    session.startRunning()
    }
    func stopRunning() {
    session.stopRunning()
    }
    }
    εΫϦʔϯશମͷΩϟϓνϟʹ͸

    AVCaptureScreenInput͕༻ҙ͞Ε͍ͯΔɻ

    View Slide

  35. Development Tips
    ΢Πϯυ΢ͷҰཡ
    if let windowList = CGWindowListCopyWindowInfo([.optionOnScreenOnly, .excludeDesktopElements],
    kCGNullWindowID) {
    arrayController.content = (windowList as NSArray).filter{ (entry) -> Bool in
    if let entry = entry as? NSDictionary, let sharingState = entry[kCGWindowSharingState]
    as? Int, sharingState != CGWindowSharingType.none.rawValue {
    return true
    }
    return false
    }
    ΢Πϯυ΢͝ͱͷεΫϦʔϯγϣοτΛࡱΔʹ͸

    ·ͣ΢Πϯυ΢ͷҰཡ͔ΒIDΛऔಘͯ͠ɺ

    View Slide

  36. Development Tips
    ΢Πϯυ΢ͷΩϟϓνϟ
    if let windowID = self.settings["windowID"] as? Int,
    let windowImage = CGWindowListCreateImage(.null, .optionIncludingWindow,
    CGWindowID(windowID), [.bestResolution, .boundsIgnoreFraming]) {
    let ciImage = CIImage(cgImage: windowImage)
    ઐ༻ͷؔ਺ʹ΢Πϯυ΢IDΛ౉͢ɻ

    View Slide

  37. Development Tips
    Son of GrabʢGrabͱ͍͏Screenshot.appͷલ਎ͷΑ͏ͳΞϓϦͷαϯϓϧʣ
    https://developer.apple.com/library/archive/samplecode/SonOfGrab/
    Introduction/Intro.html#//apple_ref/doc/uid/DTS10004490-Intro-
    DontLinkElementID_2
    ΢Πϯυ΢͝ͱͷεΫϦʔϯγϣοτ͸

    ͜ͷαϯϓϧίʔυ͕ศརɻ

    XIBʹΤϥʔ͕ग़ΔͷͰ։͍ͯߋ৽͕ඞཁʢࣗಈʣɻ

    View Slide

  38. Development Tips
    σʔλͷड͚౉͠ʢϓϩηεؒ௨৴ʣ
    • File I/O

    • Distributed Notification

    • XPC Service

    • HTTPS (Maybe)

    • Unix IPC (Maybe)
    ઃఆΛม͑ͨΓ֎͔ΒσʔλΛ౉͍ͨ͠ͱ͍͏ͱ͖ɺ

    ϓϩηεؒ௨৴͕࢖͑Δʢ͸ͣʣɻ

    ͨͩFileܦ༝΍XPC Service͸ಈ͖·ͤΜͰͨ͠ɻ

    ΋ͱ΋ͱͷAppleͷαϯϓϧͰ͸Mach PortΛ࢖ͬͨ

    ϓϩηεؒ௨৴Λ͍ͯ͠ΔͷͰɺগͳ͘ͱ΋

    ͦΕ͸࢖͑Δʢ͸ͣʣ

    View Slide