Slide 1
Slide 1 text
Kishikawa Katsumi
Virtual Web CameraΛ࡞Ζ͏
CoreMediaIO Device Abstraction Layer Plug-In
Slide 2
Slide 2 text
Snap Camera
Slide 3
Slide 3 text
No content
Slide 4
Slide 4 text
Snap Camera Package Installer
Slide 5
Slide 5 text
Snap Camera Package Installer
Slide 6
Slide 6 text
Snap Camera Package Installer
Slide 7
Slide 7 text
No content
Slide 8
Slide 8 text
No content
Slide 9
Slide 9 text
CoreMediaIO DAL Plug-In
Slide 10
Slide 10 text
No content
Slide 11
Slide 11 text
Sample Code
• https://developer.apple.com/library/archive/samplecode/CoreMediaIO/Introduction/Intro.html
• https://github.com/lvsti/CoreMediaIO-DAL-Example
• https://github.com/johnboiles/obs-mac-virtualcam
• https://github.com/johnboiles/coremediaio-dal-minimal-example
• https://github.com/seanchas116/SimpleDALPlugin
• https://github.com/kishikawakatsumi/VirtualCameraComposer-Example
Slide 12
Slide 12 text
Apple's Sample Code
Slide 13
Slide 13 text
No content
Slide 14
Slide 14 text
No content
Slide 15
Slide 15 text
johnboiles/obs-mac-virtualcam
Slide 16
Slide 16 text
johnboiles/coremediaio-dal-minimal-example
Slide 17
Slide 17 text
seanchas116/SimpleDALPlugin
Slide 18
Slide 18 text
Virtual Camera
CoreMediaIO DAL Plug-In
Slide 19
Slide 19 text
CoreMediaIO DAL Plug-In
Slide 20
Slide 20 text
CoreMediaIO DAL Plug-In
ݻఆͷIDɻCoreMediaIOͷϔομʹॻ͍ͯ͋Δɻ
Slide 21
Slide 21 text
CoreMediaIO DAL Plug-In
ҙͷIDɻuuidgenͳͲͰదʹ࡞Δɻ
ॳظԽ࣌ʹ͞ΕΔɻ
Slide 22
Slide 22 text
CoreMediaIO DAL Plug-In
ΤϯτϦʔϙΠϯτɻ
ಉ໊͡લͷؔΛ༻ҙ͢Δɻ
Slide 23
Slide 23 text
CoreMediaIO DAL Plug-In
#import
#import "PlugInInterface.h"
#import "Logging.h"
//! PlugInMain is the entrypoint for the plugin
extern "C" {
void* PlugInMain(CFAllocatorRef allocator, CFUUIDRef requestedTypeUUID) {
DLogFunc(@"");
if (!CFEqual(requestedTypeUUID, kCMIOHardwarePlugInTypeID)) {
return 0;
}
return PlugInRef();
}
}
Slide 24
Slide 24 text
CoreMediaIO DAL Plug-In
#import
#import "PlugInInterface.h"
#import "Logging.h"
//! PlugInMain is the entrypoint for the plugin
extern "C" {
void* PlugInMain(CFAllocatorRef allocator, CFUUIDRef requestedTypeUUID) {
DLogFunc(@"");
if (!CFEqual(requestedTypeUUID, kCMIOHardwarePlugInTypeID)) {
return 0;
}
return PlugInRef();
}
}
C++Ͱॻ͘ͳΒϚϯάϦϯά͞Εͳ͍Α͏ʹ͢Δɻ
Slide 25
Slide 25 text
CoreMediaIO DAL Plug-In
import Foundation
import CoreMediaIO
@_cdecl("simpleDALPluginMain")
func simpleDALPluginMain(allocator: CFAllocator,
requestedTypeUUID: CFUUID) -> CMIOHardwarePlugInRef {
NSLog("simpleDALPluginMain")
return pluginRef
} SwiftͰಉ͡ɻ
Slide 26
Slide 26 text
CoreMediaIO DAL Plug-In
PlugIn *plugIn = [PlugIn SharedPlugIn];
plugIn.objectId = objectID;
Device *device = [[Device alloc] init];
CMIOObjectID deviceId;
error = CMIOObjectCreate(PlugInRef(), kCMIOObjectSystemObject, kCMIODeviceClassID, &deviceId);
...
Stream *stream = [[Stream alloc] init];
CMIOObjectID streamId;
error = CMIOObjectCreate(PlugInRef(), deviceId, kCMIOStreamClassID, &streamId);
...
stream.objectId = streamId;
[[ObjectStore SharedObjectStore] setObject:stream forObjectId:streamId];
device.streamId = streamId;
// Tell the system about the Device
error = CMIOObjectsPublishedAndDied(PlugInRef(), kCMIOObjectSystemObject, 1, &deviceId, 0, 0);
...
// Tell the system about the Stream
error = CMIOObjectsPublishedAndDied(PlugInRef(), deviceId, 1, &streamId, 0, 0);
...
ΠχγϟϥΠζ͕ݺͼग़͞ΕͨΒ
σόΠεετϦʔϜΛొ͢Δɻ
͜ΕͰΧϝϥσόΠεͱͯ͠બ
Ͱ͖ΔΑ͏ʹͳΔɻ
Slide 27
Slide 27 text
CoreMediaIO DAL Plug-In
CMSampleBufferRef buffer;
err = CMIOSampleBufferCreateForImageBuffer(
kCFAllocatorDefault,
pixelBuffer,
format,
&timing,
self.sequenceNumber,
kCMIOSampleBufferNoDiscontinuities,
&buffer
);
if (err != noErr) {
DLog(@"CMIOSampleBufferCreateForImageBuffer err %d", err);
}
CMSimpleQueueEnqueue(self.queue, buffer);
// Inform the clients that the queue has been altered
if (self.alteredProc != NULL) {
(self.alteredProc)(self.objectId, buffer, self.alteredRefCon);
}
͋ͱαϯϓϧόοϑΝΛ
Ͳ͏ʹ͔ͯ͠࡞ͬͯ
ΩϡʔʹՃ͍͚ͯͩ͘͠ɻ
Slide 28
Slide 28 text
Development Tips
Slide 29
Slide 29 text
Development Tips
UnknownͷAVCaptureDeviceͱͯ͠ݟ͑Δ
import Cocoa
import AVFoundation
class ViewController: NSViewController, AVCaptureVideoDataOutputSampleBufferDelegate {
private let session = AVCaptureSession()
override func viewDidLoad() {
super.viewDidLoad()
view.wantsLayer = true
AVCaptureDevice.DiscoverySession(deviceTypes: [.externalUnknown], mediaType: .video, position: .unspecified).devices.forEach {
do {
let input = try AVCaptureDeviceInput(device: $0) // ଞʹϓϥάΠϯ͕͋Δ߹దʹௐ͢Δ
if session.canAddInput(input) {
session.addInput(input)
}
} catch {
print(error)
}
}
let previewLayer = AVCaptureVideoPreviewLayer()
previewLayer.autoresizingMask = [.layerWidthSizable, .layerHeightSizable]
previewLayer.session = session
if let layer = view.layer {
previewLayer.frame = layer.bounds
layer.addSublayer(previewLayer)
}
session.startRunning()
}
}
σόοάಈ࡞֬ೝΛ؆୯ʹ͢Δʹɾɾɾ
Slide 30
Slide 30 text
Development Tips
UnknownͷAVCaptureDeviceͱͯ͠ݟ͑Δ
import Cocoa
import AVFoundation
class ViewController: NSViewController, AVCaptureVideoDataOutputSampleBufferDelegate {
private let session = AVCaptureSession()
override func viewDidLoad() {
super.viewDidLoad()
view.wantsLayer = true
AVCaptureDevice.DiscoverySession(deviceTypes: [.externalUnknown], mediaType: .video, position: .unspecified).devices.forEach {
do {
let input = try AVCaptureDeviceInput(device: $0) // ଞʹϓϥάΠϯ͕͋Δ߹దʹௐ͢Δ
if session.canAddInput(input) {
session.addInput(input)
}
} catch {
print(error)
}
}
let previewLayer = AVCaptureVideoPreviewLayer()
previewLayer.autoresizingMask = [.layerWidthSizable, .layerHeightSizable]
previewLayer.session = session
if let layer = view.layer {
previewLayer.frame = layer.bounds
layer.addSublayer(previewLayer)
}
session.startRunning()
}
}
Slide 31
Slide 31 text
Development Tips
UnknownͷAVCaptureDeviceͱͯ͠ݟ͑Δ
import Cocoa
import AVFoundation
class ViewController: NSViewController, AVCaptureVideoDataOutputSampleBufferDelegate {
private let session = AVCaptureSession()
override func viewDidLoad() {
super.viewDidLoad()
view.wantsLayer = true
AVCaptureDevice.DiscoverySession(deviceTypes: [.externalUnknown], mediaType: .video, position: .unspecified).devices.forEach {
do {
let input = try AVCaptureDeviceInput(device: $0) // ଞʹϓϥάΠϯ͕͋Δ߹దʹௐ͢Δ
if session.canAddInput(input) {
session.addInput(input)
}
} catch {
print(error)
}
}
let previewLayer = AVCaptureVideoPreviewLayer()
previewLayer.autoresizingMask = [.layerWidthSizable, .layerHeightSizable]
previewLayer.session = session
if let layer = view.layer {
previewLayer.frame = layer.bounds
layer.addSublayer(previewLayer)
}
session.startRunning()
}
}
ϓϥάΠϯΤΫεςϯγϣϯͱ͍ͬͨιϑτΣΞ
ͱʹ͔͘σόοά͕େม͕ͩɺ͜Ε͚ͩͰಉ͡ϓϩηεͰಈ͘ͷͰ
ϩάͳͲ͕શ෦ݟ͑ΔɻPreviewLayerͰಈ࡞֬ೝͰ͖Δɻ
ͨͩผͷΞϓϦͰಈ͔͢߹ͱڍಈ͕ҟͳΔ͜ͱ͕͋ΔͷͰదʹ֬ೝ͢Δ
Slide 32
Slide 32 text
Development Tips
αϯϓϧόοϑΝΛCIImageʹͯ͠ҐஔΛม͑ͯ߹
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection:
AVCaptureConnection) {
if output == self.cameraCapture.output {
guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }
let cameraImage = CIImage(cvImageBuffer: imageBuffer)
guard let screenImageBuffer = lastScreenImageBuffer else { return }
let screenImage = CIImage(cvImageBuffer: screenImageBuffer)
let translatedCameraImage = cameraImage.transformed(by: CGAffineTransform(translationX:
screenImage.extent.width - cameraImage.extent.width, y: 0))
let compositedImage = translatedCameraImage.composited(over: screenImage)
... αϯϓϧόοϑΝͱʹ͔͘CIImageʹͯ͠͠·͏ͱɺૢ࡞͕؆୯ɻ
߹֦େॖখɾҠಈCIImageͷϝιουΛݺͿ͚ͩɻ
͜ΕΠϯυͷΩϟϓνϟͱΧϝϥͷө૾Λ߹͍ͯ͠Δɻ
Slide 33
Slide 33 text
Development Tips
αϯϓϧόοϑΝΛCIImageʹͯ͠ςΩετΛΦʔόʔϨΠ
let windowImage = CGWindowListCreateImage(.null, .optionIncludingWindow, CGWindowID(windowID),
[.bestResolution, .boundsIgnoreFraming]) {
let ciImage = CIImage(cgImage: windowImage)
if let text = self.settings["text"] as? String, !text.isEmpty {
let bitmapImageRep = NSBitmapImageRep(ciImage: ciImage)
let g = NSGraphicsContext(bitmapImageRep: bitmapImageRep)
NSGraphicsContext.saveGraphicsState()
NSGraphicsContext.current = g
(text as NSString).draw(at: NSPoint(x: 200, y: ciImage.extent.height - 400), withAttributes: [.fon
NSFont.boldSystemFont(ofSize: 400), .foregroundColor: NSColor.black])
NSGraphicsContext.restoreGraphicsState()
αϯϓϧόοϑΝʹςΩετΛࡌͤΔͷ
CIImageʹͯ͠draw͢ΕOKɻ
Slide 34
Slide 34 text
Development Tips
εΫϦʔϯશମͷΩϟϓνϟ
class ScreenCapture: NSObject {
private let session = AVCaptureSession()
let output = AVCaptureVideoDataOutput()
override init() {
session.sessionPreset = .high
if let input = AVCaptureScreenInput(displayID: CGMainDisplayID()) {
if session.canAddInput(input) {
session.addInput(input)
if session.canAddOutput(output) {
session.addOutput(output)
}
}
}
}
func startRunning() {
session.startRunning()
}
func stopRunning() {
session.stopRunning()
}
}
εΫϦʔϯશମͷΩϟϓνϟʹ
AVCaptureScreenInput͕༻ҙ͞Ε͍ͯΔɻ
Slide 35
Slide 35 text
Development Tips
ΠϯυͷҰཡ
if let windowList = CGWindowListCopyWindowInfo([.optionOnScreenOnly, .excludeDesktopElements],
kCGNullWindowID) {
arrayController.content = (windowList as NSArray).filter{ (entry) -> Bool in
if let entry = entry as? NSDictionary, let sharingState = entry[kCGWindowSharingState]
as? Int, sharingState != CGWindowSharingType.none.rawValue {
return true
}
return false
}
Πϯυ͝ͱͷεΫϦʔϯγϣοτΛࡱΔʹ
·ͣΠϯυͷҰཡ͔ΒIDΛऔಘͯ͠ɺ
Slide 36
Slide 36 text
Development Tips
ΠϯυͷΩϟϓνϟ
if let windowID = self.settings["windowID"] as? Int,
let windowImage = CGWindowListCreateImage(.null, .optionIncludingWindow,
CGWindowID(windowID), [.bestResolution, .boundsIgnoreFraming]) {
let ciImage = CIImage(cgImage: windowImage)
ઐ༻ͷؔʹΠϯυIDΛ͢ɻ
Slide 37
Slide 37 text
Development Tips
Son of GrabʢGrabͱ͍͏Screenshot.appͷલͷΑ͏ͳΞϓϦͷαϯϓϧʣ
https://developer.apple.com/library/archive/samplecode/SonOfGrab/
Introduction/Intro.html#//apple_ref/doc/uid/DTS10004490-Intro-
DontLinkElementID_2
Πϯυ͝ͱͷεΫϦʔϯγϣοτ
͜ͷαϯϓϧίʔυ͕ศརɻ
XIBʹΤϥʔ͕ग़ΔͷͰ։͍ͯߋ৽͕ඞཁʢࣗಈʣɻ
Slide 38
Slide 38 text
Development Tips
σʔλͷड͚͠ʢϓϩηεؒ௨৴ʣ
• File I/O
• Distributed Notification
• XPC Service
• HTTPS (Maybe)
• Unix IPC (Maybe)
ઃఆΛม͑ͨΓ֎͔ΒσʔλΛ͍ͨ͠ͱ͍͏ͱ͖ɺ
ϓϩηεؒ௨৴͕͑Δʢͣʣɻ
ͨͩFileܦ༝XPC Serviceಈ͖·ͤΜͰͨ͠ɻ
ͱͱͷAppleͷαϯϓϧͰMach PortΛͬͨ
ϓϩηεؒ௨৴Λ͍ͯ͠ΔͷͰɺগͳ͘ͱ
ͦΕ͑Δʢͣʣ