Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
仮想カメラで切り開く拡張現実の世界
satoshi0212
December 08, 2020
Programming
0
390
仮想カメラで切り開く拡張現実の世界
satoshi0212
December 08, 2020
Tweet
Share
More Decks by satoshi0212
See All by satoshi0212
NDIとARKitを連動させた新しい映像表現
satoshi0212
3
660
100日間AR表現を実装して見つけた面白い実装を全力解説
satoshi0212
5
1.5k
Working on mobile AR implementation, what I've implemented and beyond
satoshi0212
0
350
macOS仮想カメラ「テロップカム」 実装方法とその先
satoshi0212
5
2.1k
ARで悪の組織の会議を実現する
satoshi0212
0
190
クロマキー合成を使い透過動画をAR空間に表示する
satoshi0212
3
7.5k
ARKit Maniacs
satoshi0212
1
2.9k
ARで作る価値のある物についての考察
satoshi0212
3
470
インタラクティブ画面遷移の実践的解説
satoshi0212
6
5.1k
Other Decks in Programming
See All in Programming
Jetpack Compose 完全に理解した
mkeeda
1
440
Azure Functionsをサクッと開発、サクッとデプロイ/vscodeconf2023-baba
nina01
1
330
低レイヤーから始める GUI
fadis
18
9.3k
フロントエンドで学んだことをデータ分析で使ってみた話
daichi_igarashi
0
170
Most Valuable Bug(?) ~インシデント未遂から得た学び~
tatsumiakahori
0
140
Enumを自動で網羅的にテストしてみた
estie
0
1.2k
OIDC仕様に準拠した Makuake ID連携基盤構築の裏側
ymtdzzz
0
300
Spring BootとKubernetesで実現する今どきのDevOps入門
xblood
0
340
ECテックカンファレンス2023
kspace
1
220
量子コンピュータ時代のプログラミングセミナー / 20221222_Amplify_seminar _route_optimization
fixstars
0
240
ペパカレで入社した私が感じた2つのギャップと向き合い方
kosuke_ito
0
160
xarray-Datatree: Hierarchical Data Structures for Multi-Model Science
tomnicholas
0
200
Featured
See All Featured
Docker and Python
trallard
30
1.9k
What's new in Ruby 2.0
geeforr
336
30k
Pencils Down: Stop Designing & Start Developing
hursman
114
10k
From Idea to $5000 a Month in 5 Months
shpigford
374
44k
The Invisible Customer
myddelton
113
12k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
224
50k
Documentation Writing (for coders)
carmenintech
51
2.9k
Web development in the modern age
philhawksworth
197
9.6k
Rails Girls Zürich Keynote
gr2m
87
12k
For a Future-Friendly Web
brad_frost
166
7.7k
Raft: Consensus for Rubyists
vanstee
130
5.7k
5 minutes of I Can Smell Your CMS
philhawksworth
198
18k
Transcript
ԾΧϝϥͰΓ։֦͘ுݱ࣮ͷੈք 93,BJHJ ෦ஐ5XJUUFS!TINEFWFMPQ
ԾΧϝϥʹڵຯ͕͋ΔΤϯδχΞʹಧ͚ େܕελδΦͰͳ͘σεΫτοϓεϚʔτϑΥϯ ఆλʔήοτ
όʔνϟϧΧϝϥ࡞ΕΔʂָͦ͠͏ʂ ͱ͍͏ฏΛ։͍ͯ΄͍͠ʂ
࣮ํ๏ ݱ࣮֦ுͱͯ͠ͷར༻
ԾΧϝϥ 4OBQ$BNFSB NNINN
None
None
None
None
None
None
None
࣮ํ๏ ݱ࣮֦ுͱͯ͠ͷར༻
5XJUUFSͰ(8όʔνϟϧΧϝϥ࡞νϟϨϯδͰ࡞աఔπΠʔτ͍ͯ͠·͢
࡞ͬͨࡍͷۤ࿑ΛৼΓฦΓ
֎෦ϥΠϒϥϦͳ͠ɺ4XJGUͷΈͰ࣮Մೳ
ԾΧϝϥ࣮αϯϓϧ IUUQTHJUIVCDPNTBUPTIJ7JSUVBM$BNFSB4BNQMF
ߏཁૉ
ߏཁૉ ɹ$PSF.FEJB*0%"-1MVHJO ɹίϯτϩʔϥΞϓϦ
$PSF.FEJB*0%"- 1MVHJO ίϯτϩʔϥΞϓϦ /41BTUFCPBSE -JCSBSZ$PSF.FEJB*01MVH*OT%"- ԾΧϝϥͷ࣮ମ 6*Λ࣋ͨͳ͍1MVHJOʹ Λ͢ΞϓϦ
ϓϩδΣΫτ࡞ QMJTUใՃ ΤϯτϦʔϙΠϯτ࣮ ΠϯλʔϑΣΠε࣮ Χϝϥө૾ͷऔಘͱग़ྗ $PSF.FEJB*0%"-1MVHJO
ϓϩδΣΫτ࡞ QMJTUใՃ ΤϯτϦʔϙΠϯτ࣮ ΠϯλʔϑΣΠε࣮ Χϝϥө૾ͷऔಘͱग़ྗ $PSF.FEJB*0%"-1MVHJO
None
None
ϓϩδΣΫτ࡞ QMJTUใՃ ΤϯτϦʔϙΠϯτ࣮ ΠϯλʔϑΣΠε࣮ Χϝϥө૾Λग़ྗઃఆ $PSF.FEJB*0%"-1MVHJO
1MVHJOGBDUPSZJOUFSGBDFT 1MVHJOUZQFT ΛՃ
None
ࣗͰࢦఆ͢Δ*%ɻ66*%ͳͲઃఆ
ΤϯτϦʔϙΠϯτGVODUJPO໊
ԾΧϝϥ1MVH*Oݻఆ ઌड़ͷ66*%
ϓϩδΣΫτ࡞ QMJTUใՃ ΤϯτϦʔϙΠϯτ࣮ ΠϯλʔϑΣΠε࣮ Χϝϥө૾Λग़ྗઃఆ $PSF.FEJB*0%"-1MVHJO
import Foundation import CoreMediaIO @_cdecl("VirtualCameraSampleMain") func VirtualCameraSampleMain(allocator: CFAllocator, requestedTypeUUID: CFUUID)
-> CMIOHardwarePlugInRef { return pluginRef } Main.swift
import Foundation import CoreMediaIO @_cdecl("VirtualCameraSampleMain") func VirtualCameraSampleMain(allocator: CFAllocator, requestedTypeUUID: CFUUID)
-> CMIOHardwarePlugInRef { return pluginRef } Main.swift ΤϯτϦʔϙΠϯτ ΠϯλʔϑΣΠεͷࢀরΛฦ٫
ϓϩδΣΫτ࡞ QMJTUใՃ ΤϯτϦʔϙΠϯτ࣮ ΠϯλʔϑΣΠε࣮ Χϝϥө૾Λग़ྗઃఆ $PSF.FEJB*0%"-1MVHJO
private func createPluginInterface() -> CMIOHardwarePlugInInterface { return CMIOHardwarePlugInInterface( _reserved: nil,
QueryInterface: QueryInterface, AddRef: AddRef, Release: Release, Initialize: Initialize, InitializeWithObjectID: InitializeWithObjectID, Teardown: Teardown, ObjectShow: ObjectShow, ObjectHasProperty: ObjectHasProperty, ObjectIsPropertySettable: ObjectIsPropertySettable, ObjectGetPropertyDataSize: ObjectGetPropertyDataSize, ObjectGetPropertyData: ObjectGetPropertyData, ObjectSetPropertyData: ObjectSetPropertyData, DeviceSuspend: DeviceSuspend, DeviceResume: DeviceResume, DeviceStartStream: DeviceStartStream, DeviceStopStream: DeviceStopStream, DeviceProcessAVCCommand: DeviceProcessAVCCommand, DeviceProcessRS422Command: DeviceProcessRS422Command, StreamCopyBufferQueue: StreamCopyBufferQueue, StreamDeckPlay: StreamDeckPlay, StreamDeckStop: StreamDeckStop, StreamDeckJog: StreamDeckJog, StreamDeckCueTo: StreamDeckCueTo) } PluginInterface.swift (ൈਮ)
private func createPluginInterface() -> CMIOHardwarePlugInInterface { return CMIOHardwarePlugInInterface( _reserved: nil,
QueryInterface: QueryInterface, AddRef: AddRef, Release: Release, Initialize: Initialize, InitializeWithObjectID: InitializeWithObjectID, Teardown: Teardown, ObjectShow: ObjectShow, ObjectHasProperty: ObjectHasProperty, ObjectIsPropertySettable: ObjectIsPropertySettable, ObjectGetPropertyDataSize: ObjectGetPropertyDataSize, ObjectGetPropertyData: ObjectGetPropertyData, ObjectSetPropertyData: ObjectSetPropertyData, DeviceSuspend: DeviceSuspend, DeviceResume: DeviceResume, DeviceStartStream: DeviceStartStream, DeviceStopStream: DeviceStopStream, DeviceProcessAVCCommand: DeviceProcessAVCCommand, DeviceProcessRS422Command: DeviceProcessRS422Command, StreamCopyBufferQueue: StreamCopyBufferQueue, StreamDeckPlay: StreamDeckPlay, StreamDeckStop: StreamDeckStop, StreamDeckJog: StreamDeckJog, StreamDeckCueTo: StreamDeckCueTo) } PluginInterface.swift (ൈਮ) ΠϯλʔϑΣΠε͕ظ͢ΔͷΛฦ٫
PluginInterface.swift (ൈਮ) ৄࡉιʔεࢀর
ϓϩδΣΫτ࡞ QMJTUใՃ ΤϯτϦʔϙΠϯτ࣮ ΠϯλʔϑΣΠε࣮ Χϝϥө૾औಘͱग़ྗ $PSF.FEJB*0%"-1MVHJO
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, ɹɹɹɹɹɹɹɹɹɹɹ ɹɹfrom connection:
AVCaptureConnection) { if output == cameraCapture.output { guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return } let cameraImage = CIImage(cvImageBuffer: imageBuffer) let compositedImage = compose(bgImage: cameraImage, overlayImage: self.textImage) var pixelBuffer: CVPixelBuffer? _ = CVPixelBufferCreate( kCFAllocatorDefault, Int(compositedImage.extent.size.width), Int(compositedImage.extent.height), kCVPixelFormatType_32BGRA, self.CVPixelBufferCreateOptions as CFDictionary, &pixelBuffer ) if let pixelBuffer = pixelBuffer { context.render(compositedImage, to: pixelBuffer) delegate?.videoComposer(self, didComposeImageBuffer: pixelBuffer) } } } VideoComposer.swift (ൈਮ)
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, ɹɹɹɹɹɹɹɹɹɹɹ ɹɹfrom connection:
AVCaptureConnection) { if output == cameraCapture.output { guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return } let cameraImage = CIImage(cvImageBuffer: imageBuffer) let compositedImage = compose(bgImage: cameraImage, overlayImage: self.textImage) var pixelBuffer: CVPixelBuffer? _ = CVPixelBufferCreate( kCFAllocatorDefault, Int(compositedImage.extent.size.width), Int(compositedImage.extent.height), kCVPixelFormatType_32BGRA, self.CVPixelBufferCreateOptions as CFDictionary, &pixelBuffer ) if let pixelBuffer = pixelBuffer { context.render(compositedImage, to: pixelBuffer) delegate?.videoComposer(self, didComposeImageBuffer: pixelBuffer) } } } VideoComposer.swift (ൈਮ) ผ్ੜͨ͠ΦʔόʔϨΠςΩετը૾ͱ߹ EFMFHBUFܦ༝ͰQJYFM#V⒎FSΛ͢
private lazy var timer: DispatchSourceTimer = { let interval =
1.0 / Double(frameRate) let timer = DispatchSource.makeTimerSource() timer.schedule(deadline: .now() + interval, repeating: interval) timer.setEventHandler(handler: { [weak self] in self?.enqueueBuffer() }) return timer }() Stream.swift (ൈਮ)
private lazy var timer: DispatchSourceTimer = { let interval =
1.0 / Double(frameRate) let timer = DispatchSource.makeTimerSource() timer.schedule(deadline: .now() + interval, repeating: interval) timer.setEventHandler(handler: { [weak self] in self?.enqueueBuffer() }) return timer }() Stream.swift (ൈਮ) GSBNF3BUFࢦఆ͠ߋ৽ॲཧݺͼग़͠
private func enqueueBuffer() { (தུ) var sampleBufferUnmanaged: Unmanaged<CMSampleBuffer>? = nil
error = CMIOSampleBufferCreateForImageBuffer( kCFAllocatorDefault, pixelBuffer, formatDescription, &timing, sequenceNumber, UInt32(kCMIOSampleBufferNoDiscontinuities), &sampleBufferUnmanaged ) guard error == noErr else { log("CMIOSampleBufferCreateForImageBuffer Error: \(error)") return } CMSimpleQueueEnqueue(queue, element: sampleBufferUnmanaged!.toOpaque()) queueAlteredProc?(objectID, sampleBufferUnmanaged!.toOpaque(), queueAlteredRefCon) sequenceNumber += 1 } Stream.swift (ൈਮ)
private func enqueueBuffer() { (தུ) var sampleBufferUnmanaged: Unmanaged<CMSampleBuffer>? = nil
error = CMIOSampleBufferCreateForImageBuffer( kCFAllocatorDefault, pixelBuffer, formatDescription, &timing, sequenceNumber, UInt32(kCMIOSampleBufferNoDiscontinuities), &sampleBufferUnmanaged ) guard error == noErr else { log("CMIOSampleBufferCreateForImageBuffer Error: \(error)") return } CMSimpleQueueEnqueue(queue, element: sampleBufferUnmanaged!.toOpaque()) queueAlteredProc?(objectID, sampleBufferUnmanaged!.toOpaque(), queueAlteredRefCon) sequenceNumber += 1 } Stream.swift (ൈਮ) QJYFM#V⒎FS͔Βੜͨ͠ $.4BNQMF#V⒎FSΛRVFVFʹՃ
@IBAction func sendButton_action(_ sender: Any) { SettingsPasteboard.shared.settings["text1"] = mainTextField.stringValue SettingsPasteboard.shared.update()
} ViewController.swift (ൈਮ)
None
QMVHJOϑΝΠϧΛஔ
-JCSBSZ$PSF.FEJB*01MVH*OT%"-
දࣔ͞Εͨʂ
None
ߏཁૉ ɹ$PSF.FEJB*0%"-1MVHJO ɹίϯτϩʔϥΞϓϦ
ίϯτϩʔϥΞϓϦ
$PSF.FEJB*0%"- 1MVHJO ίϯτϩʔϥΞϓϦ /41BTUFCPBSE -JCSBSZ$PSF.FEJB*01MVH*OT%"- ԾΧϝϥͷ࣮ମ 6*Λ࣋ͨͳ͍1MVHJOʹ Λ͢ΞϓϦ
1BTUFCPBSEʹΛஔׂ͘
ϓϩδΣΫτ࡞ ίϯτϩʔϧஔ /41BTUFCPBSEͰσʔλڞ༗ ίϯτϩʔϥΞϓϦ
None
None
ϓϩδΣΫτ࡞ ίϯτϩʔϧஔ /41BTUFCPBSEͰσʔλڞ༗ ίϯτϩʔϥΞϓϦ
None
None
ϓϩδΣΫτ࡞ ίϯτϩʔϧஔ /41BTUFCPBSEͰσʔλڞ༗ ίϯτϩʔϥΞϓϦ
extension NSPasteboard.Name { static let main = NSPasteboard.Name(Config.mainAppBundleIdentifier) } extension
NSPasteboard.PasteboardType { static let plain = NSPasteboard.PasteboardType(rawValue: "public.utf8-plain-text") } class SettingsPasteboard { static let shared = SettingsPasteboard() open var settings = [String: Any]() open func current() -> [String: Any] { let pasteboard = NSPasteboard(name: .main) if let element = pasteboard.pasteboardItems?.last, let str = element.string(forType: .plain), let data = str.data(using: .utf8) { do { let json = try JSONSerialization.jsonObject(with: data, options : .allowFragments) as! [String: Any] settings = json } catch { print("can't convert json") } } return settings } open func update() { let jsonStr = stringify(json: settings) let pasteboard = NSPasteboard(name: .main) pasteboard.declareTypes([.string], owner: nil) pasteboard.setString(jsonStr, forType: .string) } ... } SettingsPasteboard.swift (ൈਮ)
None
$PSF.FEJB*0%"- 1MVHJO ίϯτϩʔϥΞϓϦ /41BTUFCPBSE -JCSBSZ$PSF.FEJB*01MVH*OT%"- ԾΧϝϥͷ࣮ମ 6*Λ࣋ͨͳ͍1MVHJOʹ Λ͢ΞϓϦ
ίϯτϩʔϥΞϓϦ͔ΒΛૹ৴
None
None
None
࣮ํ๏ ݱ࣮֦ுͱͯ͠ͷར༻
'BDFUJNFΧϝϥө૾ Իೖྗ ը૾Ճ "1*Ϩεϙϯε ΩʔϘʔυೖྗ ,FZOPUFը໘ FUD ֎෦ ԾΧϝϥ ग़ྗՃػߏ
ݱ࣮֦ுͱͯ͠ͷԾΧϝϥ ֤छೖྗ
NNINNϏοάϋϯυϞʔυ
Ή͜͏ͷ͘ʹϑΟϧλɺϚΠϖʔδɺϝλόʔε
ϑΟϧλҟٞ͋Γɺͬͨ
ϑΟϧλΠϩϞωΞత
UPOBSJԕִϦΞϧλΠϜө૾
ϚΠϯυϚοϓ
None
δΣωϨΠςΟϒදݱ
ελϯϓϦΞΫγϣϯ
Իೝࣝͱ༁
None
None
ݱ࣮֦ுͱͯ͠ͷԾΧϝϥ
ࢀߟ IUUQTEFWFMPQFSBQQMFDPNMJCSBSZBSDIJWFTBNQMFDPEF$PSF.FEJB*0*OUSPEVDUJPO*OUSPIUNM IUUQTHJUIVCDPNKPIOCPJMFTDPSFNFEJBJPEBMNJOJNBMFYBNQMF IUUQTHJUIVCDPNTFBODIBT4JNQMF%"-1MVHJO IUUQTTQFBLFSEFDLDPNLJTIJLBXBLBUTVNJWJSUVBMXFCDBNFSBXP[VPSPV
5XJUUFSͰ࠷৽ใൃ৴த !TINEFWFMPQ ԾΧϝϥ࣮αϯϓϧ IUUQTHJUIVCDPNTBUPTIJ7JSUVBM$BNFSB4BNQMF