Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
クロマキー合成を使い透過動画をAR空間に表示する
satoshi0212
September 06, 2019
Technology
3
7.5k
クロマキー合成を使い透過動画をAR空間に表示する
satoshi0212
September 06, 2019
Tweet
Share
More Decks by satoshi0212
See All by satoshi0212
NDIとARKitを連動させた新しい映像表現
satoshi0212
3
660
100日間AR表現を実装して見つけた面白い実装を全力解説
satoshi0212
5
1.5k
Working on mobile AR implementation, what I've implemented and beyond
satoshi0212
0
350
仮想カメラで切り開く拡張現実の世界
satoshi0212
0
390
macOS仮想カメラ「テロップカム」 実装方法とその先
satoshi0212
5
2.1k
ARで悪の組織の会議を実現する
satoshi0212
0
190
ARKit Maniacs
satoshi0212
1
2.9k
ARで作る価値のある物についての考察
satoshi0212
3
470
インタラクティブ画面遷移の実践的解説
satoshi0212
6
5.1k
Other Decks in Technology
See All in Technology
私見「UNIXの考え方」/20230124-kameda-unix-phylosophy
opelab
0
160
cdk deployに必要な権限ってなんだ?
kinyok
0
130
チケットNFTの仕組み
sbtechnight
0
330
Astroで始める爆速個人サイト開発
takanorip
12
8.4k
ML PM, DS PMってどんな仕事をしているの?
line_developers
PRO
1
210
ステート管理を超えるRecoil運用の考え方
uhyo
7
5.4k
ChatGPT for Hacking
anugrahsr
0
1k
20230121_データ分析系コミュニティ_サテライト企画
doradora09
0
510
lt53
98_justdoit
0
110
DNS権威サーバのクラウドサービス向けに行われた攻撃および対策 / DNS Pseudo-Random Subdomain Attack and mitigations
kazeburo
5
1.1k
230125 モニターマウントLT ITガジェット翁(Ryu.Cyber)さん
comucal
PRO
0
930
2022年に起きたフロントエンドの変化
sakito
29
17k
Featured
See All Featured
A better future with KSS
kneath
230
16k
Three Pipe Problems
jasonvnalue
89
8.9k
Building a Scalable Design System with Sketch
lauravandoore
451
31k
Embracing the Ebb and Flow
colly
75
3.6k
What’s in a name? Adding method to the madness
productmarketing
12
1.9k
Facilitating Awesome Meetings
lara
33
4.6k
Six Lessons from altMBA
skipperchong
15
2.3k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
236
1.1M
From Idea to $5000 a Month in 5 Months
shpigford
374
44k
Thoughts on Productivity
jonyablonski
49
2.7k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
13
1.1k
Keith and Marios Guide to Fast Websites
keithpitt
407
21k
Transcript
ΫϩϚΩʔ߹Λ͍ ಁաಈըΛ"3ۭؒʹදࣔ͢Δ TBUPTIJ෦ஐ "CFNB57 (JU)VCTBUPTIJ5XJUUFS!TINEFWFMPQ J04%$ NJO4FTTJPO !1
෦ஐ ݄ʹ"CFNB57ೖࣾɻ J04νʔϜͰΞϓϦ։ൃɻଟͷػೳΛ ΰϦΰϦ࣮ɻ ݄ΑΓ/FX%FWJDFνʔϜʹ ͯεϚʔτεϐʔΧʔ57σόΠεͳ Ͳ৽ن։Λத৺ͱͨ͠ௐࠪɾ։ൃΛ ୲ɻ αΠόʔΤʔδΣϯτࣾΤϯδχΞά ϧʔϓʮ93ΪϧυʯϦʔμʔɻ
!2
ఆಡऀ "3,JUαϯϓϧಈ͔ͨ͜͠ͱ͋Δ͘Β͍ .FUBMগ͔͠Δ .FUBM4IBEFSͬͨ͜ͱ͕ͳ͍ 6OJUZͰͳ͘ωΠςΟϒͰॻ͖͍ͨ !3
ఆಡऀ .FUBMͷֶशʹͪ͜Β͕͓͢͢Ί !4 IUUQTCPPUIQNKBJUFNT TIV͞Μ .FUBMೖ
ࠓͷτʔΫͰ͍͑ͨ͜ͱ !5
ࠓͷτʔΫͰ͍͑ͨ͜ͱ "3,JUΛ͍ͦͷதͰө૾Λ࠶ੜ͢Δํ๏ ͜ΕͰ৭ʑͰ͖ͦ͏ʂͱ͍͏ظײ !6 લͷࣗʹݟͤͨΒϚδͰتͿ༰ʂ ಈ͘αϯϓϧίʔυ͋ͬͯ͋Γ͕͍ͨͥʂ
ΫϩϚΩʔ߹ !7
ΫϩϚΩʔ߹ ग़యϑϦʔඦՊࣄయʰΟΩϖσΟΞʢ8JLJQFEJBʣʱ IUUQTKBXJLJQFEJBPSHXJLJ&"'&"%&&&"%&#$ ΫϩϚΩʔʢ$ISPNBLFZʣ͘͠ΫϩϚ Ωʔ߹ʢΫϩϚΩʔ͝͏͍ͤʣΩʔΠϯά ͷҰछͰɺಛఆͷ৭ͷ͔Βө૾ͷҰ෦Λಁ ໌ʹ͠ɺͦ͜ʹผͷө૾Λ߹͢Δٕज़ɻ !8
ग़యIUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPODPSFJNBHF
[email protected]
@
[email protected]
@F⒎FDU !9
ࠓճ࡞Δͷͪ͜Β !10
!11 ग़య.JLB3JLB0⒏DJBM άϦʔϯόοΫΞΠυϧɹ.JLB 3JLBIUUQTXXXZPVUVCFDPNXBUDI WX0OTBMKG"P
!12
࣍ !13
࣍ "3,JUΛ͍ͳ͕ΒಈըΛ࠶ੜ͢Δ ಁաಈըΛ࠶ੜ͢Δ ॲཧͷྲྀΕ ΫϩϚΩʔॲཧ.FUBMγΣʔμʔղઆ ͓·͚6*7JFXΛಁաදࣔͯ͠ΈΔ %FNP !14
࣍ "3,JUΛ͍ͳ͕ΒಈըΛ࠶ੜ͢Δ ಁաಈըΛ࠶ੜ͢Δ ॲཧͷྲྀΕ ΫϩϚΩʔॲཧ.FUBMγΣʔμʔղઆ ͓·͚6*7JFXΛಁաදࣔͯ͠ΈΔ %FNP !15
"3,JUΛ͍ͳ͕ΒಈըΛ࠶ੜ͢Δ !16
!17 4DFOF,JU 4QSJUF,JU .FUBM $POUFOU5FDIOPMPHZ
!18 4DFOF,JU 4QSJUF,JU .FUBM $POUFOU5FDIOPMPHZ ˞ࠓճ"34$/7JFXΛ༻ɻ 4$/3FOEFSFSͰඳըͨ͠Γ.FUBMͷΈͰඳը͢Δ͜ͱՄೳɻ
"3,JUΛ͍ͳ͕ΒಈըΛ࠶ੜ͢Δ !19
"3,JUΛ͍ͳ͕ΒಈըΛ࠶ੜ͢Δ 4QSJUF,JUͷ4,7JEFP/PEFಈըΛ࠶ੜͰ͖Δ !20
"3,JUΛ͍ͳ͕ΒಈըΛ࠶ੜ͢Δ 4QSJUF,JUͷ4,7JEFP/PEFಈըΛ࠶ੜͰ͖Δ !21 4DFOF,JUͷ4$//PEF4,7JEFP/PEFΛදࣔཁૉʹͰ͖Δ
"3,JUΛ͍ͳ͕ΒಈըΛ࠶ੜ͢Δ 4QSJUF,JUͷ4,7JEFP/PEFಈըΛ࠶ੜͰ͖Δ !22 4$//PEF"3ۭؒʹஔͰ͖Δ 4DFOF,JUͷ4$//PEF4,7JEFP/PEFΛදࣔཁૉʹͰ͖Δ
!23 4$/4DFOF 4$//PEF 4$/1MBOF 4,4DFOF 4$/.BUFSJBM 4,7JEFP/PEF "71MBZFS
func createVideoNode(size: CGFloat, videoUrl: URL) -> SCNNode { // αΠζ͕খ͍͞ͱϏσΦͷղ૾͕མͪΔ
let skSceneSize = CGSize(width: 1024, height: 1024) // AVPlayerੜ let avPlayer = AVPlayer(url: videoUrl) // SKVideoNodeੜ let skVideoNode = SKVideoNode(avPlayer: avPlayer) skVideoNode.position = CGPoint(x: skSceneSize.width / 2.0, y: skSceneSize.height / 2.0) skVideoNode.size = skSceneSize skVideoNode.yScale = -1.0 // ࠲ඪܥΛ্Լٯʹ͢Δ skVideoNode.play() // SKSceneੜ let skScene = SKScene(size: skSceneSize) skScene.addChild(skVideoNode) // SCNMaterialੜ let material = SCNMaterial() material.diffuse.contents = skScene material.isDoubleSided = true // SCNNodeੜ let node = SCNNode() // SCNPlane(=SCNGeometryΛܧঝͨ͠Ϋϥε)ੜ node.geometry = SCNPlane(width: size, height: size) node.geometry?.materials = [material] node.scale = SCNVector3(1, 0.5625, 1) return node } !24 "71MBZFS 4,7JEFP/PEF 4,4DFOF 4$/.BUFSJBM 4$/(FPNFUSZ 4$//PEF
func createVideoNode(size: CGFloat, videoUrl: URL) -> SCNNode { // αΠζ͕খ͍͞ͱϏσΦͷղ૾͕མͪΔ
let skSceneSize = CGSize(width: 1024, height: 1024) // AVPlayerੜ let avPlayer = AVPlayer(url: videoUrl) // SKVideoNodeੜ let skVideoNode = SKVideoNode(avPlayer: avPlayer) skVideoNode.position = CGPoint(x: skSceneSize.width / 2.0, y: skSceneSize.height / 2.0) skVideoNode.size = skSceneSize skVideoNode.yScale = -1.0 // ࠲ඪܥΛ্Լٯʹ͢Δ skVideoNode.play() // SKSceneੜ let skScene = SKScene(size: skSceneSize) skScene.addChild(skVideoNode) // SCNMaterialੜ let material = SCNMaterial() material.diffuse.contents = skScene material.isDoubleSided = true // SCNNodeੜ let node = SCNNode() // SCNPlane(=SCNGeometryΛܧঝͨ͠Ϋϥε)ੜ node.geometry = SCNPlane(width: size, height: size) node.geometry?.materials = [material] node.scale = SCNVector3(1, 0.5625, 1) return node } !25 "71MBZFS 4,7JEFP/PEF 4,4DFOF 4$/.BUFSJBM 4$/(FPNFUSZ 4$//PEF
func createVideoNode(size: CGFloat, videoUrl: URL) -> SCNNode { // αΠζ͕খ͍͞ͱϏσΦͷղ૾͕མͪΔ
let skSceneSize = CGSize(width: 1024, height: 1024) // AVPlayerੜ let avPlayer = AVPlayer(url: videoUrl) // SKVideoNodeੜ let skVideoNode = SKVideoNode(avPlayer: avPlayer) skVideoNode.position = CGPoint(x: skSceneSize.width / 2.0, y: skSceneSize.height / 2.0) skVideoNode.size = skSceneSize skVideoNode.yScale = -1.0 // ࠲ඪܥΛ্Լٯʹ͢Δ skVideoNode.play() // SKSceneੜ let skScene = SKScene(size: skSceneSize) skScene.addChild(skVideoNode) // SCNMaterialੜ let material = SCNMaterial() material.diffuse.contents = skScene material.isDoubleSided = true // SCNNodeੜ let node = SCNNode() // SCNPlane(=SCNGeometryΛܧঝͨ͠Ϋϥε)ੜ node.geometry = SCNPlane(width: size, height: size) node.geometry?.materials = [material] node.scale = SCNVector3(1, 0.5625, 1) return node } !26 "71MBZFS 4,7JEFP/PEF 4,4DFOF 4$/.BUFSJBM 4$/(FPNFUSZ 4$//PEF
func createVideoNode(size: CGFloat, videoUrl: URL) -> SCNNode { // αΠζ͕খ͍͞ͱϏσΦͷղ૾͕མͪΔ
let skSceneSize = CGSize(width: 1024, height: 1024) // AVPlayerੜ let avPlayer = AVPlayer(url: videoUrl) // SKVideoNodeੜ let skVideoNode = SKVideoNode(avPlayer: avPlayer) skVideoNode.position = CGPoint(x: skSceneSize.width / 2.0, y: skSceneSize.height / 2.0) skVideoNode.size = skSceneSize skVideoNode.yScale = -1.0 // ࠲ඪܥΛ্Լٯʹ͢Δ skVideoNode.play() // SKSceneੜ let skScene = SKScene(size: skSceneSize) skScene.addChild(skVideoNode) // SCNMaterialੜ let material = SCNMaterial() material.diffuse.contents = skScene material.isDoubleSided = true // SCNNodeੜ let node = SCNNode() // SCNPlane(=SCNGeometryΛܧঝͨ͠Ϋϥε)ੜ node.geometry = SCNPlane(width: size, height: size) node.geometry?.materials = [material] node.scale = SCNVector3(1, 0.5625, 1) return node } !27 "71MBZFS 4,7JEFP/PEF 4,4DFOF 4$/.BUFSJBM 4$/(FPNFUSZ 4$//PEF
func createVideoNode(size: CGFloat, videoUrl: URL) -> SCNNode { // αΠζ͕খ͍͞ͱϏσΦͷղ૾͕མͪΔ
let skSceneSize = CGSize(width: 1024, height: 1024) // AVPlayerੜ let avPlayer = AVPlayer(url: videoUrl) // SKVideoNodeੜ let skVideoNode = SKVideoNode(avPlayer: avPlayer) skVideoNode.position = CGPoint(x: skSceneSize.width / 2.0, y: skSceneSize.height / 2.0) skVideoNode.size = skSceneSize skVideoNode.yScale = -1.0 // ࠲ඪܥΛ্Լٯʹ͢Δ skVideoNode.play() // SKSceneੜ let skScene = SKScene(size: skSceneSize) skScene.addChild(skVideoNode) // SCNMaterialੜ let material = SCNMaterial() material.diffuse.contents = skScene material.isDoubleSided = true // SCNNodeੜ let node = SCNNode() // SCNPlane(=SCNGeometryΛܧঝͨ͠Ϋϥε)ੜ node.geometry = SCNPlane(width: size, height: size) node.geometry?.materials = [material] node.scale = SCNVector3(1, 0.5625, 1) return node } !28 "71MBZFS 4,7JEFP/PEF 4,4DFOF 4$/.BUFSJBM 4$/(FPNFUSZ 4$//PEF
func createVideoNode(size: CGFloat, videoUrl: URL) -> SCNNode { // αΠζ͕খ͍͞ͱϏσΦͷղ૾͕མͪΔ
let skSceneSize = CGSize(width: 1024, height: 1024) // AVPlayerੜ let avPlayer = AVPlayer(url: videoUrl) // SKVideoNodeੜ let skVideoNode = SKVideoNode(avPlayer: avPlayer) skVideoNode.position = CGPoint(x: skSceneSize.width / 2.0, y: skSceneSize.height / 2.0) skVideoNode.size = skSceneSize skVideoNode.yScale = -1.0 // ࠲ඪܥΛ্Լٯʹ͢Δ skVideoNode.play() // SKSceneੜ let skScene = SKScene(size: skSceneSize) skScene.addChild(skVideoNode) // SCNMaterialੜ let material = SCNMaterial() material.diffuse.contents = skScene material.isDoubleSided = true // SCNNodeੜ let node = SCNNode() // SCNPlane(=SCNGeometryΛܧঝͨ͠Ϋϥε)ੜ node.geometry = SCNPlane(width: size, height: size) node.geometry?.materials = [material] node.scale = SCNVector3(1, 0.5625, 1) return node } !29 "71MBZFS 4,7JEFP/PEF 4,4DFOF 4$/.BUFSJBM 4$/(FPNFUSZ 4$//PEF
override func viewDidLoad() { super.viewDidLoad() sceneView.delegate = self sceneView.scene =
SCNScene() let videoUrl = Bundle.main.url(forResource: "video001", withExtension: "mp4")! let videoNode = createVideoNode(size: 1, videoUrl: videoUrl) videoNode.position = SCNVector3(0, 0, -3.0) sceneView.scene.rootNode.addChildNode(videoNode) } !30
!31
!32
࣍ "3,JUΛ͍ͳ͕ΒಈըΛ࠶ੜ͢Δ ಁաಈըΛ࠶ੜ͢Δ ॲཧͷྲྀΕ ΫϩϚΩʔॲཧ.FUBMγΣʔμʔղઆ ͓·͚6*7JFXΛಁաදࣔͯ͠ΈΔ %FNP !33
࣍ "3,JUΛ͍ͳ͕ΒಈըΛ࠶ੜ͢Δ ಁաಈըΛ࠶ੜ͢Δ ॲཧͷྲྀΕ ΫϩϚΩʔॲཧ.FUBMγΣʔμʔղઆ ͓·͚6*7JFXΛಁաදࣔͯ͠ΈΔ %FNP !34
ॲཧͷྲྀΕ !35
!36 4$/4DFOF 4$//PEF 4$/1MBOF 4,4DFOF 4$/.BUFSJBM 4,7JEFP/PEF "71MBZFS
!37 4$/4DFOF 4$//PEF 4$/1MBOF 7JEFP.FUBM7JFX
!38 4$/4DFOF 4$//PEF 4$/1MBOF 7JEFP.FUBM7JFX .5,7JFXΛܧঝͨ͠ΧελϜΫϥε
!39 7JEFP.FUBM7JFX.5,7JFX "71MBZFS $*$POUFYU .5,7JFX GVODESBX ݱࡏͷಈըσʔλΛը૾Խ Ұ࣌.5,7JFXʹϨϯμϦϯά ΫϩϚΩʔγΣʔμʔॲཧ
override func draw(_ dirtyRect: CGRect) { guard let device =
device, let drawable = currentDrawable, let tempDrawable = bufferMtkView.currentDrawable, let image = makeCurrentVideoImage() else { return } ciContext.render(image, to: tempDrawable.texture, commandBuffer: nil, bounds: bounds, colorSpace: colorSpace) colorPixelFormat = tempDrawable.texture.pixelFormat let commandBuffer = commandQueue.makeCommandBuffer()! let commandEncoder = commandBuffer.makeComputeCommandEncoder()! commandEncoder.setComputePipelineState(pipelineState) commandEncoder.setTexture(tempDrawable.texture, index: 0) commandEncoder.setTexture(drawable.texture, index: 1) let factors: [Float] = [ 0, // red 1, // green 0, // blue 0.43, // threshold 0.11 // smoothing ] for i in 0..<factors.count { var factor = factors[i] let size = max(MemoryLayout<Float>.size, 16) let buffer = device.makeBuffer( bytes: &factor, length: size, options: [.storageModeShared] ) commandEncoder.setBuffer(buffer, offset: 0, index: i) } commandEncoder.dispatchThreadgroups(threadgroupsPerGrid, threadsPerThreadgroup: threadsPerThreadgroup) commandEncoder.endEncoding() commandBuffer.present(drawable) commandBuffer.commit() } !40
!41 7JEFP.FUBM7JFX.5,7JFX "71MBZFS $*$POUFYU .5,7JFX GVODESBX ݱࡏͷಈըσʔλΛը૾Խ Ұ࣌.5,7JFXʹϨϯμϦϯά ΫϩϚΩʔγΣʔμʔॲཧ
override func draw(_ dirtyRect: CGRect) { guard let device =
device, let drawable = currentDrawable, let tempDrawable = bufferMtkView.currentDrawable, let image = makeCurrentVideoImage() else { return } ciContext.render(image, to: tempDrawable.texture, commandBuffer: nil, bounds: bounds, colorSpace: colorSpace) colorPixelFormat = tempDrawable.texture.pixelFormat let commandBuffer = commandQueue.makeCommandBuffer()! let commandEncoder = commandBuffer.makeComputeCommandEncoder()! commandEncoder.setComputePipelineState(pipelineState) commandEncoder.setTexture(tempDrawable.texture, index: 0) commandEncoder.setTexture(drawable.texture, index: 1) let factors: [Float] = [ 0, // red 1, // green 0, // blue 0.43, // threshold 0.11 // smoothing ] for i in 0..<factors.count { var factor = factors[i] let size = max(MemoryLayout<Float>.size, 16) let buffer = device.makeBuffer( bytes: &factor, length: size, options: [.storageModeShared] ) commandEncoder.setBuffer(buffer, offset: 0, index: i) } commandEncoder.dispatchThreadgroups(threadgroupsPerGrid, threadsPerThreadgroup: threadsPerThreadgroup) commandEncoder.endEncoding() commandBuffer.present(drawable) commandBuffer.commit() } !42
private let videoOutput = AVPlayerItemVideoOutput.init(pixelBufferAttributes: [ kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange,
] as [String: Any]) func setupPlayer(url: URL) { player = AVPlayer(url: url) guard let player = player, let videoItem = player.currentItem else { return } videoItem.add(videoOutput) } private func makeCurrentVideoImage() -> CIImage? { guard let player = player, let videoItem = player.currentItem else { return nil } let time = videoItem.currentTime() guard videoOutput.hasNewPixelBuffer(forItemTime: time), let pixelBuffer = videoOutput.copyPixelBuffer(forItemTime: time, itemTimeForDisplay: nil) else { return nil } return CIImage(cvPixelBuffer: pixelBuffer) } !43
private let videoOutput = AVPlayerItemVideoOutput.init(pixelBufferAttributes: [ kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange,
] as [String: Any]) func setupPlayer(url: URL) { player = AVPlayer(url: url) guard let player = player, let videoItem = player.currentItem else { return } videoItem.add(videoOutput) } private func makeCurrentVideoImage() -> CIImage? { guard let player = player, let videoItem = player.currentItem else { return nil } let time = videoItem.currentTime() guard videoOutput.hasNewPixelBuffer(forItemTime: time), let pixelBuffer = videoOutput.copyPixelBuffer(forItemTime: time, itemTimeForDisplay: nil) else { return nil } return CIImage(cvPixelBuffer: pixelBuffer) } !44
!45 7JEFP.FUBM7JFX.5,7JFX "71MBZFS $*$POUFYU .5,7JFX GVODESBX ݱࡏͷಈըσʔλΛը૾Խ Ұ࣌.5,7JFXʹϨϯμϦϯά ΫϩϚΩʔγΣʔμʔॲཧ
override func draw(_ dirtyRect: CGRect) { guard let device =
device, let drawable = currentDrawable, let tempDrawable = bufferMtkView.currentDrawable, let image = makeCurrentVideoImage() else { return } ciContext.render(image, to: tempDrawable.texture, commandBuffer: nil, bounds: bounds, colorSpace: colorSpace) colorPixelFormat = tempDrawable.texture.pixelFormat let commandBuffer = commandQueue.makeCommandBuffer()! let commandEncoder = commandBuffer.makeComputeCommandEncoder()! commandEncoder.setComputePipelineState(pipelineState) commandEncoder.setTexture(tempDrawable.texture, index: 0) commandEncoder.setTexture(drawable.texture, index: 1) let factors: [Float] = [ 0, // red 1, // green 0, // blue 0.43, // threshold 0.11 // smoothing ] for i in 0..<factors.count { var factor = factors[i] let size = max(MemoryLayout<Float>.size, 16) let buffer = device.makeBuffer( bytes: &factor, length: size, options: [.storageModeShared] ) commandEncoder.setBuffer(buffer, offset: 0, index: i) } commandEncoder.dispatchThreadgroups(threadgroupsPerGrid, threadsPerThreadgroup: threadsPerThreadgroup) commandEncoder.endEncoding() commandBuffer.present(drawable) commandBuffer.commit() } !46
!47 7JEFP.FUBM7JFX.5,7JFX "71MBZFS $*$POUFYU .5,7JFX GVODESBX ݱࡏͷಈըσʔλΛը૾Խ Ұ࣌.5,7JFXʹϨϯμϦϯά ΫϩϚΩʔγΣʔμʔॲཧ
override func draw(_ dirtyRect: CGRect) { guard let device =
device, let drawable = currentDrawable, let tempDrawable = bufferMtkView.currentDrawable, let image = makeCurrentVideoImage() else { return } ciContext.render(image, to: tempDrawable.texture, commandBuffer: nil, bounds: bounds, colorSpace: colorSpace) colorPixelFormat = tempDrawable.texture.pixelFormat let commandBuffer = commandQueue.makeCommandBuffer()! let commandEncoder = commandBuffer.makeComputeCommandEncoder()! commandEncoder.setComputePipelineState(pipelineState) commandEncoder.setTexture(tempDrawable.texture, index: 0) commandEncoder.setTexture(drawable.texture, index: 1) let factors: [Float] = [ 0, // red 1, // green 0, // blue 0.43, // threshold 0.11 // smoothing ] for i in 0..<factors.count { var factor = factors[i] let size = max(MemoryLayout<Float>.size, 16) let buffer = device.makeBuffer( bytes: &factor, length: size, options: [.storageModeShared] ) commandEncoder.setBuffer(buffer, offset: 0, index: i) } commandEncoder.dispatchThreadgroups(threadgroupsPerGrid, threadsPerThreadgroup: threadsPerThreadgroup) commandEncoder.endEncoding() commandBuffer.present(drawable) commandBuffer.commit() } !48
࣍ "3,JUΛ͍ͳ͕ΒಈըΛ࠶ੜ͢Δ ಁաಈըΛ࠶ੜ͢Δ ॲཧͷྲྀΕ ΫϩϚΩʔॲཧ.FUBMγΣʔμʔղઆ ͓·͚6*7JFXΛಁաදࣔͯ͠ΈΔ %FNP !49
࣍ "3,JUΛ͍ͳ͕ΒಈըΛ࠶ੜ͢Δ ಁաಈըΛ࠶ੜ͢Δ ॲཧͷྲྀΕ ΫϩϚΩʔॲཧ.FUBMγΣʔμʔղઆ ͓·͚6*7JFXΛಁաදࣔͯ͠ΈΔ %FNP !50
.FUBMγΣʔμʔ !51
.FUBMγΣʔμʔ !52 छྨ ֓ཁ 7FSUFY ֤ͷ࠲ඪใΛฦ٫͢Δ'VODUJPO 'SBHNFOU ֤ͷ৭ใΛฦ٫͢Δ'VODUJPO $PNQVUF ,FSOFM
େنͳσʔλΛฒྻॲཧ͢Δࡍͷ൚༻తͳΧελϜ 'VODUJPO
γΣʔμʔͰͷΫϩϚΩʔ߹ॲཧ ,FSOFMͷγΣʔμʔͰ NBTL$PMPS ৭ Λ:$S$Cม ࠲ඪͷ3(#Λ:$S$Cม :$S$Cɺᮢɺ৭ͱͷڑʹΑΓಁաΛࢉग़ ࠲ඪͷ3(#Λࢉग़ͨ͠ಁաͰग़ྗ͢Δ৭Λܭࢉ !53
#include <metal_stdlib> using namespace metal; /// - SeeAlso: http://www.fundza.com/rman_shaders/smoothstep/index.html kernel
void ChromaKeyFilter( // ೖྗը૾ใ texture2d<float, access::read> inTexture [[ texture(0) ]], // ग़ྗը૾ใ texture2d<float, access::write> outTexture [[ texture(1) ]], // ϚεΫରͷ const device float *colorRed [[ buffer(0) ]], // ϚεΫରͷ const device float *colorGreen [[ buffer(1) ]], // ϚεΫରͷ੨ const device float *colorBlue [[ buffer(2) ]], // ಁա͕0-1ͷྖҬͷԼݶ const device float *threshold [[ buffer(3) ]], // ಁա͕0-1ͷྖҬͷԼݶ͔Β্ݶ·Ͱͷઈର const device float *smoothing [[ buffer(4) ]], // ࠲ඪใ uint2 gid [[ thread_position_in_grid ]]) { const float4 inColor = inTexture.read(gid); // maskͷରͷ৭ɻ৭Λಁա͢ΔͨΊmaskColorfloat3(0, 1, 0)ʹͳΔ const float3 maskColor = float3(*colorRed, *colorGreen, *colorBlue); // RGB͔ΒYͷࣹ const float3 YVector = float3(0.2989, 0.5866, 0.1145); // maskColor(৭)ΛYCrCbม const float maskY = dot(maskColor, YVector); const float maskCr = 0.7131 * (maskColor.r - maskY); const float maskCb = 0.5647 * (maskColor.b - maskY); // ࠲ඪͷRGBΛYCrCbม const float Y = dot(inColor.rgb, YVector); const float Cr = 0.7131 * (inColor.r - Y); const float Cb = 0.5647 * (inColor.b - Y); // YCrCbɺᮢɺ৭ͱͷڑʹΑΓಁաΛࢉग़ const float alpha = smoothstep( *threshold, *threshold + *smoothing, distance(float2(Cr, Cb), float2(maskCr, maskCb)) ); // ࠲ඪͷRGBΛࢉग़ͨ͠ಁաͰग़ྗ͢Δ৭Λܭࢉ const float4 outColor = alpha * float4(inColor.r, inColor.g, inColor.b, 1.0); outTexture.write(outColor, gid); } !54
#include <metal_stdlib> using namespace metal; /// - SeeAlso: http://www.fundza.com/rman_shaders/smoothstep/index.html kernel
void ChromaKeyFilter( // ೖྗը૾ใ texture2d<float, access::read> inTexture [[ texture(0) ]], // ग़ྗը૾ใ texture2d<float, access::write> outTexture [[ texture(1) ]], // ϚεΫରͷ const device float *colorRed [[ buffer(0) ]], // ϚεΫରͷ const device float *colorGreen [[ buffer(1) ]], // ϚεΫରͷ੨ const device float *colorBlue [[ buffer(2) ]], // ಁա͕0-1ͷྖҬͷԼݶ const device float *threshold [[ buffer(3) ]], // ಁա͕0-1ͷྖҬͷԼݶ͔Β্ݶ·Ͱͷઈର const device float *smoothing [[ buffer(4) ]], // ࠲ඪใ uint2 gid [[ thread_position_in_grid ]]) { !55
{ const float4 inColor = inTexture.read(gid); // maskͷରͷ৭ɻ৭Λಁա͢ΔͨΊmaskColorfloat3(0, 1, 0)ʹͳΔ
const float3 maskColor = float3(*colorRed, *colorGreen, *colorBlue); // RGB͔ΒYͷࣹ const float3 YVector = float3(0.2989, 0.5866, 0.1145); // maskColor(৭)ΛYCrCbม const float maskY = dot(maskColor, YVector); const float maskCr = 0.7131 * (maskColor.r - maskY); const float maskCb = 0.5647 * (maskColor.b - maskY); // ࠲ඪͷRGBΛYCrCbม const float Y = dot(inColor.rgb, YVector); const float Cr = 0.7131 * (inColor.r - Y); const float Cb = 0.5647 * (inColor.b - Y); // YCrCbɺᮢɺ৭ͱͷڑʹΑΓಁաΛࢉग़ const float alpha = smoothstep( *threshold, *threshold + *smoothing, distance(float2(Cr, Cb), float2(maskCr, maskCb)) ); // ࠲ඪͷRGBΛࢉग़ͨ͠ಁաͰग़ྗ͢Δ৭Λܭࢉ const float4 outColor = alpha * float4(inColor.r, inColor.g, inColor.b, 1.0); outTexture.write(outColor, gid); } !56
!57
!58 7JEFP.FUBM7JFX.5,7JFX "71MBZFS $*$POUFYU .5,7JFX GVODESBX ݱࡏͷಈըσʔλΛը૾Խ Ұ࣌.5,7JFXʹϨϯμϦϯά ΫϩϚΩʔγΣʔμʔॲཧ
!59
!60
࣍ "3,JUΛ͍ͳ͕ΒಈըΛ࠶ੜ͢Δ ಁաಈըΛ࠶ੜ͢Δ ॲཧͷྲྀΕ ΫϩϚΩʔॲཧ.FUBMγΣʔμʔղઆ ͓·͚6*7JFXΛಁաදࣔͯ͠ΈΔ %FNP !61
࣍ "3,JUΛ͍ͳ͕ΒಈըΛ࠶ੜ͢Δ ಁաಈըΛ࠶ੜ͢Δ ॲཧͷྲྀΕ ΫϩϚΩʔॲཧ.FUBMγΣʔμʔղઆ ͓·͚6*7JFXΛಁաදࣔͯ͠ΈΔ %FNP !62
7JFXΛಁաදࣔ͢Δ !63
7JFXΛಁաදࣔ͢Δ 6*7JFXΛը૾Խ͢Δख๏Ͱޭ ͨͩ͠ඳըύϑΥʔϚϯεམͪΔ !64
!65 7JEFP.FUBM7JFX.5,7JFX "71MBZFS $*$POUFYU .5,7JFX GVODESBX ݱࡏͷಈըσʔλΛը૾Խ Ұ࣌.5,7JFXʹϨϯμϦϯά ΫϩϚΩʔγΣʔμʔॲཧ
!66 7JEFP.FUBM7JFX.5,7JFX "71MBZFS $*$POUFYU .5,7JFX GVODESBX 6*7JFXΛը૾Խ Ұ࣌.5,7JFXʹϨϯμϦϯά ΫϩϚΩʔγΣʔμʔॲཧ
override func draw(_ dirtyRect: CGRect) { guard let device =
device, let drawable = currentDrawable, let tempDrawable = bufferMtkView.currentDrawable, let image = makeCurrentVideoImage() else { return } ciContext.render(image, to: tempDrawable.texture, commandBuffer: nil, bounds: bounds, colorSpace: colorSpace) colorPixelFormat = tempDrawable.texture.pixelFormat let commandBuffer = commandQueue.makeCommandBuffer()! let commandEncoder = commandBuffer.makeComputeCommandEncoder()! commandEncoder.setComputePipelineState(pipelineState) commandEncoder.setTexture(tempDrawable.texture, index: 0) commandEncoder.setTexture(drawable.texture, index: 1) let factors: [Float] = [ 0, // red 1, // green 0, // blue 0.43, // threshold 0.11 // smoothing ] for i in 0..<factors.count { var factor = factors[i] let size = max(MemoryLayout<Float>.size, 16) let buffer = device.makeBuffer( bytes: &factor, length: size, options: [.storageModeShared] ) commandEncoder.setBuffer(buffer, offset: 0, index: i) } commandEncoder.dispatchThreadgroups(threadgroupsPerGrid, threadsPerThreadgroup: threadsPerThreadgroup) commandEncoder.endEncoding() commandBuffer.present(drawable) commandBuffer.commit() } !67
override func draw(_ dirtyRect: CGRect) { guard let device =
device, let drawable = currentDrawable, let tempDrawable = bufferMtkView.currentDrawable, let image = makeCurrentViewImage() else { return } ciContext.render(image, to: tempDrawable.texture, commandBuffer: nil, bounds: bounds, colorSpace: colorSpace) colorPixelFormat = tempDrawable.texture.pixelFormat let commandBuffer = commandQueue.makeCommandBuffer()! let commandEncoder = commandBuffer.makeComputeCommandEncoder()! commandEncoder.setComputePipelineState(pipelineState) commandEncoder.setTexture(tempDrawable.texture, index: 0) commandEncoder.setTexture(drawable.texture, index: 1) let factors: [Float] = [ 0, // red 1, // green 0, // blue 0.43, // threshold 0.11 // smoothing ] for i in 0..<factors.count { var factor = factors[i] let size = max(MemoryLayout<Float>.size, 16) let buffer = device.makeBuffer( bytes: &factor, length: size, options: [.storageModeShared] ) commandEncoder.setBuffer(buffer, offset: 0, index: i) } commandEncoder.dispatchThreadgroups(threadgroupsPerGrid, threadsPerThreadgroup: threadsPerThreadgroup) commandEncoder.endEncoding() commandBuffer.present(drawable) commandBuffer.commit() } !68
func makeCurrentViewImage(view: UIView) -> CIImage? { UIGraphicsBeginImageContextWithOptions(view.frame.size, true, 0) view.drawHierarchy(in:
view.bounds, afterScreenUpdates: false) let image = UIGraphicsGetImageFromCurrentImageContext()! UIGraphicsEndImageContext(); return CIImage(image: image) } !69
8,8FC7JFXΛಁաͤ͞Δͱ !70
!71
!72
8FC35$ଓͨ͠7JFXΛಁաͤ͞Δͱ !73
!74
࣍ "3,JUΛ͍ͳ͕ΒಈըΛ࠶ੜ͢Δ ಁաಈըΛ࠶ੜ͢Δ ॲཧͷྲྀΕ ΫϩϚΩʔॲཧ.FUBMγΣʔμʔղઆ ͓·͚6*7JFXΛಁաදࣔͯ͠ΈΔ %FNP !75
࣍ "3,JUΛ͍ͳ͕ΒಈըΛ࠶ੜ͢Δ ಁաಈըΛ࠶ੜ͢Δ ॲཧͷྲྀΕ ΫϩϚΩʔॲཧ.FUBMγΣʔμʔղઆ ͓·͚6*7JFXΛಁաදࣔͯ͠ΈΔ %FNP !76
%FNP Χϝϥө૾ͷΫϩϚΩʔγΣʔμʔॲཧ ಁաಈը࠶ੜ !77
ิJ04Ҏ߱Ͱ )&7$7JEFPXJUI"MQIB લޙͷΧϝϥಉ࣌༻ "37JFX !78
αϯϓϧίʔυʹ͍ͭͯ $*'JMUFSͰͷ੩ࢭըΫϩϚΩʔॲཧ .FUBMͰͷಈըΫϩϚΩʔॲཧ "3,JUͱಈըදࣔ "3,JUͱಁաಈըදࣔ "3,JUͱ8FC35$ͱಁաදࣔ !79
ؔ࿈ใɾΫϨδοτ άϦʔϯόοΫΞΠυϧɹ.JLB 3JLB IUUQTXXXZPVUVCFDPNXBUDI WX0OTBMKG"P )PMPHSBQIJD1SPKFDUJPO IUUQTXXXTIBEFSUPZDPNWJFX.UK9[N <J04><"3,JU>ۙॴͷறंͰɺେܕσΟεϓϨΠͰϏσΦΛݟΔʂ IUUQTEFWDMBTTNFUIPEKQTNBSUQIPOFJQIPOFJPTBSLJUWJEFP "3,JUͷͨΊͷ%ֶ
IUUQTRJJUBDPNLCPZJUFNTGFGEEBDGC "3,JU4QSJUF,JUTFUQJYFM#V⒎FS"UUSJCVUFTUP4,7JEFP/PEFPSNBLFUSBOTQBSFOUQJYFMTJOWJEFP DISPNBLFZF⒎FDU BOPUIFSXBZ IUUQTTUBDLPWFSqPXDPNRVFTUJPOTBSLJUTQSJUFLJUTFUQJYFMCV⒎FSBUUSJCVUFTUPTLWJEFPOPEFPSNBLFUSBOTQBSFOU )&7$7JEFPXJUI"MQIB IUUQTEFWFMPQFSBQQMFDPNWJEFPTQMBZXXED !80
࡞Γ͍ͨͷ͜ΜͳΠϝʔδ !81
!82 ग़య,FJJDIJ.BUTVEB ):1&33&"-*5:IUUQTXXXZPVUVCFDPNXBUDI W:+HJW:[4T
!83 ग़యϤγμλΧϢΩ
[email protected]
@
[email protected]
@ IUUQTUXJUUFSDPN@@
[email protected]
@TUBUVT T
&OKPZJ04%$ IUUQTRJJUBDPNTBUPTIJJUFNTEFBFBDCFCFG !84 5XJUUFS !TINEFWFMPQ ϑΥϩʔɺථ͓ئ͍͠·͢ʂ ιʔείʔυͷϦϯΫͪ͜Β͔Β IUUQTHJUIVCDPNTBUPTIJ"3,JU$ISPNB,FZ4BNQMFT
ิଞʹͲ͏͍͏ํ๏͕͋Δͷ͔ !85
!86 ΫϩϚΩʔॲཧ$*'JMUFSΛDBQUVSF0VUQVU @EJE0VUQVUGSPN Ͱద༻ 4,&⒎FDU/PEFʹΫϩϚΩʔॲཧ$*'JMUFSઃఆ 4$/.BUFSJBMͷTIBEFS.PEJpFSTʹΫϩϚΩʔγΣʔμʔॲཧઃఆ
4$/.BUFSJBMʹΞϧϑΝઃఆγΣʔμʔॲཧɺUSBOTQBSFOUDPOUFOUTʹϚεΫಈըઃఆ όοϑΝྖҬʹΫϩϚΩʔγΣʔμʔద༻݁ՌΛอ࣋͠ඳըʹ༻
Ҋ֎৭ʑ͋ͬͨ !87