Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
iPhoneでコスプレをする技術
Search
Tatsuya Tanaka
September 13, 2018
Programming
3
1.1k
iPhoneでコスプレをする技術
iPhoneでコスプレをする技術
俺コン 2018 Summer
#orecon_ios
Tatsuya Tanaka
September 13, 2018
Tweet
Share
More Decks by Tatsuya Tanaka
See All by Tatsuya Tanaka
iPhoneのセンサー情報をmacOSアプリでリアルタイム活用するための技術
tattn
1
660
Better use of SwiftUI
tattn
0
450
Swift Concurrencyによる安全で快適な非同期処理
tattn
2
1.3k
iOSアプリの技術選択2022
tattn
7
3.9k
Widget Suggestions 対応と ヤフーの新OS対応
tattn
1
1.3k
WidgetKitで良い体験を作るには / Good experience with WidgetKit
tattn
2
1.7k
既存アプリにSwiftUIをどう組み込んでいくか
tattn
8
2.4k
iOS 14からのアプリ内課金
tattn
5
2.9k
iOS 14の位置情報系アップデート
tattn
0
22k
Other Decks in Programming
See All in Programming
Back to the Future: Let me tell you about the ACP protocol
terhechte
0
130
CSC509 Lecture 04
javiergs
PRO
0
290
Чего вы не знали о строках в Python – Василий Рябов, PythoNN
sobolevn
0
160
非同期jobをtransaction内で 呼ぶなよ!絶対に呼ぶなよ!
alstrocrack
0
520
Catch Up: Go Style Guide Update
andpad
0
170
dynamic!
moro
9
6.4k
株式会社 Sun terras カンパニーデック
sunterras
0
220
Introducing ReActionView: A new ActionView-Compatible ERB Engine @ Kaigi on Rails 2025, Tokyo, Japan
marcoroth
3
920
いま中途半端なSwift 6対応をするより、Default ActorやApproachable Concurrencyを有効にしてからでいいんじゃない?
yimajo
2
340
CSC509 Lecture 01
javiergs
PRO
1
430
プログラミングどうやる? ~テスト駆動開発から学ぶ達人の型~
a_okui
0
190
Pull-Requestの内容を1クリックで動作確認可能にするワークフロー
natmark
2
450
Featured
See All Featured
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
35
3.2k
How to Ace a Technical Interview
jacobian
280
23k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
2.6k
It's Worth the Effort
3n
187
28k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
162
15k
Writing Fast Ruby
sferik
629
62k
Context Engineering - Making Every Token Count
addyosmani
5
180
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
140
34k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
tammyeverts
9
580
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
45
2.5k
Why You Should Never Use an ORM
jnunemaker
PRO
59
9.6k
Transcript
iPhoneͰίεϓϨΛ͢Δٕज़ ాதୡ (@tattn) Զίϯ 2018 Summer #orecon_ios
ాத ୡ (@tattn) • Yahoo!Ҋ • iOSΞϓϦΤϯδχΞ @tattn @tanakasan2525 @tattn
ࡢͷApple Event؍·͔ͨ͠ʁ
iOS12ͷհϖʔδ IUUQTXXXBQQMFDPNKQJPTJPT
Ҋ͕ϖʔδʹࡌΓ·ͨ͠ IUUQTXXXBQQMFDPNKQJPTJPT
iPhoneͰίεϓϨΛ͢Δٕज़ IUUQTGPSUFFKQJPTEDKBQBOQSPQPTBMECGCCGCBBGCDGF
όʔνϟϧYouTuber ͍ͬͯ·͔͢ʁ
όʔνϟϧYouTuber (VTuber) Βͳ͍ਓϠϑʔͰݕࡧʂʮVTuberʯ YouTubeͳͲͷಈը৴αʔϏε্Ͱ 2D3DͷΞόλʔͰ৴͍ͯ͠Δਓͷ͜ͱɻ ϦΞϧͱόʔνϟϧͷମΛಉظ͢Δٕज़Λ͍ͬͯΔ͜ͱ͕ଟ͍ɻ
VTuberΛࢧ͑Δٕज़ • ମΛಈ͔͢ • खಈɺKinectɺVRػثɺϞʔγϣϯΩϟϓνϟʔػث • දΛಈ͔͢ • खಈɺFaceRigɺDlibɺVisionɺARKit •
(Λม͢Δ) • ιϑτΣΞ (࿀ͳͲ)ɺϋʔυΣΞ (VT-3ͳͲ) • දࣔ͢Δ • UnityɺFaceRigɺSceneKit (ҰྫͰ͢)
ٕज़తʹ໘ന͘ɺ৭ʑ࡞ͬͯΈ·ͨ͠ ※͜ΕOculus RiftͱUnityΛ͍ͬͯ·͢ VRδΣϯΨ VRΩϟονϘʔϧ VTuberͬΆ͍ࢹ ©Kizuna AI IUUQTUXJUUFSDPNUBOBLBTBOTUBUVT
εϚϗͰVTuberʹͳΕΔ
ͦ͏ɺiPhoneͳΒͶɻ
͜ΜͳΞϓϦΛ࡞Γ·ͨ͠ Α͔ͬͨΒ༡ΜͰΈͯͶˠ iPhone X × ARKit × SceneKit CoreAnimation ×
Accelerate ʮVTuberʯͰݕࡧ
༻ٕज़ Face Tracking ϨϯμϦϯάɾϞʔγϣϯ TrueDepthΧϝϥ ܭࢉ iPhone X × ARKit
× SceneKit CoreAnimation × Accelerate
ARKit ARFaceAnchorͰإͷ࢟ΛऔಘΛऔಘ BlendShapeLocationͰإͷύʔπͷঢ়ଶΛऔಘ IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPOBSLJUBSGBDFBODIPS
Face Trackingͷ࣮ߦ guard ARFaceTrackingConfiguration.isSupported else { return } let configuration
= ARFaceTrackingConfiguration() configuration.isLightEstimationEnabled = true let arSession = ARSession() arSession.delegate = self arSession.run(configuration, options: [.resetTracking, .removeExistingAnchors])
إͷύʔπͷঢ়ଶऔಘ extension ViewController: ARSessionDelegate { func session(_ session: ARSession, didUpdate
anchors: [ARAnchor]) { guard let faceAnchor = anchors .lazy .compactMap({ $0 as? ARFaceAnchor }) .first else { return } if let mouthOpenness = faceAnchor.blendShapes[.jawOpen]?.doubleValue { let e = 0.05 // ޱΛด͘͢͢͡Δ let mouthOpennessX8 = mouthOpenness * 8 // ޱΛ։͚͘͢͢Δ let value = mouthOpennessX8 < e ? 0 : mouthOpennessX8 > 1 - e ? 1 : mouthOpennessX8 // Ϟσϧͷߋ৽ }
SceneKit SCNSceneʹΩϟϥΫλʔΛஔ͠ SCNViewͰSceneΛදࣔ ©Kizuna AI
ϞσϧͷಡΈࠐΈ ϑϦʔͷϞσϧ͕ଟ͘ɺ දͷϞʔϑΟϯάϞʔγϣϯͳͲͷ ޓੑ͕ߴ͍MMDϞσϧΛࠓճ༻ ※MMD=MikuMikuDance IUUQEJDOJDPWJEFPKQBNJLVNJLVEBODF
MMDͷಡΈࠐΈ ϑΥʔϚοτͷ༷ࣗମެ։͞Ε͍ͯΔ IUUQLLILTFFTBBOFUDBUFHPSZIUNM ↑zipͷதʹ༷ॻ͕͋Δ IUUQTHJUIVCDPNNBHJDJFO..%4DFOF,JU ࠓճMMDSceneKitΛ༻
Ωϟϥͷύʔπͷऔಘ let model: SCNNode = try loadMMDModel(fromName: "kizunaai/kizunaai.pmx") let neck
= model.childNode(withName: "ट", recursively: true) let leftEye = model.childNode(withName: "ࠨ", recursively: true) let rightEye = model.childNode(withName: "ӈ", recursively: true) MMDͷNode (Ϙʔϯ) ໊ʹ҉తͳϓϩτίϧ͕͋ΔͷͰ ผͷϞσϧͦͷ··Ͱେମಈ͘ (ͨ·ʹશ֯ΧλΧφͱ֯ΧλΧφͷදهΏΕ͕...ţŜŖŪͳͲ)
CoreAnimation / SceneKit let keyPath = "morpher.weights.eye" var animation =
CAKeyframeAnimation(keyPath: keyPath) animation.values = [AnyObject]() animation.keyTimes = [NSNumber]() animation.timingFunctions = [CAMediaTimingFunction]() ... let animationGroup = CAAnimationGroup() animationGroup.animations = [CAAnimation]() ... animationGroup.animations?.append(animation) ... let animation = SCNAnimation(caAnimation: animationGroup) let player = SCNAnimationPlayer(animation: animation) model.addAnimationPlayer(player, forKey: nil) player.play()
Accelerate.framework ύϑΥʔϚϯεʹಛԽͨ͠ࢉज़ԋࢉϑϨʔϜϫʔΫ ࠓճ simd ͱ͍͏ খ͞ͳϕΫτϧͱߦྻΛѻ͏ϞδϡʔϧΛར༻ ※ simd = Single
Instruction Multiple Data
औಘͨ͠σʔλΛମͱटʹө͢Δ func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) { guard
let faceAnchor = anchors .lazy.compactMap({ $0 as? ARFaceAnchor }).first else { return } let transform: simd_float4x4 = faceAnchor.transform let quaternion = simd_quaternion(transform) neck.simdWorldOrientation = quaternion let translate4: simd_float4 = transform.columns.3 let translate = simd_float3(translate4.x * 5, translate4.y * 5, translate4.z * 50 + 20) // w = 1 model.simdPosition = translate ࠨӈ͋·Γಈ͔ͣɺલޙಈ͖͘͢͢Δ ARKit/SceneKitӈखܥ
͜Ε·Ͱઆ໌ٕͨ͠ज़ΛΈ߹ΘͤΔͱ όʔνϟϧͳମΛಘΔ͜ͱ͕Ͱ͖·͢ʂ
VRM
VRMͱ IUUQTEXBOHPHJUIVCJPWSN υϫϯΰ͕࡞ͬͨVR༻ͷ3DϞσϧϑΥʔϚοτ 4݄ʹެ։͞ΕɺVRք۾Ͱʹͳ͍ͬͯΔ ɾχίχཱମ ɾόʔνϟϧΩϟετ ɾcluster ɾVDraw ɾVΧπ ͳͲͰར༻͞Ε͍ͯΔ
VRMΛiOSͰಡΈࠐΊΔϥΠϒϥϦ VRMΛiOSͰಡΈࠐΊΔϥΠϒϥϦΛ࡞Γ·ͨ͠ IUUQTHJUIVCDPNUBUUO73.,JU VRMKit
VRMΛiOSͰಡΈࠐΊΔϥΠϒϥϦ IUUQTHJUIVCDPNUBUUO73.,JU VRMKit ϝλσʔλͷऔಘͱ Ϟσϧͷද͕ࣔՄೳ
͊͞ɺΈͳ͞Μόʔνϟϧͳੈք
͋Γ͕ͱ͏͍͟͝·ͨ͠