Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
iPhoneでコスプレをする技術
Search
Tatsuya Tanaka
September 13, 2018
Programming
3
1k
iPhoneでコスプレをする技術
iPhoneでコスプレをする技術
俺コン 2018 Summer
#orecon_ios
Tatsuya Tanaka
September 13, 2018
Tweet
Share
More Decks by Tatsuya Tanaka
See All by Tatsuya Tanaka
iPhoneのセンサー情報をmacOSアプリでリアルタイム活用するための技術
tattn
1
630
Better use of SwiftUI
tattn
0
440
Swift Concurrencyによる安全で快適な非同期処理
tattn
2
1.3k
iOSアプリの技術選択2022
tattn
7
3.8k
Widget Suggestions 対応と ヤフーの新OS対応
tattn
1
1.3k
WidgetKitで良い体験を作るには / Good experience with WidgetKit
tattn
2
1.6k
既存アプリにSwiftUIをどう組み込んでいくか
tattn
8
2.4k
iOS 14からのアプリ内課金
tattn
5
2.9k
iOS 14の位置情報系アップデート
tattn
0
22k
Other Decks in Programming
See All in Programming
Railsアプリケーションと パフォーマンスチューニング ー 秒間5万リクエストの モバイルオーダーシステムを支える事例 ー Rubyセミナー 大阪
falcon8823
5
1.4k
なぜ適用するか、移行して理解するClean Architecture 〜構造を超えて設計を継承する〜 / Why Apply, Migrate and Understand Clean Architecture - Inherit Design Beyond Structure
seike460
PRO
3
790
Node-RED を(HTTP で)つなげる MCP サーバーを作ってみた
highu
0
120
「テストは愚直&&網羅的に書くほどよい」という誤解 / Test Smarter, Not Harder
munetoshi
0
190
High-Level Programming Languages in AI Era -Human Thought and Mind-
hayat01sh1da
PRO
0
840
AI駆動のマルチエージェントによる業務フロー自動化の設計と実践
h_okkah
0
210
Goで作る、開発・CI環境
sin392
0
260
PHPで始める振る舞い駆動開発(Behaviour-Driven Development)
ohmori_yusuke
2
420
システム成長を止めない!本番無停止テーブル移行の全貌
sakawe_ee
1
220
Webの外へ飛び出せ NativePHPが切り拓くPHPの未来
takuyakatsusa
2
580
猫と暮らす Google Nest Cam生活🐈 / WebRTC with Google Nest Cam
yutailang0119
0
160
技術同人誌をMCP Serverにしてみた
74th
1
680
Featured
See All Featured
Making Projects Easy
brettharned
116
6.3k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.4k
A designer walks into a library…
pauljervisheath
207
24k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
331
22k
Writing Fast Ruby
sferik
628
62k
Statistics for Hackers
jakevdp
799
220k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
138
34k
The Straight Up "How To Draw Better" Workshop
denniskardys
235
140k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
251
21k
Imperfection Machines: The Place of Print at Facebook
scottboms
267
13k
Transcript
iPhoneͰίεϓϨΛ͢Δٕज़ ాதୡ (@tattn) Զίϯ 2018 Summer #orecon_ios
ాத ୡ (@tattn) • Yahoo!Ҋ • iOSΞϓϦΤϯδχΞ @tattn @tanakasan2525 @tattn
ࡢͷApple Event؍·͔ͨ͠ʁ
iOS12ͷհϖʔδ IUUQTXXXBQQMFDPNKQJPTJPT
Ҋ͕ϖʔδʹࡌΓ·ͨ͠ IUUQTXXXBQQMFDPNKQJPTJPT
iPhoneͰίεϓϨΛ͢Δٕज़ IUUQTGPSUFFKQJPTEDKBQBOQSPQPTBMECGCCGCBBGCDGF
όʔνϟϧYouTuber ͍ͬͯ·͔͢ʁ
όʔνϟϧYouTuber (VTuber) Βͳ͍ਓϠϑʔͰݕࡧʂʮVTuberʯ YouTubeͳͲͷಈը৴αʔϏε্Ͱ 2D3DͷΞόλʔͰ৴͍ͯ͠Δਓͷ͜ͱɻ ϦΞϧͱόʔνϟϧͷମΛಉظ͢Δٕज़Λ͍ͬͯΔ͜ͱ͕ଟ͍ɻ
VTuberΛࢧ͑Δٕज़ • ମΛಈ͔͢ • खಈɺKinectɺVRػثɺϞʔγϣϯΩϟϓνϟʔػث • දΛಈ͔͢ • खಈɺFaceRigɺDlibɺVisionɺARKit •
(Λม͢Δ) • ιϑτΣΞ (࿀ͳͲ)ɺϋʔυΣΞ (VT-3ͳͲ) • දࣔ͢Δ • UnityɺFaceRigɺSceneKit (ҰྫͰ͢)
ٕज़తʹ໘ന͘ɺ৭ʑ࡞ͬͯΈ·ͨ͠ ※͜ΕOculus RiftͱUnityΛ͍ͬͯ·͢ VRδΣϯΨ VRΩϟονϘʔϧ VTuberͬΆ͍ࢹ ©Kizuna AI IUUQTUXJUUFSDPNUBOBLBTBOTUBUVT
εϚϗͰVTuberʹͳΕΔ
ͦ͏ɺiPhoneͳΒͶɻ
͜ΜͳΞϓϦΛ࡞Γ·ͨ͠ Α͔ͬͨΒ༡ΜͰΈͯͶˠ iPhone X × ARKit × SceneKit CoreAnimation ×
Accelerate ʮVTuberʯͰݕࡧ
༻ٕज़ Face Tracking ϨϯμϦϯάɾϞʔγϣϯ TrueDepthΧϝϥ ܭࢉ iPhone X × ARKit
× SceneKit CoreAnimation × Accelerate
ARKit ARFaceAnchorͰإͷ࢟ΛऔಘΛऔಘ BlendShapeLocationͰإͷύʔπͷঢ়ଶΛऔಘ IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPOBSLJUBSGBDFBODIPS
Face Trackingͷ࣮ߦ guard ARFaceTrackingConfiguration.isSupported else { return } let configuration
= ARFaceTrackingConfiguration() configuration.isLightEstimationEnabled = true let arSession = ARSession() arSession.delegate = self arSession.run(configuration, options: [.resetTracking, .removeExistingAnchors])
إͷύʔπͷঢ়ଶऔಘ extension ViewController: ARSessionDelegate { func session(_ session: ARSession, didUpdate
anchors: [ARAnchor]) { guard let faceAnchor = anchors .lazy .compactMap({ $0 as? ARFaceAnchor }) .first else { return } if let mouthOpenness = faceAnchor.blendShapes[.jawOpen]?.doubleValue { let e = 0.05 // ޱΛด͘͢͢͡Δ let mouthOpennessX8 = mouthOpenness * 8 // ޱΛ։͚͘͢͢Δ let value = mouthOpennessX8 < e ? 0 : mouthOpennessX8 > 1 - e ? 1 : mouthOpennessX8 // Ϟσϧͷߋ৽ }
SceneKit SCNSceneʹΩϟϥΫλʔΛஔ͠ SCNViewͰSceneΛදࣔ ©Kizuna AI
ϞσϧͷಡΈࠐΈ ϑϦʔͷϞσϧ͕ଟ͘ɺ දͷϞʔϑΟϯάϞʔγϣϯͳͲͷ ޓੑ͕ߴ͍MMDϞσϧΛࠓճ༻ ※MMD=MikuMikuDance IUUQEJDOJDPWJEFPKQBNJLVNJLVEBODF
MMDͷಡΈࠐΈ ϑΥʔϚοτͷ༷ࣗମެ։͞Ε͍ͯΔ IUUQLLILTFFTBBOFUDBUFHPSZIUNM ↑zipͷதʹ༷ॻ͕͋Δ IUUQTHJUIVCDPNNBHJDJFO..%4DFOF,JU ࠓճMMDSceneKitΛ༻
Ωϟϥͷύʔπͷऔಘ let model: SCNNode = try loadMMDModel(fromName: "kizunaai/kizunaai.pmx") let neck
= model.childNode(withName: "ट", recursively: true) let leftEye = model.childNode(withName: "ࠨ", recursively: true) let rightEye = model.childNode(withName: "ӈ", recursively: true) MMDͷNode (Ϙʔϯ) ໊ʹ҉తͳϓϩτίϧ͕͋ΔͷͰ ผͷϞσϧͦͷ··Ͱେମಈ͘ (ͨ·ʹશ֯ΧλΧφͱ֯ΧλΧφͷදهΏΕ͕...ţŜŖŪͳͲ)
CoreAnimation / SceneKit let keyPath = "morpher.weights.eye" var animation =
CAKeyframeAnimation(keyPath: keyPath) animation.values = [AnyObject]() animation.keyTimes = [NSNumber]() animation.timingFunctions = [CAMediaTimingFunction]() ... let animationGroup = CAAnimationGroup() animationGroup.animations = [CAAnimation]() ... animationGroup.animations?.append(animation) ... let animation = SCNAnimation(caAnimation: animationGroup) let player = SCNAnimationPlayer(animation: animation) model.addAnimationPlayer(player, forKey: nil) player.play()
Accelerate.framework ύϑΥʔϚϯεʹಛԽͨ͠ࢉज़ԋࢉϑϨʔϜϫʔΫ ࠓճ simd ͱ͍͏ খ͞ͳϕΫτϧͱߦྻΛѻ͏ϞδϡʔϧΛར༻ ※ simd = Single
Instruction Multiple Data
औಘͨ͠σʔλΛମͱटʹө͢Δ func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) { guard
let faceAnchor = anchors .lazy.compactMap({ $0 as? ARFaceAnchor }).first else { return } let transform: simd_float4x4 = faceAnchor.transform let quaternion = simd_quaternion(transform) neck.simdWorldOrientation = quaternion let translate4: simd_float4 = transform.columns.3 let translate = simd_float3(translate4.x * 5, translate4.y * 5, translate4.z * 50 + 20) // w = 1 model.simdPosition = translate ࠨӈ͋·Γಈ͔ͣɺલޙಈ͖͘͢͢Δ ARKit/SceneKitӈखܥ
͜Ε·Ͱઆ໌ٕͨ͠ज़ΛΈ߹ΘͤΔͱ όʔνϟϧͳମΛಘΔ͜ͱ͕Ͱ͖·͢ʂ
VRM
VRMͱ IUUQTEXBOHPHJUIVCJPWSN υϫϯΰ͕࡞ͬͨVR༻ͷ3DϞσϧϑΥʔϚοτ 4݄ʹެ։͞ΕɺVRք۾Ͱʹͳ͍ͬͯΔ ɾχίχཱମ ɾόʔνϟϧΩϟετ ɾcluster ɾVDraw ɾVΧπ ͳͲͰར༻͞Ε͍ͯΔ
VRMΛiOSͰಡΈࠐΊΔϥΠϒϥϦ VRMΛiOSͰಡΈࠐΊΔϥΠϒϥϦΛ࡞Γ·ͨ͠ IUUQTHJUIVCDPNUBUUO73.,JU VRMKit
VRMΛiOSͰಡΈࠐΊΔϥΠϒϥϦ IUUQTHJUIVCDPNUBUUO73.,JU VRMKit ϝλσʔλͷऔಘͱ Ϟσϧͷද͕ࣔՄೳ
͊͞ɺΈͳ͞Μόʔνϟϧͳੈք
͋Γ͕ͱ͏͍͟͝·ͨ͠