Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
iPhoneでコスプレをする技術
Search
Tatsuya Tanaka
September 13, 2018
Programming
3
1k
iPhoneでコスプレをする技術
iPhoneでコスプレをする技術
俺コン 2018 Summer
#orecon_ios
Tatsuya Tanaka
September 13, 2018
Tweet
Share
More Decks by Tatsuya Tanaka
See All by Tatsuya Tanaka
iPhoneのセンサー情報をmacOSアプリでリアルタイム活用するための技術
tattn
1
620
Better use of SwiftUI
tattn
0
430
Swift Concurrencyによる安全で快適な非同期処理
tattn
2
1.3k
iOSアプリの技術選択2022
tattn
7
3.8k
Widget Suggestions 対応と ヤフーの新OS対応
tattn
1
1.3k
WidgetKitで良い体験を作るには / Good experience with WidgetKit
tattn
2
1.6k
既存アプリにSwiftUIをどう組み込んでいくか
tattn
8
2.4k
iOS 14からのアプリ内課金
tattn
5
2.9k
iOS 14の位置情報系アップデート
tattn
0
22k
Other Decks in Programming
See All in Programming
統一感のある Go コードを生成 AI の力で手にいれる
otakakot
0
3k
「ElixirでIoT!!」のこれまでとこれから
takasehideki
0
360
AIネイティブなプロダクトをGolangで挑む取り組み
nmatsumoto4
0
110
LINEヤフー データグループ紹介
lycorp_recruit_jp
0
670
Perlで痩せる
yuukis
1
680
データベースコネクションプール(DBCP)の変遷と理解
fujikawa8
1
260
Perplexity Slack Botを作ってAI活用を進めた話 / AI Engineering Summit プレイベント
n3xem
0
660
機械学習って何? 5分で解説頑張ってみる
kuroneko2828
0
210
DroidKnights 2025 - 다양한 스크롤 뷰에서의 영상 재생
gaeun5744
3
270
カクヨムAndroidアプリのリブート
numeroanddev
0
430
C++20 射影変換
faithandbrave
0
480
Cursor Meetup Tokyo ゲノミクスとCursor: 進化と制約のあいだ
koido
2
1k
Featured
See All Featured
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
[RailsConf 2023] Rails as a piece of cake
palkan
55
5.6k
The Art of Programming - Codeland 2020
erikaheidi
54
13k
Designing for humans not robots
tammielis
253
25k
The Pragmatic Product Professional
lauravandoore
35
6.7k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.3k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
667
120k
Being A Developer After 40
akosma
90
590k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.7k
Large-scale JavaScript Application Architecture
addyosmani
512
110k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
228
22k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Transcript
iPhoneͰίεϓϨΛ͢Δٕज़ ాதୡ (@tattn) Զίϯ 2018 Summer #orecon_ios
ాத ୡ (@tattn) • Yahoo!Ҋ • iOSΞϓϦΤϯδχΞ @tattn @tanakasan2525 @tattn
ࡢͷApple Event؍·͔ͨ͠ʁ
iOS12ͷհϖʔδ IUUQTXXXBQQMFDPNKQJPTJPT
Ҋ͕ϖʔδʹࡌΓ·ͨ͠ IUUQTXXXBQQMFDPNKQJPTJPT
iPhoneͰίεϓϨΛ͢Δٕज़ IUUQTGPSUFFKQJPTEDKBQBOQSPQPTBMECGCCGCBBGCDGF
όʔνϟϧYouTuber ͍ͬͯ·͔͢ʁ
όʔνϟϧYouTuber (VTuber) Βͳ͍ਓϠϑʔͰݕࡧʂʮVTuberʯ YouTubeͳͲͷಈը৴αʔϏε্Ͱ 2D3DͷΞόλʔͰ৴͍ͯ͠Δਓͷ͜ͱɻ ϦΞϧͱόʔνϟϧͷମΛಉظ͢Δٕज़Λ͍ͬͯΔ͜ͱ͕ଟ͍ɻ
VTuberΛࢧ͑Δٕज़ • ମΛಈ͔͢ • खಈɺKinectɺVRػثɺϞʔγϣϯΩϟϓνϟʔػث • දΛಈ͔͢ • खಈɺFaceRigɺDlibɺVisionɺARKit •
(Λม͢Δ) • ιϑτΣΞ (࿀ͳͲ)ɺϋʔυΣΞ (VT-3ͳͲ) • දࣔ͢Δ • UnityɺFaceRigɺSceneKit (ҰྫͰ͢)
ٕज़తʹ໘ന͘ɺ৭ʑ࡞ͬͯΈ·ͨ͠ ※͜ΕOculus RiftͱUnityΛ͍ͬͯ·͢ VRδΣϯΨ VRΩϟονϘʔϧ VTuberͬΆ͍ࢹ ©Kizuna AI IUUQTUXJUUFSDPNUBOBLBTBOTUBUVT
εϚϗͰVTuberʹͳΕΔ
ͦ͏ɺiPhoneͳΒͶɻ
͜ΜͳΞϓϦΛ࡞Γ·ͨ͠ Α͔ͬͨΒ༡ΜͰΈͯͶˠ iPhone X × ARKit × SceneKit CoreAnimation ×
Accelerate ʮVTuberʯͰݕࡧ
༻ٕज़ Face Tracking ϨϯμϦϯάɾϞʔγϣϯ TrueDepthΧϝϥ ܭࢉ iPhone X × ARKit
× SceneKit CoreAnimation × Accelerate
ARKit ARFaceAnchorͰإͷ࢟ΛऔಘΛऔಘ BlendShapeLocationͰإͷύʔπͷঢ়ଶΛऔಘ IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPOBSLJUBSGBDFBODIPS
Face Trackingͷ࣮ߦ guard ARFaceTrackingConfiguration.isSupported else { return } let configuration
= ARFaceTrackingConfiguration() configuration.isLightEstimationEnabled = true let arSession = ARSession() arSession.delegate = self arSession.run(configuration, options: [.resetTracking, .removeExistingAnchors])
إͷύʔπͷঢ়ଶऔಘ extension ViewController: ARSessionDelegate { func session(_ session: ARSession, didUpdate
anchors: [ARAnchor]) { guard let faceAnchor = anchors .lazy .compactMap({ $0 as? ARFaceAnchor }) .first else { return } if let mouthOpenness = faceAnchor.blendShapes[.jawOpen]?.doubleValue { let e = 0.05 // ޱΛด͘͢͢͡Δ let mouthOpennessX8 = mouthOpenness * 8 // ޱΛ։͚͘͢͢Δ let value = mouthOpennessX8 < e ? 0 : mouthOpennessX8 > 1 - e ? 1 : mouthOpennessX8 // Ϟσϧͷߋ৽ }
SceneKit SCNSceneʹΩϟϥΫλʔΛஔ͠ SCNViewͰSceneΛදࣔ ©Kizuna AI
ϞσϧͷಡΈࠐΈ ϑϦʔͷϞσϧ͕ଟ͘ɺ දͷϞʔϑΟϯάϞʔγϣϯͳͲͷ ޓੑ͕ߴ͍MMDϞσϧΛࠓճ༻ ※MMD=MikuMikuDance IUUQEJDOJDPWJEFPKQBNJLVNJLVEBODF
MMDͷಡΈࠐΈ ϑΥʔϚοτͷ༷ࣗମެ։͞Ε͍ͯΔ IUUQLLILTFFTBBOFUDBUFHPSZIUNM ↑zipͷதʹ༷ॻ͕͋Δ IUUQTHJUIVCDPNNBHJDJFO..%4DFOF,JU ࠓճMMDSceneKitΛ༻
Ωϟϥͷύʔπͷऔಘ let model: SCNNode = try loadMMDModel(fromName: "kizunaai/kizunaai.pmx") let neck
= model.childNode(withName: "ट", recursively: true) let leftEye = model.childNode(withName: "ࠨ", recursively: true) let rightEye = model.childNode(withName: "ӈ", recursively: true) MMDͷNode (Ϙʔϯ) ໊ʹ҉తͳϓϩτίϧ͕͋ΔͷͰ ผͷϞσϧͦͷ··Ͱେମಈ͘ (ͨ·ʹશ֯ΧλΧφͱ֯ΧλΧφͷදهΏΕ͕...ţŜŖŪͳͲ)
CoreAnimation / SceneKit let keyPath = "morpher.weights.eye" var animation =
CAKeyframeAnimation(keyPath: keyPath) animation.values = [AnyObject]() animation.keyTimes = [NSNumber]() animation.timingFunctions = [CAMediaTimingFunction]() ... let animationGroup = CAAnimationGroup() animationGroup.animations = [CAAnimation]() ... animationGroup.animations?.append(animation) ... let animation = SCNAnimation(caAnimation: animationGroup) let player = SCNAnimationPlayer(animation: animation) model.addAnimationPlayer(player, forKey: nil) player.play()
Accelerate.framework ύϑΥʔϚϯεʹಛԽͨ͠ࢉज़ԋࢉϑϨʔϜϫʔΫ ࠓճ simd ͱ͍͏ খ͞ͳϕΫτϧͱߦྻΛѻ͏ϞδϡʔϧΛར༻ ※ simd = Single
Instruction Multiple Data
औಘͨ͠σʔλΛମͱटʹө͢Δ func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) { guard
let faceAnchor = anchors .lazy.compactMap({ $0 as? ARFaceAnchor }).first else { return } let transform: simd_float4x4 = faceAnchor.transform let quaternion = simd_quaternion(transform) neck.simdWorldOrientation = quaternion let translate4: simd_float4 = transform.columns.3 let translate = simd_float3(translate4.x * 5, translate4.y * 5, translate4.z * 50 + 20) // w = 1 model.simdPosition = translate ࠨӈ͋·Γಈ͔ͣɺલޙಈ͖͘͢͢Δ ARKit/SceneKitӈखܥ
͜Ε·Ͱઆ໌ٕͨ͠ज़ΛΈ߹ΘͤΔͱ όʔνϟϧͳମΛಘΔ͜ͱ͕Ͱ͖·͢ʂ
VRM
VRMͱ IUUQTEXBOHPHJUIVCJPWSN υϫϯΰ͕࡞ͬͨVR༻ͷ3DϞσϧϑΥʔϚοτ 4݄ʹެ։͞ΕɺVRք۾Ͱʹͳ͍ͬͯΔ ɾχίχཱମ ɾόʔνϟϧΩϟετ ɾcluster ɾVDraw ɾVΧπ ͳͲͰར༻͞Ε͍ͯΔ
VRMΛiOSͰಡΈࠐΊΔϥΠϒϥϦ VRMΛiOSͰಡΈࠐΊΔϥΠϒϥϦΛ࡞Γ·ͨ͠ IUUQTHJUIVCDPNUBUUO73.,JU VRMKit
VRMΛiOSͰಡΈࠐΊΔϥΠϒϥϦ IUUQTHJUIVCDPNUBUUO73.,JU VRMKit ϝλσʔλͷऔಘͱ Ϟσϧͷද͕ࣔՄೳ
͊͞ɺΈͳ͞Μόʔνϟϧͳੈք
͋Γ͕ͱ͏͍͟͝·ͨ͠