Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
iPhoneでコスプレをする技術
Search
Tatsuya Tanaka
September 13, 2018
Programming
3
900
iPhoneでコスプレをする技術
iPhoneでコスプレをする技術
俺コン 2018 Summer
#orecon_ios
Tatsuya Tanaka
September 13, 2018
Tweet
Share
More Decks by Tatsuya Tanaka
See All by Tatsuya Tanaka
iPhoneのセンサー情報をmacOSアプリでリアルタイム活用するための技術
tattn
1
450
Better use of SwiftUI
tattn
0
350
Swift Concurrencyによる安全で快適な非同期処理
tattn
2
1k
iOSアプリの技術選択2022
tattn
7
3.7k
Widget Suggestions 対応と ヤフーの新OS対応
tattn
1
1.2k
WidgetKitで良い体験を作るには / Good experience with WidgetKit
tattn
2
1.4k
既存アプリにSwiftUIをどう組み込んでいくか
tattn
8
2.1k
iOS 14からのアプリ内課金
tattn
5
2.5k
iOS 14の位置情報系アップデート
tattn
0
22k
Other Decks in Programming
See All in Programming
しくじり先生 Image Matching Challenge 2024 編
goosehaaan
0
810
AWSでゲームサーバーを運用! Amazon GameLiftのお話
iriikeita
0
200
MIERUNE BBQにおけるユーザー中心設計()
mierune
PRO
1
110
Google's Recipe for Scaling (Web) Security – LocoMocoSec 2024
lweichselbaum
0
170
Rubyのパフォーマンスプロファイリングの改善 / Enhancing performance profiling for Ruby
osyoyu
1
410
リハビリmruby
kishima
1
160
feature環境をGitHub ActionsとCloudFormationでいい感じに管理する
nealle
2
310
CSC307 Lecture 05
javiergs
PRO
0
210
CSC307 Lecture 13
javiergs
PRO
0
150
AHC035解説
terryu16
0
710
Play Billing Library 7.0.0 変更点まとめ@potatotips#88
kako351
0
160
Harnessing Large Language Models for Training-free Video Anomaly Detection
tereka114
1
1.3k
Featured
See All Featured
Navigating Team Friction
lara
181
13k
Fireside Chat
paigeccino
25
2.8k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
662
120k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
17
8.7k
A Tale of Four Properties
chriscoyier
155
22k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
20
7.2k
5 minutes of I Can Smell Your CMS
philhawksworth
200
19k
WebSockets: Embracing the real-time Web
robhawkes
59
7.2k
Teambox: Starting and Learning
jrom
130
8.6k
Keith and Marios Guide to Fast Websites
keithpitt
408
22k
4 Signs Your Business is Dying
shpigford
178
21k
RailsConf 2023
tenderlove
16
720
Transcript
iPhoneͰίεϓϨΛ͢Δٕज़ ాதୡ (@tattn) Զίϯ 2018 Summer #orecon_ios
ాத ୡ (@tattn) • Yahoo!Ҋ • iOSΞϓϦΤϯδχΞ @tattn @tanakasan2525 @tattn
ࡢͷApple Event؍·͔ͨ͠ʁ
iOS12ͷհϖʔδ IUUQTXXXBQQMFDPNKQJPTJPT
Ҋ͕ϖʔδʹࡌΓ·ͨ͠ IUUQTXXXBQQMFDPNKQJPTJPT
iPhoneͰίεϓϨΛ͢Δٕज़ IUUQTGPSUFFKQJPTEDKBQBOQSPQPTBMECGCCGCBBGCDGF
όʔνϟϧYouTuber ͍ͬͯ·͔͢ʁ
όʔνϟϧYouTuber (VTuber) Βͳ͍ਓϠϑʔͰݕࡧʂʮVTuberʯ YouTubeͳͲͷಈը৴αʔϏε্Ͱ 2D3DͷΞόλʔͰ৴͍ͯ͠Δਓͷ͜ͱɻ ϦΞϧͱόʔνϟϧͷମΛಉظ͢Δٕज़Λ͍ͬͯΔ͜ͱ͕ଟ͍ɻ
VTuberΛࢧ͑Δٕज़ • ମΛಈ͔͢ • खಈɺKinectɺVRػثɺϞʔγϣϯΩϟϓνϟʔػث • දΛಈ͔͢ • खಈɺFaceRigɺDlibɺVisionɺARKit •
(Λม͢Δ) • ιϑτΣΞ (࿀ͳͲ)ɺϋʔυΣΞ (VT-3ͳͲ) • දࣔ͢Δ • UnityɺFaceRigɺSceneKit (ҰྫͰ͢)
ٕज़తʹ໘ന͘ɺ৭ʑ࡞ͬͯΈ·ͨ͠ ※͜ΕOculus RiftͱUnityΛ͍ͬͯ·͢ VRδΣϯΨ VRΩϟονϘʔϧ VTuberͬΆ͍ࢹ ©Kizuna AI IUUQTUXJUUFSDPNUBOBLBTBOTUBUVT
εϚϗͰVTuberʹͳΕΔ
ͦ͏ɺiPhoneͳΒͶɻ
͜ΜͳΞϓϦΛ࡞Γ·ͨ͠ Α͔ͬͨΒ༡ΜͰΈͯͶˠ iPhone X × ARKit × SceneKit CoreAnimation ×
Accelerate ʮVTuberʯͰݕࡧ
༻ٕज़ Face Tracking ϨϯμϦϯάɾϞʔγϣϯ TrueDepthΧϝϥ ܭࢉ iPhone X × ARKit
× SceneKit CoreAnimation × Accelerate
ARKit ARFaceAnchorͰإͷ࢟ΛऔಘΛऔಘ BlendShapeLocationͰإͷύʔπͷঢ়ଶΛऔಘ IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPOBSLJUBSGBDFBODIPS
Face Trackingͷ࣮ߦ guard ARFaceTrackingConfiguration.isSupported else { return } let configuration
= ARFaceTrackingConfiguration() configuration.isLightEstimationEnabled = true let arSession = ARSession() arSession.delegate = self arSession.run(configuration, options: [.resetTracking, .removeExistingAnchors])
إͷύʔπͷঢ়ଶऔಘ extension ViewController: ARSessionDelegate { func session(_ session: ARSession, didUpdate
anchors: [ARAnchor]) { guard let faceAnchor = anchors .lazy .compactMap({ $0 as? ARFaceAnchor }) .first else { return } if let mouthOpenness = faceAnchor.blendShapes[.jawOpen]?.doubleValue { let e = 0.05 // ޱΛด͘͢͢͡Δ let mouthOpennessX8 = mouthOpenness * 8 // ޱΛ։͚͘͢͢Δ let value = mouthOpennessX8 < e ? 0 : mouthOpennessX8 > 1 - e ? 1 : mouthOpennessX8 // Ϟσϧͷߋ৽ }
SceneKit SCNSceneʹΩϟϥΫλʔΛஔ͠ SCNViewͰSceneΛදࣔ ©Kizuna AI
ϞσϧͷಡΈࠐΈ ϑϦʔͷϞσϧ͕ଟ͘ɺ දͷϞʔϑΟϯάϞʔγϣϯͳͲͷ ޓੑ͕ߴ͍MMDϞσϧΛࠓճ༻ ※MMD=MikuMikuDance IUUQEJDOJDPWJEFPKQBNJLVNJLVEBODF
MMDͷಡΈࠐΈ ϑΥʔϚοτͷ༷ࣗମެ։͞Ε͍ͯΔ IUUQLLILTFFTBBOFUDBUFHPSZIUNM ↑zipͷதʹ༷ॻ͕͋Δ IUUQTHJUIVCDPNNBHJDJFO..%4DFOF,JU ࠓճMMDSceneKitΛ༻
Ωϟϥͷύʔπͷऔಘ let model: SCNNode = try loadMMDModel(fromName: "kizunaai/kizunaai.pmx") let neck
= model.childNode(withName: "ट", recursively: true) let leftEye = model.childNode(withName: "ࠨ", recursively: true) let rightEye = model.childNode(withName: "ӈ", recursively: true) MMDͷNode (Ϙʔϯ) ໊ʹ҉తͳϓϩτίϧ͕͋ΔͷͰ ผͷϞσϧͦͷ··Ͱେମಈ͘ (ͨ·ʹશ֯ΧλΧφͱ֯ΧλΧφͷදهΏΕ͕...ţŜŖŪͳͲ)
CoreAnimation / SceneKit let keyPath = "morpher.weights.eye" var animation =
CAKeyframeAnimation(keyPath: keyPath) animation.values = [AnyObject]() animation.keyTimes = [NSNumber]() animation.timingFunctions = [CAMediaTimingFunction]() ... let animationGroup = CAAnimationGroup() animationGroup.animations = [CAAnimation]() ... animationGroup.animations?.append(animation) ... let animation = SCNAnimation(caAnimation: animationGroup) let player = SCNAnimationPlayer(animation: animation) model.addAnimationPlayer(player, forKey: nil) player.play()
Accelerate.framework ύϑΥʔϚϯεʹಛԽͨ͠ࢉज़ԋࢉϑϨʔϜϫʔΫ ࠓճ simd ͱ͍͏ খ͞ͳϕΫτϧͱߦྻΛѻ͏ϞδϡʔϧΛར༻ ※ simd = Single
Instruction Multiple Data
औಘͨ͠σʔλΛମͱटʹө͢Δ func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) { guard
let faceAnchor = anchors .lazy.compactMap({ $0 as? ARFaceAnchor }).first else { return } let transform: simd_float4x4 = faceAnchor.transform let quaternion = simd_quaternion(transform) neck.simdWorldOrientation = quaternion let translate4: simd_float4 = transform.columns.3 let translate = simd_float3(translate4.x * 5, translate4.y * 5, translate4.z * 50 + 20) // w = 1 model.simdPosition = translate ࠨӈ͋·Γಈ͔ͣɺલޙಈ͖͘͢͢Δ ARKit/SceneKitӈखܥ
͜Ε·Ͱઆ໌ٕͨ͠ज़ΛΈ߹ΘͤΔͱ όʔνϟϧͳମΛಘΔ͜ͱ͕Ͱ͖·͢ʂ
VRM
VRMͱ IUUQTEXBOHPHJUIVCJPWSN υϫϯΰ͕࡞ͬͨVR༻ͷ3DϞσϧϑΥʔϚοτ 4݄ʹެ։͞ΕɺVRք۾Ͱʹͳ͍ͬͯΔ ɾχίχཱମ ɾόʔνϟϧΩϟετ ɾcluster ɾVDraw ɾVΧπ ͳͲͰར༻͞Ε͍ͯΔ
VRMΛiOSͰಡΈࠐΊΔϥΠϒϥϦ VRMΛiOSͰಡΈࠐΊΔϥΠϒϥϦΛ࡞Γ·ͨ͠ IUUQTHJUIVCDPNUBUUO73.,JU VRMKit
VRMΛiOSͰಡΈࠐΊΔϥΠϒϥϦ IUUQTHJUIVCDPNUBUUO73.,JU VRMKit ϝλσʔλͷऔಘͱ Ϟσϧͷද͕ࣔՄೳ
͊͞ɺΈͳ͞Μόʔνϟϧͳੈք
͋Γ͕ͱ͏͍͟͝·ͨ͠