iPhoneでコスプレをする技術

Transcript

1. iPhoneͰίεϓϨΛ͢Δٕज़ ాதୡ໵ (@tattn) Զίϯ 2018 Summer #orecon_ios

2. ాத ୡ໵ (@tattn) • Yahoo!৐׵Ҋ಺ • iOSΞϓϦΤϯδχΞ @tattn @tanakasan2525 @tattn

3. ࡢ೔ͷApple Event؍·͔ͨ͠ʁ

4. iOS12ͷ঺հϖʔδ IUUQTXXXBQQMFDPNKQJPTJPT

5. ৐׵Ҋ಺͕ϖʔδʹࡌΓ·ͨ͠ IUUQTXXXBQQMFDPNKQJPTJPT

6. iPhoneͰίεϓϨΛ͢Δٕज़ IUUQTGPSUFFKQJPTEDKBQBOQSPQPTBMECGCCGCBBGCDGF

7. όʔνϟϧYouTuber
 ஌͍ͬͯ·͔͢ʁ

8. όʔνϟϧYouTuber (VTuber) ஌Βͳ͍ਓ͸ϠϑʔͰݕࡧʂʮVTuberʯ YouTubeͳͲͷಈը഑৴αʔϏε্Ͱ
 2D΍3DͷΞόλʔͰ഑৴͍ͯ͠Δਓͷ͜ͱɻ
 ϦΞϧͱόʔνϟϧͷମΛಉظ͢Δٕज़Λ࢖͍ͬͯΔ͜ͱ͕ଟ͍ɻ

9. VTuberΛࢧ͑Δٕज़ • ମΛಈ͔͢ • खಈɺKinectɺVRػثɺϞʔγϣϯΩϟϓνϟʔػث • ද৘Λಈ͔͢ • खಈɺFaceRigɺDlibɺVisionɺARKit •

   (੠Λม׵͢Δ) • ιϑτ΢ΣΞ (࿀੠ͳͲ)ɺϋʔυ΢ΣΞ (VT-3ͳͲ) • දࣔ͢Δ • UnityɺFaceRigɺSceneKit (ҰྫͰ͢)

10. ٕज़తʹ΋໘ന͘ɺ৭ʑ࡞ͬͯΈ·ͨ͠ ※͜Ε͸Oculus RiftͱUnityΛ࢖͍ͬͯ·͢ VRδΣϯΨ VRΩϟονϘʔϧ VTuberͬΆ͍ࢹ఺ ©Kizuna AI IUUQTUXJUUFSDPNUBOBLBTBOTUBUVT

11. εϚϗͰ΋VTuberʹͳΕΔ

12. ͦ͏ɺiPhoneͳΒͶɻ

13. ͜ΜͳΞϓϦΛ࡞Γ·ͨ͠ Α͔ͬͨΒ༡ΜͰΈͯͶˠ iPhone X × ARKit × SceneKit CoreAnimation ×

    Accelerate ʮVTuberʯͰݕࡧ

14. ࢖༻ٕज़ Face Tracking ϨϯμϦϯάɾϞʔγϣϯ TrueDepthΧϝϥ ܭࢉ iPhone X × ARKit

    × SceneKit CoreAnimation × Accelerate

15. ARKit ARFaceAnchorͰإͷ࢟੎ΛऔಘΛऔಘ
 BlendShapeLocationͰإͷύʔπͷঢ়ଶΛऔಘ IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPOBSLJUBSGBDFBODIPS

16. Face Trackingͷ࣮ߦ guard ARFaceTrackingConfiguration.isSupported else { return } let configuration

    = ARFaceTrackingConfiguration() configuration.isLightEstimationEnabled = true let arSession = ARSession() arSession.delegate = self arSession.run(configuration, options: [.resetTracking, .removeExistingAnchors])

17. إͷύʔπͷঢ়ଶऔಘ extension ViewController: ARSessionDelegate { func session(_ session: ARSession, didUpdate

    anchors: [ARAnchor]) { guard let faceAnchor = anchors .lazy .compactMap({ $0 as? ARFaceAnchor }) .first else { return } if let mouthOpenness = faceAnchor.blendShapes[.jawOpen]?.doubleValue { let e = 0.05 // ޱΛด͡΍͘͢͢Δ let mouthOpennessX8 = mouthOpenness * 8 // ޱΛ։͚΍͘͢͢Δ let value = mouthOpennessX8 < e ? 0 : mouthOpennessX8 > 1 - e ? 1 : mouthOpennessX8 // Ϟσϧͷߋ৽ }

18. SceneKit SCNSceneʹΩϟϥΫλʔΛ഑ஔ͠
 SCNViewͰSceneΛදࣔ ©Kizuna AI

19. ϞσϧͷಡΈࠐΈ ϑϦʔͷϞσϧ͕ଟ͘ɺ
 ද৘ͷϞʔϑΟϯά΍ϞʔγϣϯͳͲͷ
 ޓ׵ੑ͕ߴ͍MMDϞσϧΛࠓճ͸࢖༻ ※MMD=MikuMikuDance IUUQEJDOJDPWJEFPKQBNJLVNJLVEBODF

20. MMDͷಡΈࠐΈ ϑΥʔϚοτͷ࢓༷ࣗମ͸ެ։͞Ε͍ͯΔ IUUQLLILTFFTBBOFUDBUFHPSZIUNM ↑zipͷதʹ࢓༷ॻ͕͋Δ IUUQTHJUIVCDPNNBHJDJFO..%4DFOF,JU ࠓճ͸MMDSceneKitΛ࢖༻

21. Ωϟϥͷύʔπͷऔಘ let model: SCNNode = try loadMMDModel(fromName: "kizunaai/kizunaai.pmx") let neck

    = model.childNode(withName: "ट", recursively: true) let leftEye = model.childNode(withName: "ࠨ໨", recursively: true) let rightEye = model.childNode(withName: "ӈ໨", recursively: true) MMDͷNode (Ϙʔϯ) ໊ʹ͸҉໧తͳϓϩτίϧ͕͋ΔͷͰ
 ผͷϞσϧ΋ͦͷ··Ͱେମಈ͘ 
 (ͨ·ʹશ֯ΧλΧφͱ൒֯ΧλΧφͷදهΏΕ͕...ţŜŖŪͳͲ)

22. CoreAnimation / SceneKit let keyPath = "morpher.weights.eye" var animation =

    CAKeyframeAnimation(keyPath: keyPath) animation.values = [AnyObject]() animation.keyTimes = [NSNumber]() animation.timingFunctions = [CAMediaTimingFunction]() ... let animationGroup = CAAnimationGroup() animationGroup.animations = [CAAnimation]() ... animationGroup.animations?.append(animation) ... let animation = SCNAnimation(caAnimation: animationGroup) let player = SCNAnimationPlayer(animation: animation) model.addAnimationPlayer(player, forKey: nil) player.play()

23. Accelerate.framework ύϑΥʔϚϯεʹಛԽͨ͠ࢉज़ԋࢉϑϨʔϜϫʔΫ ࠓճ͸ simd ͱ͍͏
 খ͞ͳϕΫτϧͱߦྻΛѻ͏ϞδϡʔϧΛར༻ ※ simd = Single

    Instruction Multiple Data

24. औಘͨ͠σʔλΛମͱटʹ൓ө͢Δ func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) { guard

    let faceAnchor = anchors .lazy.compactMap({ $0 as? ARFaceAnchor }).first else { return } let transform: simd_float4x4 = faceAnchor.transform let quaternion = simd_quaternion(transform) neck.simdWorldOrientation = quaternion let translate4: simd_float4 = transform.columns.3 let translate = simd_float3(translate4.x * 5, translate4.y * 5, translate4.z * 50 + 20) // w = 1 model.simdPosition = translate ࠨӈ͸͋·Γಈ͔ͣɺલޙ͸ಈ͖΍͘͢͢Δ ARKit/SceneKit͸ӈखܥ

25. ͜Ε·Ͱઆ໌ٕͨ͠ज़Λ૊Έ߹ΘͤΔͱ όʔνϟϧͳମΛಘΔ͜ͱ͕Ͱ͖·͢ʂ

26. VRM

27. VRMͱ͸ IUUQTEXBOHPHJUIVCJPWSN υϫϯΰ͕࡞ͬͨVR༻ͷ3DϞσϧϑΥʔϚοτ
 4݄ʹެ։͞ΕɺVRք۾Ͱ࿩୊ʹͳ͍ͬͯΔ ɾχίχཱମ
 ɾόʔνϟϧΩϟετ
 ɾcluster
 ɾVDraw
 ɾVΧπ ͳͲͰར༻͞Ε͍ͯΔ

28. VRMΛiOSͰಡΈࠐΊΔϥΠϒϥϦ VRMΛiOSͰಡΈࠐΊΔϥΠϒϥϦΛ࡞Γ·ͨ͠ IUUQTHJUIVCDPNUBUUO73.,JU VRMKit

29. VRMΛiOSͰಡΈࠐΊΔϥΠϒϥϦ IUUQTHJUIVCDPNUBUUO73.,JU VRMKit ϝλσʔλͷऔಘͱ
 Ϟσϧͷද͕ࣔՄೳ

30. ͊͞ɺΈͳ͞Μ΋όʔνϟϧͳੈք΁

31. ͋Γ͕ͱ͏͍͟͝·ͨ͠