iPhoneでコスプレをする技術

iPhoneでコスプレをする技術

俺コン 2018 Summer
#orecon_ios

Transcript

1. iPhoneͰίεϓϨΛ͢Δٕज़
   ాதୡ໵ (@tattn)
   Զίϯ 2018 Summer #orecon_ios
   

   View Slide

2. ాத ୡ໵ (@tattn)
   • Yahoo!৐׵Ҋ಺
   • iOSΞϓϦΤϯδχΞ
   @tattn
   @tanakasan2525
   @tattn
   

   View Slide

3. ࡢ೔ͷApple Event؍·͔ͨ͠ʁ
   

   View Slide

4. iOS12ͷ঺հϖʔδ
   IUUQTXXXBQQMFDPNKQJPTJPT
   

   View Slide

5. ৐׵Ҋ಺͕ϖʔδʹࡌΓ·ͨ͠
   IUUQTXXXBQQMFDPNKQJPTJPT
   

   View Slide

6. iPhoneͰίεϓϨΛ͢Δٕज़
   IUUQTGPSUFFKQJPTEDKBQBOQSPQPTBMECGCCGCBBGCDGF
   

   View Slide

7. όʔνϟϧYouTuber

   ஌͍ͬͯ·͔͢ʁ
   

   View Slide

8. όʔνϟϧYouTuber (VTuber)
   ஌Βͳ͍ਓ͸ϠϑʔͰݕࡧʂʮVTuberʯ
   YouTubeͳͲͷಈը഑৴αʔϏε্Ͱ

   2D΍3DͷΞόλʔͰ഑৴͍ͯ͠Δਓͷ͜ͱɻ

   ϦΞϧͱόʔνϟϧͷମΛಉظ͢Δٕज़Λ࢖͍ͬͯΔ͜ͱ͕ଟ͍ɻ
   

   View Slide

9. VTuberΛࢧ͑Δٕज़
   • ମΛಈ͔͢
   • खಈɺKinectɺVRػثɺϞʔγϣϯΩϟϓνϟʔػث
   • ද৘Λಈ͔͢
   • खಈɺFaceRigɺDlibɺVisionɺARKit
   • (੠Λม׵͢Δ)
   • ιϑτ΢ΣΞ (࿀੠ͳͲ)ɺϋʔυ΢ΣΞ (VT-3ͳͲ)
   • දࣔ͢Δ
   • UnityɺFaceRigɺSceneKit (ҰྫͰ͢)
   

   View Slide

10. ٕज़తʹ΋໘ന͘ɺ৭ʑ࡞ͬͯΈ·ͨ͠
    ※͜Ε͸Oculus RiftͱUnityΛ࢖͍ͬͯ·͢
    VRδΣϯΨ
    VRΩϟονϘʔϧ
    VTuberͬΆ͍ࢹ఺
    ©Kizuna AI
    IUUQTUXJUUFSDPNUBOBLBTBOTUBUVT
    

    View Slide

11. εϚϗͰ΋VTuberʹͳΕΔ
    

    View Slide

12. ͦ͏ɺiPhoneͳΒͶɻ
    

    View Slide

13. ͜ΜͳΞϓϦΛ࡞Γ·ͨ͠
    Α͔ͬͨΒ༡ΜͰΈͯͶˠ
    iPhone X
    ×
    ARKit
    ×
    SceneKit
    CoreAnimation
    ×
    Accelerate
    ʮVTuberʯͰݕࡧ
    

    View Slide

14. ࢖༻ٕज़
    Face Tracking
    ϨϯμϦϯάɾϞʔγϣϯ
    TrueDepthΧϝϥ
    ܭࢉ
    iPhone X
    ×
    ARKit
    ×
    SceneKit
    CoreAnimation
    ×
    Accelerate
    

    View Slide

15. ARKit
    ARFaceAnchorͰإͷ࢟੎ΛऔಘΛऔಘ

    BlendShapeLocationͰإͷύʔπͷঢ়ଶΛऔಘ
    IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPOBSLJUBSGBDFBODIPS
    

    View Slide

16. Face Trackingͷ࣮ߦ
    guard ARFaceTrackingConfiguration.isSupported else { return }
    let configuration = ARFaceTrackingConfiguration()
    configuration.isLightEstimationEnabled = true
    let arSession = ARSession()
    arSession.delegate = self
    arSession.run(configuration, options: [.resetTracking,
    .removeExistingAnchors])
    

    View Slide

17. إͷύʔπͷঢ়ଶऔಘ
    extension ViewController: ARSessionDelegate {
    func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
    guard let faceAnchor = anchors
    .lazy
    .compactMap({ $0 as? ARFaceAnchor })
    .first else { return }
    if let mouthOpenness = faceAnchor.blendShapes[.jawOpen]?.doubleValue {
    let e = 0.05 // ޱΛด͡΍͘͢͢Δ
    let mouthOpennessX8 = mouthOpenness * 8 // ޱΛ։͚΍͘͢͢Δ
    let value = mouthOpennessX8 < e ? 0 : mouthOpennessX8 > 1 - e ? 1 : mouthOpennessX8
    // Ϟσϧͷߋ৽
    }
    

    View Slide

18. SceneKit
    SCNSceneʹΩϟϥΫλʔΛ഑ஔ͠

    SCNViewͰSceneΛදࣔ
    ©Kizuna AI
    

    View Slide

19. ϞσϧͷಡΈࠐΈ
    ϑϦʔͷϞσϧ͕ଟ͘ɺ

    ද৘ͷϞʔϑΟϯά΍ϞʔγϣϯͳͲͷ

    ޓ׵ੑ͕ߴ͍MMDϞσϧΛࠓճ͸࢖༻
    ※MMD=MikuMikuDance
    IUUQEJDOJDPWJEFPKQBNJLVNJLVEBODF
    

    View Slide

20. MMDͷಡΈࠐΈ
    ϑΥʔϚοτͷ࢓༷ࣗମ͸ެ։͞Ε͍ͯΔ
    IUUQLLILTFFTBBOFUDBUFHPSZIUNM
    ↑zipͷதʹ࢓༷ॻ͕͋Δ
    IUUQTHJUIVCDPNNBHJDJFO..%4DFOF,JU
    ࠓճ͸MMDSceneKitΛ࢖༻
    

    View Slide

21. Ωϟϥͷύʔπͷऔಘ
    let model: SCNNode = try loadMMDModel(fromName: "kizunaai/kizunaai.pmx")
    let neck = model.childNode(withName: "ट", recursively: true)
    let leftEye = model.childNode(withName: "ࠨ໨", recursively: true)
    let rightEye = model.childNode(withName: "ӈ໨", recursively: true)
    MMDͷNode (Ϙʔϯ) ໊ʹ͸҉໧తͳϓϩτίϧ͕͋ΔͷͰ

    ผͷϞσϧ΋ͦͷ··Ͱେମಈ͘ 

    (ͨ·ʹશ֯ΧλΧφͱ൒֯ΧλΧφͷදهΏΕ͕...ţŜŖŪͳͲ)
    

    View Slide

22. CoreAnimation / SceneKit
    let keyPath = "morpher.weights.eye"
    var animation = CAKeyframeAnimation(keyPath: keyPath)
    animation.values = [AnyObject]()
    animation.keyTimes = [NSNumber]()
    animation.timingFunctions = [CAMediaTimingFunction]()
    ...
    let animationGroup = CAAnimationGroup()
    animationGroup.animations = [CAAnimation]()
    ...
    animationGroup.animations?.append(animation)
    ...
    let animation = SCNAnimation(caAnimation: animationGroup)
    let player = SCNAnimationPlayer(animation: animation)
    model.addAnimationPlayer(player, forKey: nil)
    player.play()
    

    View Slide

23. Accelerate.framework
    ύϑΥʔϚϯεʹಛԽͨ͠ࢉज़ԋࢉϑϨʔϜϫʔΫ
    ࠓճ͸ simd ͱ͍͏

    খ͞ͳϕΫτϧͱߦྻΛѻ͏ϞδϡʔϧΛར༻
    ※ simd = Single Instruction Multiple Data
    

    View Slide

24. औಘͨ͠σʔλΛମͱटʹ൓ө͢Δ
    func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
    guard let faceAnchor = anchors
    .lazy.compactMap({ $0 as? ARFaceAnchor }).first else { return }
    let transform: simd_float4x4 = faceAnchor.transform
    let quaternion = simd_quaternion(transform)
    neck.simdWorldOrientation = quaternion
    let translate4: simd_float4 = transform.columns.3
    let translate = simd_float3(translate4.x * 5,
    translate4.y * 5,
    translate4.z * 50 + 20) // w = 1
    model.simdPosition = translate
    ࠨӈ͸͋·Γಈ͔ͣɺલޙ͸ಈ͖΍͘͢͢Δ
    ARKit/SceneKit͸ӈखܥ
    

    View Slide

25. ͜Ε·Ͱઆ໌ٕͨ͠ज़Λ૊Έ߹ΘͤΔͱ
    όʔνϟϧͳମΛಘΔ͜ͱ͕Ͱ͖·͢ʂ
    

    View Slide

26. VRM
    

    View Slide

27. VRMͱ͸
    IUUQTEXBOHPHJUIVCJPWSN
    υϫϯΰ͕࡞ͬͨVR༻ͷ3DϞσϧϑΥʔϚοτ

    4݄ʹެ։͞ΕɺVRք۾Ͱ࿩୊ʹͳ͍ͬͯΔ
    ɾχίχཱମ

    ɾόʔνϟϧΩϟετ

    ɾcluster

    ɾVDraw

    ɾVΧπ ͳͲͰར༻͞Ε͍ͯΔ
    

    View Slide

28. VRMΛiOSͰಡΈࠐΊΔϥΠϒϥϦ
    VRMΛiOSͰಡΈࠐΊΔϥΠϒϥϦΛ࡞Γ·ͨ͠
    IUUQTHJUIVCDPNUBUUO73.,JU
    VRMKit
    

    View Slide

29. VRMΛiOSͰಡΈࠐΊΔϥΠϒϥϦ
    IUUQTHJUIVCDPNUBUUO73.,JU
    VRMKit
    ϝλσʔλͷऔಘͱ

    Ϟσϧͷද͕ࣔՄೳ
    

    View Slide

30. ͊͞ɺΈͳ͞Μ΋όʔνϟϧͳੈք΁
    

    View Slide

31. ͋Γ͕ͱ͏͍͟͝·ͨ͠
    

    View Slide