Upgrade to Pro — share decks privately, control downloads, hide ads and more …

iPhoneでコスプレをする技術

 iPhoneでコスプレをする技術

iPhoneでコスプレをする技術

俺コン 2018 Summer
#orecon_ios

Tatsuya Tanaka

September 13, 2018
Tweet

More Decks by Tatsuya Tanaka

Other Decks in Programming

Transcript

  1. iPhoneͰίεϓϨΛ͢Δٕज़
    ాதୡ໵ (@tattn)
    Զίϯ 2018 Summer #orecon_ios

    View Slide

  2. ాத ୡ໵ (@tattn)
    • Yahoo!৐׵Ҋ಺
    • iOSΞϓϦΤϯδχΞ
    @tattn
    @tanakasan2525
    @tattn

    View Slide

  3. ࡢ೔ͷApple Event؍·͔ͨ͠ʁ

    View Slide

  4. iOS12ͷ঺հϖʔδ
    IUUQTXXXBQQMFDPNKQJPTJPT

    View Slide

  5. ৐׵Ҋ಺͕ϖʔδʹࡌΓ·ͨ͠
    IUUQTXXXBQQMFDPNKQJPTJPT

    View Slide

  6. iPhoneͰίεϓϨΛ͢Δٕज़
    IUUQTGPSUFFKQJPTEDKBQBOQSPQPTBMECGCCGCBBGCDGF

    View Slide

  7. όʔνϟϧYouTuber

    ஌͍ͬͯ·͔͢ʁ

    View Slide

  8. όʔνϟϧYouTuber (VTuber)
    ஌Βͳ͍ਓ͸ϠϑʔͰݕࡧʂʮVTuberʯ
    YouTubeͳͲͷಈը഑৴αʔϏε্Ͱ

    2D΍3DͷΞόλʔͰ഑৴͍ͯ͠Δਓͷ͜ͱɻ

    ϦΞϧͱόʔνϟϧͷମΛಉظ͢Δٕज़Λ࢖͍ͬͯΔ͜ͱ͕ଟ͍ɻ

    View Slide

  9. VTuberΛࢧ͑Δٕज़
    • ମΛಈ͔͢
    • खಈɺKinectɺVRػثɺϞʔγϣϯΩϟϓνϟʔػث
    • ද৘Λಈ͔͢
    • खಈɺFaceRigɺDlibɺVisionɺARKit
    • (੠Λม׵͢Δ)
    • ιϑτ΢ΣΞ (࿀੠ͳͲ)ɺϋʔυ΢ΣΞ (VT-3ͳͲ)
    • දࣔ͢Δ
    • UnityɺFaceRigɺSceneKit (ҰྫͰ͢)

    View Slide

  10. ٕज़తʹ΋໘ന͘ɺ৭ʑ࡞ͬͯΈ·ͨ͠
    ※͜Ε͸Oculus RiftͱUnityΛ࢖͍ͬͯ·͢
    VRδΣϯΨ
    VRΩϟονϘʔϧ
    VTuberͬΆ͍ࢹ఺
    ©Kizuna AI
    IUUQTUXJUUFSDPNUBOBLBTBOTUBUVT

    View Slide

  11. εϚϗͰ΋VTuberʹͳΕΔ

    View Slide

  12. ͦ͏ɺiPhoneͳΒͶɻ

    View Slide

  13. ͜ΜͳΞϓϦΛ࡞Γ·ͨ͠
    Α͔ͬͨΒ༡ΜͰΈͯͶˠ
    iPhone X
    ×
    ARKit
    ×
    SceneKit
    CoreAnimation
    ×
    Accelerate
    ʮVTuberʯͰݕࡧ

    View Slide

  14. ࢖༻ٕज़
    Face Tracking
    ϨϯμϦϯάɾϞʔγϣϯ
    TrueDepthΧϝϥ
    ܭࢉ
    iPhone X
    ×
    ARKit
    ×
    SceneKit
    CoreAnimation
    ×
    Accelerate

    View Slide

  15. ARKit
    ARFaceAnchorͰإͷ࢟੎ΛऔಘΛऔಘ

    BlendShapeLocationͰإͷύʔπͷঢ়ଶΛऔಘ
    IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPOBSLJUBSGBDFBODIPS

    View Slide

  16. Face Trackingͷ࣮ߦ
    guard ARFaceTrackingConfiguration.isSupported else { return }
    let configuration = ARFaceTrackingConfiguration()
    configuration.isLightEstimationEnabled = true
    let arSession = ARSession()
    arSession.delegate = self
    arSession.run(configuration, options: [.resetTracking,
    .removeExistingAnchors])

    View Slide

  17. إͷύʔπͷঢ়ଶऔಘ
    extension ViewController: ARSessionDelegate {
    func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
    guard let faceAnchor = anchors
    .lazy
    .compactMap({ $0 as? ARFaceAnchor })
    .first else { return }
    if let mouthOpenness = faceAnchor.blendShapes[.jawOpen]?.doubleValue {
    let e = 0.05 // ޱΛด͡΍͘͢͢Δ
    let mouthOpennessX8 = mouthOpenness * 8 // ޱΛ։͚΍͘͢͢Δ
    let value = mouthOpennessX8 < e ? 0 : mouthOpennessX8 > 1 - e ? 1 : mouthOpennessX8
    // Ϟσϧͷߋ৽
    }

    View Slide

  18. SceneKit
    SCNSceneʹΩϟϥΫλʔΛ഑ஔ͠

    SCNViewͰSceneΛදࣔ
    ©Kizuna AI

    View Slide

  19. ϞσϧͷಡΈࠐΈ
    ϑϦʔͷϞσϧ͕ଟ͘ɺ

    ද৘ͷϞʔϑΟϯά΍ϞʔγϣϯͳͲͷ

    ޓ׵ੑ͕ߴ͍MMDϞσϧΛࠓճ͸࢖༻
    ※MMD=MikuMikuDance
    IUUQEJDOJDPWJEFPKQBNJLVNJLVEBODF

    View Slide

  20. MMDͷಡΈࠐΈ
    ϑΥʔϚοτͷ࢓༷ࣗମ͸ެ։͞Ε͍ͯΔ
    IUUQLLILTFFTBBOFUDBUFHPSZIUNM
    ↑zipͷதʹ࢓༷ॻ͕͋Δ
    IUUQTHJUIVCDPNNBHJDJFO..%4DFOF,JU
    ࠓճ͸MMDSceneKitΛ࢖༻

    View Slide

  21. Ωϟϥͷύʔπͷऔಘ
    let model: SCNNode = try loadMMDModel(fromName: "kizunaai/kizunaai.pmx")
    let neck = model.childNode(withName: "ट", recursively: true)
    let leftEye = model.childNode(withName: "ࠨ໨", recursively: true)
    let rightEye = model.childNode(withName: "ӈ໨", recursively: true)
    MMDͷNode (Ϙʔϯ) ໊ʹ͸҉໧తͳϓϩτίϧ͕͋ΔͷͰ

    ผͷϞσϧ΋ͦͷ··Ͱେମಈ͘ 

    (ͨ·ʹશ֯ΧλΧφͱ൒֯ΧλΧφͷදهΏΕ͕...ţŜŖŪͳͲ)

    View Slide

  22. CoreAnimation / SceneKit
    let keyPath = "morpher.weights.eye"
    var animation = CAKeyframeAnimation(keyPath: keyPath)
    animation.values = [AnyObject]()
    animation.keyTimes = [NSNumber]()
    animation.timingFunctions = [CAMediaTimingFunction]()
    ...
    let animationGroup = CAAnimationGroup()
    animationGroup.animations = [CAAnimation]()
    ...
    animationGroup.animations?.append(animation)
    ...
    let animation = SCNAnimation(caAnimation: animationGroup)
    let player = SCNAnimationPlayer(animation: animation)
    model.addAnimationPlayer(player, forKey: nil)
    player.play()

    View Slide

  23. Accelerate.framework
    ύϑΥʔϚϯεʹಛԽͨ͠ࢉज़ԋࢉϑϨʔϜϫʔΫ
    ࠓճ͸ simd ͱ͍͏

    খ͞ͳϕΫτϧͱߦྻΛѻ͏ϞδϡʔϧΛར༻
    ※ simd = Single Instruction Multiple Data

    View Slide

  24. औಘͨ͠σʔλΛମͱटʹ൓ө͢Δ
    func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
    guard let faceAnchor = anchors
    .lazy.compactMap({ $0 as? ARFaceAnchor }).first else { return }
    let transform: simd_float4x4 = faceAnchor.transform
    let quaternion = simd_quaternion(transform)
    neck.simdWorldOrientation = quaternion
    let translate4: simd_float4 = transform.columns.3
    let translate = simd_float3(translate4.x * 5,
    translate4.y * 5,
    translate4.z * 50 + 20) // w = 1
    model.simdPosition = translate
    ࠨӈ͸͋·Γಈ͔ͣɺલޙ͸ಈ͖΍͘͢͢Δ
    ARKit/SceneKit͸ӈखܥ

    View Slide

  25. ͜Ε·Ͱઆ໌ٕͨ͠ज़Λ૊Έ߹ΘͤΔͱ
    όʔνϟϧͳମΛಘΔ͜ͱ͕Ͱ͖·͢ʂ

    View Slide

  26. VRM

    View Slide

  27. VRMͱ͸
    IUUQTEXBOHPHJUIVCJPWSN
    υϫϯΰ͕࡞ͬͨVR༻ͷ3DϞσϧϑΥʔϚοτ

    4݄ʹެ։͞ΕɺVRք۾Ͱ࿩୊ʹͳ͍ͬͯΔ
    ɾχίχཱମ

    ɾόʔνϟϧΩϟετ

    ɾcluster

    ɾVDraw

    ɾVΧπ ͳͲͰར༻͞Ε͍ͯΔ

    View Slide

  28. VRMΛiOSͰಡΈࠐΊΔϥΠϒϥϦ
    VRMΛiOSͰಡΈࠐΊΔϥΠϒϥϦΛ࡞Γ·ͨ͠
    IUUQTHJUIVCDPNUBUUO73.,JU
    VRMKit

    View Slide

  29. VRMΛiOSͰಡΈࠐΊΔϥΠϒϥϦ
    IUUQTHJUIVCDPNUBUUO73.,JU
    VRMKit
    ϝλσʔλͷऔಘͱ

    Ϟσϧͷද͕ࣔՄೳ

    View Slide

  30. ͊͞ɺΈͳ͞Μ΋όʔνϟϧͳੈք΁

    View Slide

  31. ͋Γ͕ͱ͏͍͟͝·ͨ͠

    View Slide