iPhoneでコスプレをする技術

by Tatsuya Tanaka

Slide 1

Slide 1 text

iPhoneͰίεϓϨΛ͢Δٕज़ ాதୡ໵ (@tattn) Զίϯ 2018 Summer #orecon_ios

Slide 2

Slide 2 text

ాத ୡ໵ (@tattn) • Yahoo!৐׵Ҋ಺ • iOSΞϓϦΤϯδχΞ @tattn @tanakasan2525 @tattn

Slide 3

Slide 3 text

ࡢ೔ͷApple Event؍·͔ͨ͠ʁ

Slide 4

Slide 4 text

iOS12ͷ঺հϖʔδ IUUQTXXXBQQMFDPNKQJPTJPT

Slide 5

Slide 5 text

৐׵Ҋ಺͕ϖʔδʹࡌΓ·ͨ͠ IUUQTXXXBQQMFDPNKQJPTJPT

Slide 6

Slide 6 text

iPhoneͰίεϓϨΛ͢Δٕज़ IUUQTGPSUFFKQJPTEDKBQBOQSPQPTBMECGCCGCBBGCDGF

Slide 7

Slide 7 text

όʔνϟϧYouTuber  ஌͍ͬͯ·͔͢ʁ

Slide 8

Slide 8 text

όʔνϟϧYouTuber (VTuber) ஌Βͳ͍ਓ͸ϠϑʔͰݕࡧʂʮVTuberʯ YouTubeͳͲͷಈը഑৴αʔϏε্Ͱ  2D΍3DͷΞόλʔͰ഑৴͍ͯ͠Δਓͷ͜ͱɻ  ϦΞϧͱόʔνϟϧͷମΛಉظ͢Δٕज़Λ࢖͍ͬͯΔ͜ͱ͕ଟ͍ɻ

Slide 9

Slide 9 text

VTuberΛࢧ͑Δٕज़ • ମΛಈ͔͢ • खಈɺKinectɺVRػثɺϞʔγϣϯΩϟϓνϟʔػث • ද৘Λಈ͔͢ • खಈɺFaceRigɺDlibɺVisionɺARKit • (੠Λม׵͢Δ) • ιϑτ΢ΣΞ (࿀੠ͳͲ)ɺϋʔυ΢ΣΞ (VT-3ͳͲ) • දࣔ͢Δ • UnityɺFaceRigɺSceneKit (ҰྫͰ͢)

Slide 10

Slide 10 text

ٕज़తʹ΋໘ന͘ɺ৭ʑ࡞ͬͯΈ·ͨ͠ ※͜Ε͸Oculus RiftͱUnityΛ࢖͍ͬͯ·͢ VRδΣϯΨ VRΩϟονϘʔϧ VTuberͬΆ͍ࢹ఺ ©Kizuna AI IUUQTUXJUUFSDPNUBOBLBTBOTUBUVT

Slide 11

Slide 11 text

εϚϗͰ΋VTuberʹͳΕΔ

Slide 12

Slide 12 text

ͦ͏ɺiPhoneͳΒͶɻ

Slide 13

Slide 13 text

͜ΜͳΞϓϦΛ࡞Γ·ͨ͠ Α͔ͬͨΒ༡ΜͰΈͯͶˠ iPhone X × ARKit × SceneKit CoreAnimation × Accelerate ʮVTuberʯͰݕࡧ

Slide 14

Slide 14 text

࢖༻ٕज़ Face Tracking ϨϯμϦϯάɾϞʔγϣϯ TrueDepthΧϝϥ ܭࢉ iPhone X × ARKit × SceneKit CoreAnimation × Accelerate

Slide 15

Slide 15 text

ARKit ARFaceAnchorͰإͷ࢟੎ΛऔಘΛऔಘ  BlendShapeLocationͰإͷύʔπͷঢ়ଶΛऔಘ IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPOBSLJUBSGBDFBODIPS

Slide 16

Slide 16 text

Face Trackingͷ࣮ߦ guard ARFaceTrackingConfiguration.isSupported else { return } let configuration = ARFaceTrackingConfiguration() configuration.isLightEstimationEnabled = true let arSession = ARSession() arSession.delegate = self arSession.run(configuration, options: [.resetTracking, .removeExistingAnchors])

Slide 17

Slide 17 text

إͷύʔπͷঢ়ଶऔಘ extension ViewController: ARSessionDelegate { func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) { guard let faceAnchor = anchors .lazy .compactMap({ $0 as? ARFaceAnchor }) .first else { return } if let mouthOpenness = faceAnchor.blendShapes[.jawOpen]?.doubleValue { let e = 0.05 // ޱΛด͡΍͘͢͢Δ let mouthOpennessX8 = mouthOpenness * 8 // ޱΛ։͚΍͘͢͢Δ let value = mouthOpennessX8 < e ? 0 : mouthOpennessX8 > 1 - e ? 1 : mouthOpennessX8 // Ϟσϧͷߋ৽ }

Slide 18

Slide 18 text

SceneKit SCNSceneʹΩϟϥΫλʔΛ഑ஔ͠  SCNViewͰSceneΛදࣔ ©Kizuna AI

Slide 19

Slide 19 text

ϞσϧͷಡΈࠐΈ ϑϦʔͷϞσϧ͕ଟ͘ɺ  ද৘ͷϞʔϑΟϯά΍ϞʔγϣϯͳͲͷ  ޓ׵ੑ͕ߴ͍MMDϞσϧΛࠓճ͸࢖༻ ※MMD=MikuMikuDance IUUQEJDOJDPWJEFPKQBNJLVNJLVEBODF

Slide 20

Slide 20 text

MMDͷಡΈࠐΈ ϑΥʔϚοτͷ࢓༷ࣗମ͸ެ։͞Ε͍ͯΔ IUUQLLILTFFTBBOFUDBUFHPSZIUNM ↑zipͷதʹ࢓༷ॻ͕͋Δ IUUQTHJUIVCDPNNBHJDJFO..%4DFOF,JU ࠓճ͸MMDSceneKitΛ࢖༻

Slide 21

Slide 21 text

Ωϟϥͷύʔπͷऔಘ let model: SCNNode = try loadMMDModel(fromName: "kizunaai/kizunaai.pmx") let neck = model.childNode(withName: "ट", recursively: true) let leftEye = model.childNode(withName: "ࠨ໨", recursively: true) let rightEye = model.childNode(withName: "ӈ໨", recursively: true) MMDͷNode (Ϙʔϯ) ໊ʹ͸҉໧తͳϓϩτίϧ͕͋ΔͷͰ  ผͷϞσϧ΋ͦͷ··Ͱେମಈ͘   (ͨ·ʹશ֯ΧλΧφͱ൒֯ΧλΧφͷදهΏΕ͕...ţŜŖŪͳͲ)

Slide 22

Slide 22 text

CoreAnimation / SceneKit let keyPath = "morpher.weights.eye" var animation = CAKeyframeAnimation(keyPath: keyPath) animation.values = [AnyObject]() animation.keyTimes = [NSNumber]() animation.timingFunctions = [CAMediaTimingFunction]() ... let animationGroup = CAAnimationGroup() animationGroup.animations = [CAAnimation]() ... animationGroup.animations?.append(animation) ... let animation = SCNAnimation(caAnimation: animationGroup) let player = SCNAnimationPlayer(animation: animation) model.addAnimationPlayer(player, forKey: nil) player.play()

Slide 23

Slide 23 text

Accelerate.framework ύϑΥʔϚϯεʹಛԽͨ͠ࢉज़ԋࢉϑϨʔϜϫʔΫ ࠓճ͸ simd ͱ͍͏  খ͞ͳϕΫτϧͱߦྻΛѻ͏ϞδϡʔϧΛར༻ ※ simd = Single Instruction Multiple Data

Slide 24

Slide 24 text

औಘͨ͠σʔλΛମͱटʹ൓ө͢Δ func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) { guard let faceAnchor = anchors .lazy.compactMap({ $0 as? ARFaceAnchor }).first else { return } let transform: simd_float4x4 = faceAnchor.transform let quaternion = simd_quaternion(transform) neck.simdWorldOrientation = quaternion let translate4: simd_float4 = transform.columns.3 let translate = simd_float3(translate4.x * 5, translate4.y * 5, translate4.z * 50 + 20) // w = 1 model.simdPosition = translate ࠨӈ͸͋·Γಈ͔ͣɺલޙ͸ಈ͖΍͘͢͢Δ ARKit/SceneKit͸ӈखܥ

Slide 25

Slide 25 text

͜Ε·Ͱઆ໌ٕͨ͠ज़Λ૊Έ߹ΘͤΔͱ όʔνϟϧͳମΛಘΔ͜ͱ͕Ͱ͖·͢ʂ

Slide 26

Slide 26 text

VRM

Slide 27

Slide 27 text

VRMͱ͸ IUUQTEXBOHPHJUIVCJPWSN υϫϯΰ͕࡞ͬͨVR༻ͷ3DϞσϧϑΥʔϚοτ  4݄ʹެ։͞ΕɺVRք۾Ͱ࿩୊ʹͳ͍ͬͯΔ ɾχίχཱମ  ɾόʔνϟϧΩϟετ  ɾcluster  ɾVDraw  ɾVΧπ ͳͲͰར༻͞Ε͍ͯΔ

Slide 28

Slide 28 text

VRMΛiOSͰಡΈࠐΊΔϥΠϒϥϦ VRMΛiOSͰಡΈࠐΊΔϥΠϒϥϦΛ࡞Γ·ͨ͠ IUUQTHJUIVCDPNUBUUO73.,JU VRMKit

Slide 29

Slide 29 text

VRMΛiOSͰಡΈࠐΊΔϥΠϒϥϦ IUUQTHJUIVCDPNUBUUO73.,JU VRMKit ϝλσʔλͷऔಘͱ  Ϟσϧͷද͕ࣔՄೳ