iPhoneでコスプレをする技術
俺コン 2018 Summer #orecon_ios
iPhoneͰίεϓϨΛ͢Δٕज़ాதୡ (@tattn)Զίϯ 2018 Summer #orecon_ios
View Slide
ాத ୡ (@tattn)• Yahoo!Ҋ• iOSΞϓϦΤϯδχΞ@tattn@tanakasan2525@tattn
ࡢͷApple Event؍·͔ͨ͠ʁ
iOS12ͷհϖʔδIUUQTXXXBQQMFDPNKQJPTJPT
Ҋ͕ϖʔδʹࡌΓ·ͨ͠ IUUQTXXXBQQMFDPNKQJPTJPT
iPhoneͰίεϓϨΛ͢Δٕज़IUUQTGPSUFFKQJPTEDKBQBOQSPQPTBMECGCCGCBBGCDGF
όʔνϟϧYouTuber ͍ͬͯ·͔͢ʁ
όʔνϟϧYouTuber (VTuber)Βͳ͍ਓϠϑʔͰݕࡧʂʮVTuberʯYouTubeͳͲͷಈը৴αʔϏε্Ͱ 2D3DͷΞόλʔͰ৴͍ͯ͠Δਓͷ͜ͱɻ ϦΞϧͱόʔνϟϧͷମΛಉظ͢Δٕज़Λ͍ͬͯΔ͜ͱ͕ଟ͍ɻ
VTuberΛࢧ͑Δٕज़• ମΛಈ͔͢• खಈɺKinectɺVRػثɺϞʔγϣϯΩϟϓνϟʔػث• දΛಈ͔͢• खಈɺFaceRigɺDlibɺVisionɺARKit• (Λม͢Δ)• ιϑτΣΞ (࿀ͳͲ)ɺϋʔυΣΞ (VT-3ͳͲ)• දࣔ͢Δ• UnityɺFaceRigɺSceneKit (ҰྫͰ͢)
ٕज़తʹ໘ന͘ɺ৭ʑ࡞ͬͯΈ·ͨ͠※͜ΕOculus RiftͱUnityΛ͍ͬͯ·͢VRδΣϯΨVRΩϟονϘʔϧVTuberͬΆ͍ࢹ©Kizuna AIIUUQTUXJUUFSDPNUBOBLBTBOTUBUVT
εϚϗͰVTuberʹͳΕΔ
ͦ͏ɺiPhoneͳΒͶɻ
͜ΜͳΞϓϦΛ࡞Γ·ͨ͠Α͔ͬͨΒ༡ΜͰΈͯͶˠiPhone X×ARKit×SceneKitCoreAnimation×AccelerateʮVTuberʯͰݕࡧ
༻ٕज़Face TrackingϨϯμϦϯάɾϞʔγϣϯTrueDepthΧϝϥܭࢉiPhone X×ARKit×SceneKitCoreAnimation×Accelerate
ARKitARFaceAnchorͰإͷ࢟ΛऔಘΛऔಘ BlendShapeLocationͰإͷύʔπͷঢ়ଶΛऔಘIUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPOBSLJUBSGBDFBODIPS
Face Trackingͷ࣮ߦguard ARFaceTrackingConfiguration.isSupported else { return }let configuration = ARFaceTrackingConfiguration()configuration.isLightEstimationEnabled = truelet arSession = ARSession()arSession.delegate = selfarSession.run(configuration, options: [.resetTracking,.removeExistingAnchors])
إͷύʔπͷঢ়ଶऔಘextension ViewController: ARSessionDelegate {func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {guard let faceAnchor = anchors.lazy.compactMap({ $0 as? ARFaceAnchor }).first else { return }if let mouthOpenness = faceAnchor.blendShapes[.jawOpen]?.doubleValue {let e = 0.05 // ޱΛด͘͢͢͡Δlet mouthOpennessX8 = mouthOpenness * 8 // ޱΛ։͚͘͢͢Δlet value = mouthOpennessX8 < e ? 0 : mouthOpennessX8 > 1 - e ? 1 : mouthOpennessX8// Ϟσϧͷߋ৽}
SceneKitSCNSceneʹΩϟϥΫλʔΛஔ͠ SCNViewͰSceneΛදࣔ©Kizuna AI
ϞσϧͷಡΈࠐΈϑϦʔͷϞσϧ͕ଟ͘ɺ දͷϞʔϑΟϯάϞʔγϣϯͳͲͷ ޓੑ͕ߴ͍MMDϞσϧΛࠓճ༻※MMD=MikuMikuDanceIUUQEJDOJDPWJEFPKQBNJLVNJLVEBODF
MMDͷಡΈࠐΈϑΥʔϚοτͷ༷ࣗମެ։͞Ε͍ͯΔIUUQLLILTFFTBBOFUDBUFHPSZIUNM↑zipͷதʹ༷ॻ͕͋ΔIUUQTHJUIVCDPNNBHJDJFO..%4DFOF,JUࠓճMMDSceneKitΛ༻
Ωϟϥͷύʔπͷऔಘlet model: SCNNode = try loadMMDModel(fromName: "kizunaai/kizunaai.pmx")let neck = model.childNode(withName: "ट", recursively: true)let leftEye = model.childNode(withName: "ࠨ", recursively: true)let rightEye = model.childNode(withName: "ӈ", recursively: true)MMDͷNode (Ϙʔϯ) ໊ʹ҉తͳϓϩτίϧ͕͋ΔͷͰ ผͷϞσϧͦͷ··Ͱେମಈ͘ (ͨ·ʹશ֯ΧλΧφͱ֯ΧλΧφͷදهΏΕ͕...ţŜŖŪͳͲ)
CoreAnimation / SceneKitlet keyPath = "morpher.weights.eye"var animation = CAKeyframeAnimation(keyPath: keyPath)animation.values = [AnyObject]()animation.keyTimes = [NSNumber]()animation.timingFunctions = [CAMediaTimingFunction]()...let animationGroup = CAAnimationGroup()animationGroup.animations = [CAAnimation]()...animationGroup.animations?.append(animation)...let animation = SCNAnimation(caAnimation: animationGroup)let player = SCNAnimationPlayer(animation: animation)model.addAnimationPlayer(player, forKey: nil)player.play()
Accelerate.frameworkύϑΥʔϚϯεʹಛԽͨ͠ࢉज़ԋࢉϑϨʔϜϫʔΫࠓճ simd ͱ͍͏ খ͞ͳϕΫτϧͱߦྻΛѻ͏ϞδϡʔϧΛར༻※ simd = Single Instruction Multiple Data
औಘͨ͠σʔλΛମͱटʹө͢Δfunc session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {guard let faceAnchor = anchors.lazy.compactMap({ $0 as? ARFaceAnchor }).first else { return }let transform: simd_float4x4 = faceAnchor.transformlet quaternion = simd_quaternion(transform)neck.simdWorldOrientation = quaternionlet translate4: simd_float4 = transform.columns.3let translate = simd_float3(translate4.x * 5,translate4.y * 5,translate4.z * 50 + 20) // w = 1model.simdPosition = translateࠨӈ͋·Γಈ͔ͣɺલޙಈ͖͘͢͢ΔARKit/SceneKitӈखܥ
͜Ε·Ͱઆ໌ٕͨ͠ज़ΛΈ߹ΘͤΔͱόʔνϟϧͳମΛಘΔ͜ͱ͕Ͱ͖·͢ʂ
VRM
VRMͱIUUQTEXBOHPHJUIVCJPWSNυϫϯΰ͕࡞ͬͨVR༻ͷ3DϞσϧϑΥʔϚοτ 4݄ʹެ։͞ΕɺVRք۾Ͱʹͳ͍ͬͯΔɾχίχཱମ ɾόʔνϟϧΩϟετ ɾcluster ɾVDraw ɾVΧπ ͳͲͰར༻͞Ε͍ͯΔ
VRMΛiOSͰಡΈࠐΊΔϥΠϒϥϦVRMΛiOSͰಡΈࠐΊΔϥΠϒϥϦΛ࡞Γ·ͨ͠IUUQTHJUIVCDPNUBUUO73.,JUVRMKit
VRMΛiOSͰಡΈࠐΊΔϥΠϒϥϦIUUQTHJUIVCDPNUBUUO73.,JUVRMKitϝλσʔλͷऔಘͱ Ϟσϧͷද͕ࣔՄೳ
͊͞ɺΈͳ͞Μόʔνϟϧͳੈք
͋Γ͕ͱ͏͍͟͝·ͨ͠