アバターミーティングを支える技術

Gotanda.unity #16
https://meetup.unity3d.jp/jp/events/1268
https://youtu.be/FTHfVpBfpo0

Transcript

1. ΞόλʔϛʔςΟϯάΛࢧ͑ Δٕज़ 2021/1/13 Gotanda.unity #16 Ͷ͓ΓΜ 1
2. noir_neo • Ͷ͓ΓΜ • Ϋϥελʔגࣜձࣾ • ιϑτ΢ΣΞΤϯδχΞ • Ϋϥελʔͷϋτ • 2021೥ͷ໨ඪ͸ʮ͓෩࿊ͷ࣌ܭͷి ஑Λ׵͑Δʯ 2
3. ͱ͜ΖͰΈͳ͞ΜɺϏ σΦϛʔςΟϯά͠· ͢ΑͶ? 3
4. ฐࣾɺVRϛʔςΟϯά͸౰ વ΍Γ·͕͢ɺϏσΦϛʔ ςΟϯά΋͠·͢ VRHMDඃͬͯϛʔςΟϯάϊʔτऔΔͷ͸େมͳͨΊɻ 4
5. ͜Μͳײ͡ → 5
7. ҰൠతͳϏσΦϛʔςΟϯά ͷޫܠͰ͢Ͷ? (ͦ͏Ͱ͋ͬͯ΄͍͠) 7
8. ͍ΘΏΔ VTuber ΞϓϦ ͍Ζ͍Ζ͋Δɻ 8
9. ͍Ζ͍Ζࢼͨ݁͠Ռɺࣗ෼Ͱ ࡞Γ࢝Ίͨ ͦ͏͍͏͜ͱ΋͋Δɻ 9
10. ࿩͢͜ͱ • ͍ΘΏΔ VTuber ΞϓϦͷߏ੒ྫ • ஌ݟɺͩ͜ΘΓϙΠϯτͳͲ 10
11. ݟग़͠ • γεςϜ֓ཁ • إͱखͷτϥοΩϯά • ΞόλʔͷಡΈࠐΈɾදࣔɾΞχϝʔγϣϯ • ө૾ͷૹग़ 11
12. γεςϜ֓ཁ 12
14. إͱखͷτϥοΩϯά • ARKit • Leap Motion 14
15. ARKit Face Tracking • iOS σόΠεͷϑϩϯτΧϝϥͰϑΣΠετϥοΩϯά͢Δ΍ͭ • Unity Ͱ͸ ARFoundation, ARKit Face Tracking plugin Λ࢖͏ 15
16. ARKit Face Tracking • Ϋϥογϡ͢Δ • σʔλ͸ڸ • ද৘ͷϚοϐϯά 16
17. Ϋϥογϡ͢Δ • ͠͹Β͘ಈ͔ͯ͠Δͱ EXC_BAD_ACCESS ͰམͪΔ • ARFoundation 3.x ͳΒ໰୊ͳ͍ • ͜Ε͕͋ͬͯ ARFoundation 4.x ʹ্͛ΒΕͣɺ Unity2020.2 ʹ ΋্͛ΒΕͳ͍ " • Issue Ͱ͸iOS14ͷ໰୊ͱݴΘΕ͍ͯΔ(मਖ਼Λ଴͔ͭ͠ͳͦ͞͏) > github.com/Unity-Technologies/arfoundation-samples/issues/ 707 17
18. Face Tracking ͷσʔλ͸ڸ • ARKit ͷ Face Tracking ͰऔಘͰ͖Δσʔλ͸ࠨӈ൓సͯ͠ೖͬͯ ͘Δ • ΧϝϥͷલͰӈΛ޲͘ͱࠨճసΛɺӈ໨Λด͡Δͱࠨ໨Λด ͡ΔΑ͏ͳσʔλΛฦ͢ 18
19. إͷ pose Λڸࣸ͠ʹ͢Δ rotation.x *= -1; rotation.y *= -1; position.z *= -1; 19
20. BlendShapeLocation ͷࠨӈΛೖΕସ ͑Δ 20
21. { BlendShapePreset.Blink_L, new [] { ARKitBlendShapeLocation.EyeBlinkRight } }, { BlendShapePreset.Blink_R, new [] { ARKitBlendShapeLocation.EyeBlinkLeft } } 21
22. ʮࠨΛݟ͍ͯΔʯ͸ɺʮ(޲͔ͬͯ)ࠨͷ໨͕಺ଆΛݟ͍ͯΔʯͱ ʮ(޲͔ͬͯ)ӈͷ໨͕֎ଆΛݟ͍ͯΔʯ { BlendShapePreset.LookLeft, new [] { ARKitBlendShapeLocation.EyeLookInLeft, ARKitBlendShapeLocation.EyeLookOutRight } }, { BlendShapePreset.LookRight, new [] { ARKitBlendShapeLocation.EyeLookOutLeft, ARKitBlendShapeLocation.EyeLookInRight } }, 22
23. ද৘ͷϚοϐϯά • ARKitͰऔಘͰ͖Δ Blend Shape Ωʔ͸(ଟ͘ͷ)Ξόλʔͷ࣋ͭͦ Εͱ͸ҟͳΔ • ಉ͡ Blend Shape ΛΞόλʔʹ༻ҙͯ͠ɺͦͷ··ύϥϝʔ λʔΛ൓өͤ͞ΔʮύʔϑΣΫτγϯΫʯͱݺ͹ΕΔ֓೦΋ ͋Δ 23
24. ද৘ͷϚοϐϯά • aiueo ͷαϯϓϧΛऔ͓͍ͬͯͯ࠷΋ۙࣅ͢Δ΋ͷΛ࠾༻ɺͰͩ ͍͍͍͍ͨײ͡ʹͳͬͨ • ۃ୺ʹֻ͚ͨΓɺ2൪໨ʹࣅͯΔͷΛࠞͥͨΓɺͪΌΜͱޱ Λด͡ΔΑ͏ʹ͢Δͷʹຊ౰͸΋ͬͱ͍Ζ͍Ζ΍ͬͯΔ • ͜͜ͷڍಈ͸ਓʹΑͬͯ޷Έ΋෼͔ΕΔͱࢥ͏ͷͰࣗ࡞ͯͯ͠ ໘ന͍ͱ͜Ζ 24
25. Leap Motion • ϋϯυδΣενϟʔίϯτϩʔϥʔ • 2012೥ͷσόΠε • ڈ೥(2020೥9݄) IR 170 ͱ͍͏݁ߏੑೳ্͕͕ͬͨධՁΩοτ͕ ൃച͞ΕͯΞπ͍ • Unity ϓϥάΠϯ͕ఏڙ͞Ε͍ͯΔ 25
26. Leap Motion • جຊతʹ͸ LeapServiceProvider or LeapXRServiceProvider ίϯϙʔωϯτΛ࢖͑͹࣮૷͸؆୯ • LeapServiceProvider ͸صஔ͖ • LeapXRServiceProvider ͸टͱ͔಄ʹϚ΢ϯτ͢Δ༻ • ࣗ෼Ͱ࣮૷͢Δ(pure class ʹ͍ͨ͠ͱ͔ɺصஔ͖ͱटϚ΢ϯτ Λ੾Γସ͑ΒΕΔΑ͏ʹ͍ͨ͠)৔߹͸ɺ PolicyFlag Λ੾Γସ͑ Δͷͱɺ࠷దͳ Interpolate ͷΦϓγϣϯ͕ҧ͏ͬΆ͍఺ʹ஫ҙ 26
27. ΞόλʔͷಡΈࠐΈɾදࣔɾ Ξχϝʔγϣϯ • VRM • Custom UTI • Animation Rigging 27
28. VRM • VRΞϓϦέʔγϣϯ޲͚ͷਓܕ3DΞόλʔσʔλΛѻ͏ͨΊͷ ϑΝΠϧϑΥʔϚοτ • UniVRM (C#ʹΑΔඪ४࣮૷)Λ࢖͏ • (׬શʹࣗ෼༻ͩͬͨΒΞϓϦʹຒΊࠐΜͰ΋ྑ͍͕ɺͲ͏ͤΞ όλʔ͸Α͘मਖ਼͢ΔͷͰϥϯλΠϜϩʔυʹରԠ͓ͯ͘͠ͱ ָ) 28
29. Custom UTI • iOS ͷڞ༗ϘλϯΛԡͨ͠ͱ͖ʹग़ͯ ͘ΔϦετʹ Unity ੡ΞϓϦΛฒ΂Δ • ͜ΕͰVRMϑΝΠϧΛ։͚Δͱ஍ຯʹ ศར • (׬શʹࣗ෼༻ͩͬͨΒΞϓϦʹຒΊ ࠐΜͰ΋ྑ͍͕ɺͲ͏ͤΞόλʔ͸Α ͘मਖ਼͢ΔͷͰϥϯλΠϜϩʔυʹର Ԡ͓ͯ͘͠ͱָ) 29
30. Info.plist Λ͕Μ͹ͬͯॻ͘ [PostProcessBuild] public static void OnPostProcessBuild(BuildTarget target, string path) { if (target != BuildTarget.iOS) return; var plistPath = path + "/Info.plist"; var plist = new PlistDocument(); plist.ReadFromString(File.ReadAllText(plistPath)); var rootDict = plist.root; var exportedTypeDeclarations = rootDict.CreateArray("UTExportedTypeDeclarations"); var exportedTypeDeclarationsDict = exportedTypeDeclarations.AddDict(); var typeConformsTo = exportedTypeDeclarationsDict.CreateArray("UTTypeConformsTo"); typeConformsTo.AddString("public.data"); exportedTypeDeclarationsDict.SetString("UTTypeDescription", "VRM"); exportedTypeDeclarationsDict.CreateArray("UTTypeIconFiles"); var typeIdentifier = $"{PlayerSettings.applicationIdentifier}.custom"; exportedTypeDeclarationsDict.SetString("UTTypeIdentifier", typeIdentifier); var typeTagSpecification = exportedTypeDeclarationsDict.CreateDict("UTTypeTagSpecification"); var typeTagSpecificationArray = typeTagSpecification.CreateArray("public.filename-extension"); typeTagSpecificationArray.AddString("vrm"); var documentTypes = rootDict.CreateArray("CFBundleDocumentTypes"); var documentTypesDict = documentTypes.AddDict(); documentTypesDict.SetString("CFBundleTypeName", "VRM"); documentTypesDict.SetString("CFBundleTypeRole", "Viewer"); documentTypesDict.SetString("LSHandlerRank", "Alternate"); var itemContentTypes = documentTypesDict.CreateArray("LSItemContentTypes"); itemContentTypes.AddString(typeIdentifier); File.WriteAllText(plistPath, plist.WriteToString()); } https:/ /gist.github.com/noir-neo/ 91e7a09e87e15280c1b2c70648b833f1 30
31. ͋ͱ͸URLͷϋϯυϦϯάͱಉ͡ @interface UrlHandlerAppController : UnityAppController @end @implementation UrlHandlerAppController - (BOOL)application:(UIApplication *)application openURL:(NSURL *)url options:(NSDictionary<UIApplicationOpenURLOptionsKey, id> *)options { // TODO: Handling return YES; } @end ͨͩ͜͜͠Ͱड͚औͬͨURI͸࣍ճىಈ࣌ʹ͸࢖͑ͳ͍ͷͰΞό λʔΛΩϟογϡ͓ͯ͘͠࢓૊ΈͳͲ͕ผ్ඞཁ 31
32. Animation Rigging • "ΞχϝʔγϣϯԽ͞Ε͍ͯΔεέϧτϯʹର࣮ͯ͠ߦ࣌ʹϓϩ γʔδϟϧͳϞʔγϣϯΛηοτΞοϓͰ͖ΔΑ͏ʹ͢Δ" • ཁ͸ΞχϝʔγϣϯΛϥϯλΠϜͰ͍͍ײ͡ʹ্ॻ͖Ͱ͖Δ • Unity2020.2 Ͱ verified ʹͳΓ·ͨ͠Ͷ • σϑΥϧτͷΞχϝʔγϣϯʹରͯ͠(τϥοΩϯάͨ͠)֤Ϙʔ ϯͷճసΛ্ॻ͖͢Δͷʹศར 32
33. Animation Rigging • TwoBoneIK ͕ඪ४Ͱೖ͍ͬͯΔͷͰ࿹ͷ IK ʹ΋࢖͑Δ(݁ߏ͍͍ ײ͡) • (಄ͱख͔Β਎ମશମΛΑ͠ͳʹIK͢ΔɺΈ͍ͨͳͷ͸ FinalIK ͱ͔࢖͏ํָ͕) • มͳճసΛೖΕͯ΋ muscle ͷൣғͰಈ͔ͯ͘͠ΕΔͷͰ࠷ѱͷ ഁ୼ͷ࢓ํ͸͠ͳ͍ 33
34. ө૾ͷૹग़ • ө૾ͷϛϥʔ • Spout / Syphon / NDI 34
35. ࣗ෼ͰݟΔө૾͸ڸʹ͢Δ • Unity Ͱ Camera ͷө૾ΛϑϦοϓ͢Δʹ͸ projectionMatrix Λ͍ ͡ΔΒ͍͠ • culling ͱ͔ΊΜͲͦ͘͞͏ • Render Texture ʹඳ͍ͨ΋ͷΛ Canvas ͷ UV Ͱ൓సͤ͞Δͷ͕ ָ • ͍ͭͰʹɺૹग़͢Δө૾ʹ͸ UI ΛೖΕͨ͘ͳ͍ͱ͔΋͋Δͷ Ͱແବ͸ͳ͍ 35
36. Spout / Syphon / NDI • ө૾(ͷϑϨʔϜ)Λڞ༗͢Δϓϩτίϧ • Spout(Windows) / Syphon(mac) ͸Ϛγϯ಺ • NDI ͸ϩʔΧϧωοτϫʔΫ಺ • Unity Ͱ͸ Klak{Spout|Syphon|NDI} Λ࢖͏ • keijiroਆ ! 36
37. Spout / Syphon / NDI • ಁաө૾΋ѻ͑ͯ௚઀഑৴ιϑτʹೖΕΒΕΔͷͰάϦʔϯ όοΫͰൈ͍ͨΓ͠ͳͯ͘Α͍ͷͰΑ͍ 37
38. ·ͱΊ • Unity ͰΩϛ͚ͩͷόʔνϟϧΧϝϥΛ࡞ͬͯɺϏσΦϛʔςΟ ϯάͰࠩΛ෇͚Α͏ʂ 38