アバターミーティングを支える技術

Transcript

1. ΞόλʔϛʔςΟϯάΛࢧ͑ Δٕज़ 2021/1/13 Gotanda.unity #16 Ͷ͓ΓΜ 1

2. noir_neo • Ͷ͓ΓΜ • Ϋϥελʔגࣜձࣾ • ιϑτ΢ΣΞΤϯδχΞ • Ϋϥελʔͷϋτ •

   2021೥ͷ໨ඪ͸ʮ͓෩࿊ͷ࣌ܭͷి ஑Λ׵͑Δʯ 2

3. ͱ͜ΖͰΈͳ͞ΜɺϏ σΦϛʔςΟϯά͠· ͢ΑͶ? 3

4. ฐࣾɺVRϛʔςΟϯά͸౰ વ΍Γ·͕͢ɺϏσΦϛʔ ςΟϯά΋͠·͢ VRHMDඃͬͯϛʔςΟϯάϊʔτऔΔͷ͸େมͳͨΊɻ 4

5. ͜Μͳײ͡ → 5

6. 6

7. ҰൠతͳϏσΦϛʔςΟϯά ͷޫܠͰ͢Ͷ? (ͦ͏Ͱ͋ͬͯ΄͍͠) 7

8. ͍ΘΏΔ VTuber ΞϓϦ ͍Ζ͍Ζ͋Δɻ 8

9. ͍Ζ͍Ζࢼͨ݁͠Ռɺࣗ෼Ͱ ࡞Γ࢝Ίͨ ͦ͏͍͏͜ͱ΋͋Δɻ 9

10. ࿩͢͜ͱ • ͍ΘΏΔ VTuber ΞϓϦͷߏ੒ྫ • ஌ݟɺͩ͜ΘΓϙΠϯτͳͲ 10

11. ݟग़͠ • γεςϜ֓ཁ • إͱखͷτϥοΩϯά • ΞόλʔͷಡΈࠐΈɾදࣔɾΞχϝʔγϣϯ • ө૾ͷૹग़ 11

12. γεςϜ֓ཁ 12

13. 13

14. إͱखͷτϥοΩϯά • ARKit • Leap Motion 14

15. ARKit Face Tracking • iOS σόΠεͷϑϩϯτΧϝϥͰϑΣΠετϥοΩϯά͢Δ΍ͭ • Unity Ͱ͸ ARFoundation,

    ARKit Face Tracking plugin Λ࢖͏ 15

16. ARKit Face Tracking • Ϋϥογϡ͢Δ • σʔλ͸ڸ • ද৘ͷϚοϐϯά 16

17. Ϋϥογϡ͢Δ • ͠͹Β͘ಈ͔ͯ͠Δͱ EXC_BAD_ACCESS ͰམͪΔ • ARFoundation 3.x ͳΒ໰୊ͳ͍ •

    ͜Ε͕͋ͬͯ ARFoundation 4.x ʹ্͛ΒΕͣɺ Unity2020.2 ʹ ΋্͛ΒΕͳ͍ " • Issue Ͱ͸iOS14ͷ໰୊ͱݴΘΕ͍ͯΔ(मਖ਼Λ଴͔ͭ͠ͳͦ͞͏) > github.com/Unity-Technologies/arfoundation-samples/issues/ 707 17

18. Face Tracking ͷσʔλ͸ڸ • ARKit ͷ Face Tracking ͰऔಘͰ͖Δσʔλ͸ࠨӈ൓సͯ͠ೖͬͯ ͘Δ

    • ΧϝϥͷલͰӈΛ޲͘ͱࠨճసΛɺӈ໨Λด͡Δͱࠨ໨Λด ͡ΔΑ͏ͳσʔλΛฦ͢ 18

19. إͷ pose Λڸࣸ͠ʹ͢Δ rotation.x *= -1; rotation.y *= -1; position.z

    *= -1; 19

20. BlendShapeLocation ͷࠨӈΛೖΕସ ͑Δ 20

21. { BlendShapePreset.Blink_L, new [] { ARKitBlendShapeLocation.EyeBlinkRight } }, { BlendShapePreset.Blink_R,

    new [] { ARKitBlendShapeLocation.EyeBlinkLeft } } 21

22. ʮࠨΛݟ͍ͯΔʯ͸ɺʮ(޲͔ͬͯ)ࠨͷ໨͕಺ଆΛݟ͍ͯΔʯͱ ʮ(޲͔ͬͯ)ӈͷ໨͕֎ଆΛݟ͍ͯΔʯ { BlendShapePreset.LookLeft, new [] { ARKitBlendShapeLocation.EyeLookInLeft, ARKitBlendShapeLocation.EyeLookOutRight }

    }, { BlendShapePreset.LookRight, new [] { ARKitBlendShapeLocation.EyeLookOutLeft, ARKitBlendShapeLocation.EyeLookInRight } }, 22

23. ද৘ͷϚοϐϯά • ARKitͰऔಘͰ͖Δ Blend Shape Ωʔ͸(ଟ͘ͷ)Ξόλʔͷ࣋ͭͦ Εͱ͸ҟͳΔ • ಉ͡ Blend

    Shape ΛΞόλʔʹ༻ҙͯ͠ɺͦͷ··ύϥϝʔ λʔΛ൓өͤ͞ΔʮύʔϑΣΫτγϯΫʯͱݺ͹ΕΔ֓೦΋ ͋Δ 23

24. ද৘ͷϚοϐϯά • aiueo ͷαϯϓϧΛऔ͓͍ͬͯͯ࠷΋ۙࣅ͢Δ΋ͷΛ࠾༻ɺͰͩ ͍͍͍͍ͨײ͡ʹͳͬͨ • ۃ୺ʹֻ͚ͨΓɺ2൪໨ʹࣅͯΔͷΛࠞͥͨΓɺͪΌΜͱޱ Λด͡ΔΑ͏ʹ͢Δͷʹຊ౰͸΋ͬͱ͍Ζ͍Ζ΍ͬͯΔ • ͜͜ͷڍಈ͸ਓʹΑͬͯ޷Έ΋෼͔ΕΔͱࢥ͏ͷͰࣗ࡞ͯͯ͠

    ໘ന͍ͱ͜Ζ 24

25. Leap Motion • ϋϯυδΣενϟʔίϯτϩʔϥʔ • 2012೥ͷσόΠε • ڈ೥(2020೥9݄) IR 170

    ͱ͍͏݁ߏੑೳ্͕͕ͬͨධՁΩοτ͕ ൃച͞ΕͯΞπ͍ • Unity ϓϥάΠϯ͕ఏڙ͞Ε͍ͯΔ 25

26. Leap Motion • جຊతʹ͸ LeapServiceProvider or LeapXRServiceProvider ίϯϙʔωϯτΛ࢖͑͹࣮૷͸؆୯ • LeapServiceProvider

    ͸صஔ͖ • LeapXRServiceProvider ͸टͱ͔಄ʹϚ΢ϯτ͢Δ༻ • ࣗ෼Ͱ࣮૷͢Δ(pure class ʹ͍ͨ͠ͱ͔ɺصஔ͖ͱटϚ΢ϯτ Λ੾Γସ͑ΒΕΔΑ͏ʹ͍ͨ͠)৔߹͸ɺ PolicyFlag Λ੾Γସ͑ Δͷͱɺ࠷దͳ Interpolate ͷΦϓγϣϯ͕ҧ͏ͬΆ͍఺ʹ஫ҙ 26

27. ΞόλʔͷಡΈࠐΈɾදࣔɾ Ξχϝʔγϣϯ • VRM • Custom UTI • Animation Rigging

    27

28. VRM • VRΞϓϦέʔγϣϯ޲͚ͷਓܕ3DΞόλʔσʔλΛѻ͏ͨΊͷ ϑΝΠϧϑΥʔϚοτ • UniVRM (C#ʹΑΔඪ४࣮૷)Λ࢖͏ • (׬શʹࣗ෼༻ͩͬͨΒΞϓϦʹຒΊࠐΜͰ΋ྑ͍͕ɺͲ͏ͤΞ όλʔ͸Α͘मਖ਼͢ΔͷͰϥϯλΠϜϩʔυʹରԠ͓ͯ͘͠ͱ

    ָ) 28

29. Custom UTI • iOS ͷڞ༗ϘλϯΛԡͨ͠ͱ͖ʹग़ͯ ͘ΔϦετʹ Unity ੡ΞϓϦΛฒ΂Δ • ͜ΕͰVRMϑΝΠϧΛ։͚Δͱ஍ຯʹ

    ศར • (׬શʹࣗ෼༻ͩͬͨΒΞϓϦʹຒΊ ࠐΜͰ΋ྑ͍͕ɺͲ͏ͤΞόλʔ͸Α ͘मਖ਼͢ΔͷͰϥϯλΠϜϩʔυʹର Ԡ͓ͯ͘͠ͱָ) 29

30. Info.plist Λ͕Μ͹ͬͯॻ͘ [PostProcessBuild] public static void OnPostProcessBuild(BuildTarget target, string path)

    { if (target != BuildTarget.iOS) return; var plistPath = path + "/Info.plist"; var plist = new PlistDocument(); plist.ReadFromString(File.ReadAllText(plistPath)); var rootDict = plist.root; var exportedTypeDeclarations = rootDict.CreateArray("UTExportedTypeDeclarations"); var exportedTypeDeclarationsDict = exportedTypeDeclarations.AddDict(); var typeConformsTo = exportedTypeDeclarationsDict.CreateArray("UTTypeConformsTo"); typeConformsTo.AddString("public.data"); exportedTypeDeclarationsDict.SetString("UTTypeDescription", "VRM"); exportedTypeDeclarationsDict.CreateArray("UTTypeIconFiles"); var typeIdentifier = $"{PlayerSettings.applicationIdentifier}.custom"; exportedTypeDeclarationsDict.SetString("UTTypeIdentifier", typeIdentifier); var typeTagSpecification = exportedTypeDeclarationsDict.CreateDict("UTTypeTagSpecification"); var typeTagSpecificationArray = typeTagSpecification.CreateArray("public.filename-extension"); typeTagSpecificationArray.AddString("vrm"); var documentTypes = rootDict.CreateArray("CFBundleDocumentTypes"); var documentTypesDict = documentTypes.AddDict(); documentTypesDict.SetString("CFBundleTypeName", "VRM"); documentTypesDict.SetString("CFBundleTypeRole", "Viewer"); documentTypesDict.SetString("LSHandlerRank", "Alternate"); var itemContentTypes = documentTypesDict.CreateArray("LSItemContentTypes"); itemContentTypes.AddString(typeIdentifier); File.WriteAllText(plistPath, plist.WriteToString()); } https:/ /gist.github.com/noir-neo/ 91e7a09e87e15280c1b2c70648b833f1 30

31. ͋ͱ͸URLͷϋϯυϦϯάͱಉ͡ @interface UrlHandlerAppController : UnityAppController @end @implementation UrlHandlerAppController - (BOOL)application:(UIApplication

    *)application openURL:(NSURL *)url options:(NSDictionary<UIApplicationOpenURLOptionsKey, id> *)options { // TODO: Handling return YES; } @end ͨͩ͜͜͠Ͱड͚औͬͨURI͸࣍ճىಈ࣌ʹ͸࢖͑ͳ͍ͷͰΞό λʔΛΩϟογϡ͓ͯ͘͠࢓૊ΈͳͲ͕ผ్ඞཁ 31

32. Animation Rigging • "ΞχϝʔγϣϯԽ͞Ε͍ͯΔεέϧτϯʹର࣮ͯ͠ߦ࣌ʹϓϩ γʔδϟϧͳϞʔγϣϯΛηοτΞοϓͰ͖ΔΑ͏ʹ͢Δ" • ཁ͸ΞχϝʔγϣϯΛϥϯλΠϜͰ͍͍ײ͡ʹ্ॻ͖Ͱ͖Δ • Unity2020.2 Ͱ

    verified ʹͳΓ·ͨ͠Ͷ • σϑΥϧτͷΞχϝʔγϣϯʹରͯ͠(τϥοΩϯάͨ͠)֤Ϙʔ ϯͷճసΛ্ॻ͖͢Δͷʹศར 32

33. Animation Rigging • TwoBoneIK ͕ඪ४Ͱೖ͍ͬͯΔͷͰ࿹ͷ IK ʹ΋࢖͑Δ(݁ߏ͍͍ ײ͡) • (಄ͱख͔Β਎ମશମΛΑ͠ͳʹIK͢ΔɺΈ͍ͨͳͷ͸

    FinalIK ͱ͔࢖͏ํָ͕) • มͳճసΛೖΕͯ΋ muscle ͷൣғͰಈ͔ͯ͘͠ΕΔͷͰ࠷ѱͷ ഁ୼ͷ࢓ํ͸͠ͳ͍ 33

34. ө૾ͷૹग़ • ө૾ͷϛϥʔ • Spout / Syphon / NDI 34

35. ࣗ෼ͰݟΔө૾͸ڸʹ͢Δ • Unity Ͱ Camera ͷө૾ΛϑϦοϓ͢Δʹ͸ projectionMatrix Λ͍ ͡ΔΒ͍͠ •

    culling ͱ͔ΊΜͲͦ͘͞͏ • Render Texture ʹඳ͍ͨ΋ͷΛ Canvas ͷ UV Ͱ൓సͤ͞Δͷ͕ ָ • ͍ͭͰʹɺૹग़͢Δө૾ʹ͸ UI ΛೖΕͨ͘ͳ͍ͱ͔΋͋Δͷ Ͱແବ͸ͳ͍ 35

36. Spout / Syphon / NDI • ө૾(ͷϑϨʔϜ)Λڞ༗͢Δϓϩτίϧ • Spout(Windows) /

    Syphon(mac) ͸Ϛγϯ಺ • NDI ͸ϩʔΧϧωοτϫʔΫ಺ • Unity Ͱ͸ Klak{Spout|Syphon|NDI} Λ࢖͏ • keijiroਆ ! 36

37. Spout / Syphon / NDI • ಁաө૾΋ѻ͑ͯ௚઀഑৴ιϑτʹೖΕΒΕΔͷͰάϦʔϯ όοΫͰൈ͍ͨΓ͠ͳͯ͘Α͍ͷͰΑ͍ 37

38. ·ͱΊ • Unity ͰΩϛ͚ͩͷόʔνϟϧΧϝϥΛ࡞ͬͯɺϏσΦϛʔςΟ ϯάͰࠩΛ෇͚Α͏ʂ 38

39. 39