ARKit3

Transcript

1. ARKit3 @TachibanaKaoru 2019/6/11 WWDC Catch-up

2. About Me @TachibanaKaoru Freelance iOS Engineer झຯɿϘϧμϦϯάɺBotWͰཾͷྡྷङ ࠷ۙͷςʔϚɿཱྀΛ͠ͳ͕Β࢓ࣄΛ͢Δ όϦౡɺαϯτϦʔχౡɺόϯίΫɺόϧη ϩφɺΞΠεϥϯυ

3. iOSDC 2019 2019/9/5 - 2019/9/7 At Waseda University Nishi-waseda campus

   CFP ืूதʂʢ6/17·Ͱʣ

4. ARKit

5. ARKit iOS11͔Β౥ࡌ͞Ε͍ͯΔɺARઐ༻ϑϨʔϜϫʔΫ Device with A9 chip~ Unity ARKit Plugin /

   Unity AR Foundation

6. ARKit Tracking Scene Understanding Rendering

7. AR Rendering SceneKit SpriteKit Metal RealityKit ARKit3

8. ARKit 1.0 2017/06 ۭؒݕ஌ɺฏ໘ʢਫฏ໘ʣݕ஌ɺإද৘ݕ஌ 1.5 2018/01 ฏ໘ʢਨ௚໘ݕ஌ʣɺը૾ݕ஌ 2.0 2018/06 ը૾ݕ஌Ϟʔυ௥ՃɺARঢ়ଶอଘରԠɺ

   Object SacnରԠɺࢹઢɾઉݕ஌ 3.0 2019/06 BodyScanɺPeople Occupation

9. What's new in ARKit3 People Occlusion Body Detection and more...

10. People Occlusion

11. ͜͏͍͏എܠʹAR ObjectΛஔ͘ͱ……

12. ARKit2.0Ͱ͸͜͏ͳΓ·͢

13. ARKit3.0Ͱ͸͜͏ͳΓ·ͨ͠ɻ

14. ARKit2 ARKit3

15. None

16. People Occlusion ը໘಺ʹෳ਺ਓͷਓ͕͍ͯ΋ରԠՄೳ ਓͷશମ૾Ͱ͸ͳ͘ɺख΍଍ͷҰ෦͕͏͍ͭͬͯΔ͚ͩͰ΋ Occulution Մೳɻ

17. People Occlusion set frameSemantics to ARWorldTrackingConfiguration

18. Frame semantics personSegmentation depthΛߟྀ͠ͳ͍ ਓ͸ৗʹObjectͷલ໘

19. Frame semantics personSegmentationWithDepth depthΛߟྀ͢Δ ਓ͸depthʹԠͯ͡લ໘ɾഎ໘ͱͳΔ

20. AR FrameSemantics Configuration frameSemantics ARWorldTracking personSegmentation people always front ARWorldTracking

    personSegmentationWithDepth people with depth

21. SwiftStrike People Occlusion Λར༻ͨ͠ ARήʔϜ WWDCձ৔Ͱৗʹ։࠵͞Εͯ ͓Γɺ੝Γ্͕͍ͬͯ·͠ ͨɻ

22. None

23. SwiftStrikeTabletop SwiftStrikeΛখ͍͞ςʔϒϧͰ༡΂Δαϯϓϧίʔυ https://developer.apple.com/documentation/realitykit/ creating_a_game_with_reality_composer

24. Body Detection

25. Body Detection ARۭؒͷਓମΛೝࣝ͠ɺ࠲ඪΛऔಘ 3DόʔνϟϧϞσϧΛਓମͷҐஔʹ Ԡͯ͡഑ஔ 2D Body Detection / 3D

    Body Detection

26. 2D Body Detection

27. 3D Body Detection

28. 3D Skeleton 93ͷjointʢؔઅʣ jointʢؔઅʣʹ͸ҐஔΛج४ͱͨ͠਌ࢠؔ܎͕͋Δ

29. 3D Skeleton index 0 : root index 1 : hip_joint

    index 2 : left_upLeg_joint index 3 : left_leg_joint ...... index 91 : left_eyeLowerLid_joint index 92 : left_eyeBall_joint

30. 3D Skeleton index 0 : root index 1 : hip_joint

    index 2 : left_upLeg_joint index 3 : left_leg_joint ...... index 91 : left_eyeLowerLid_joint index 92 : left_eyeBall_joint skeleton3D.definition.parentIndices[index]

31. 3D Skeleton index 0 : root index 1 : hip_joint

    index 2 : left_upLeg_joint index 3 : left_leg_joint ...... index 91 : left_eyeLowerLid_joint index 92 : left_eyeBall_joint > parent : root > parent : hip_joint > parent : hip_upLeg_joint hips_joint root left_upLeg_joint left_leg_joint left_foot_joint parent

32. 3D Skeleton ਖ਼͍͠Ґஔ৘ใ͕͋Δjointͱɺͳ͍joint͕͋Δɻ ࢦɺ໨ඓɺͭ·ઌͳͲ͸ख଍ͷҐஔʹԊ͏Α͏ ʹ഑ஔ͞Ε͓ͯΓɺඞͣ͠΋ਖ਼͍͠ҐஔͰ͸ͳ ͍ɻ

33. 3D Skeleton Local TransformͱModel Transform let jointLocalTransforms = skeleton3D.jointLocalTransforms let

    jointModelTransforms = skeleton3D.jointModelTransforms

34. localTransform

35. modelTransform

36. Load my own 3D model ࣗ෼Ͱ࡞ͬͨ3DϞσϧΛskeltonʹ ͋ΘͤͯϦΞϧλΠϜදࣔ͢Δ͜ͱ ΋Մೳɻ ϞσϧϑΥʔϚοτͳͲ͸͓ͬͯެ ։͞ΕΔΑ͏ɻ

37. AR Configuration Mode Camera iOS AROrientationTracking Back Camera iOS 11.0+

    ARWorldTracking Back Camera iOS 11.0+ ARFaceTracking Front Camera iOS 11.0+ ARImageTracking Back Camera iOS 12.0+ ARObjectScanning Back Camera iOS 12.0+ ARBodyTracking Back Camera iOS 13.0+ ARPositionTracking Back Camera iOS 13.0+

38. AR FrameSemantics Configuration frameSemantics ARBodyTracking - 3D Body / 2D

    Body ARWorldTracking, ARImageTracking, AROrientationTracking bodyDetection 2D Body

39. RealityKit

40. RealityKit ARʹಛԽͨ͠3DඳըϥΠϒϥϦ iOS13 ~ RealityKitͷΈͰαϙʔτ͍ͯ͠Δػೳ΋ɻ Collavotive rendering

41. Collaborative AR

42. Collaborative AR ARKit2.0Ͱαϙʔτ͞ΕͨWorldMapͷڞ༗ػೳ͕όʔδϣϯΞοϓ RealityKitͷ৔߹ʹ͸ɺࣗಈతʹϨϯμϦϯά·Ͱͯ͘͠ΕΔɻ ૬खͷҐஔ΋ೝࣝͰ͖Δ

43. None
44. None

45. AR Quick Look

46. USDZ USD : Universal Scene Description 3D Model Format from

    Pixar

47. AR Quick Look SafariɺϝϞɺϝʔϧͳͲͰUSDZϑΝΠϧΛ։͘ ͱɺࣗಈతʹΧϝϥө૾Λ༻͍ͨARͱͯ͠දࣔ͞ ΕΔ ΞϓϦΛ࡞ΒͣʹARମݧΛఏڙͰ͖Δ iOS12 ~

48. ࣮ྫ NHK ౦ژిྗ ෱ౡୈҰݪࢠྗൃిॴAR https://www.nhk.or.jp/vr/AR/1fAR/

49. ࣮ྫ NHK ڪཽ௒AR ςΟϥϊα΢ϧεɺτϦέϥτϓεɺεϐϊα΢ϧε https://www.nhk.or.jp/special/dino/ar.html

50. What's new in iOS13 AR Quick Look λοϓͳͲͷΠϕϯτʹରԠ ΞχϝʔγϣϯʹରԠ ϢʔβʔΠϯλϥΫγϣϯΛؚΉ؆୯ͳήʔϜ͕USDZͷΈͰ࡞ΕΔΑ͏

    ʹɻ People OcclusionʹରԠ

51. RealityComposer

52. RealityComposer new 3D modeling tool for iPhone, iPad and Mac

    iOS version͸Test FlightͰ഑෍த (Apple Developer Accountඞਢ)
53. None
54. None

55. How to use RealityComposer Creating 3D Content with Reality Composer

    https://developer.apple.com/documentation/realitykit/ creating_3d_content_with_reality_composer

56. and more on ARKit3...

57. Back and Front Camera

58. Implementing with AR

59. AR ࣮૷࣌ͷࠔ೉ ޫྔ ޫྔෆ଍ ԰֎ͰͷAR࣮ߦ ։ൃ࣌ͷ؀ڥͱϢʔβʔ࣮ߦ࣌ͷ 3Dঢ়گ͕ҟͳΔ

60. AR Coaching View ARKitͰ༻ҙ͞ΕͨɺϢʔβʔʹঢ়گΛද͢ϔϧϓϏϡʔ Ξχϝʔγϣϯ෇͖

61. Record and Replay σόοά࣌ͷMovieΛXcodeͰऩूͰ͖ΔΑ͏ʹɻ

62. Record and Replay Check : ARKit Replay data

63. Check Thermal State AR͸୺຤΁ͷॲཧෛ୲͕ߴ͍ Թ౓มԽΛແࢹ͍ͯ͠ΔͱɺTemperature AlertʹͳΔ ڪΕ͕͋Δ Xcode11 ͷ Device

    Condition ը໘ͰσόοάՄೳ

64. Check Thermal State ProcessInfo.themalStateDidChangeNotification Λ؂ࢹ ProcessInfo.ThermalState ͷ஋ʹରԠ

65. None

66. Recap ARKit3͸ࠓ·ͰͰ࠷େͷΞοϓσʔτɻ͍Ζ͍ΖͰ͖ΔͷͰɺ͍ͬͯ͡ ͍Δ͚ͩͰָ͍͠ɻ iOS13ͷϩʔϯνͰҰ൪ Apple ͕ push͍ͯ͠Δͷ͸ AR ! ळʹARମݧΛఏڙͰ͖ΔΑ͏ʹͳ͍ͬͯΕ͹ɺApple

    Store Top Feature ! ARViewͰ࣮૷͠ͳͯ͘΋USDZϑΝΠϧΛஔ͍ͯ؆୯ʹ࣮૷͢Δख΋͋ Γ·͢ɻ

67. Reference ARKit https://developer.apple.com/documentation/arkit AR Quick Look https://developer.apple.com/augmented-reality/quick-look/

68. Reference Introducing ARKit 3 https://developer.apple.com/videos/play/wwdc2019/604 Building Collaborative AR Experiences https://developer.apple.com/videos/play/wwdc2019/610

    Building AR Experiences with Reality Composer https://developer.apple.com/videos/play/wwdc2019/609 Introducing RealityKit and Reality Composer https://developer.apple.com/videos/play/wwdc2019/603

69. Reference Working with USD https://developer.apple.com/videos/play/wwdc2019/602 Building Apps with RealityKit https://developer.apple.com/videos/play/wwdc2019/605

    Advances in AR Quick Look https://developer.apple.com/videos/play/wwdc2019/612/ Bringing People into AR https://developer.apple.com/videos/play/wwdc2019/607

70. Reference Creating Great Apps Using Core ML and ARKit https://developer.apple.com/videos/play/wwdc2019/228

    Designing for Adverse Network and Temperature Conditions https://developer.apple.com/videos/play/wwdc2019/422/