ARKit3

Transcript

1. ARKit3
   @TachibanaKaoru
   2019/6/11 WWDC Catch-up
   

   View Slide

2. About Me
   @TachibanaKaoru
   Freelance iOS Engineer
   झຯɿϘϧμϦϯάɺBotWͰཾͷྡྷङ
   ࠷ۙͷςʔϚɿཱྀΛ͠ͳ͕Β࢓ࣄΛ͢Δ
   όϦౡɺαϯτϦʔχౡɺόϯίΫɺόϧη
   ϩφɺΞΠεϥϯυ
   

   View Slide

3. iOSDC 2019
   2019/9/5 - 2019/9/7
   At Waseda University
   Nishi-waseda campus
   CFP ืूதʂʢ6/17·Ͱʣ
   

   View Slide

4. ARKit
   

   View Slide

5. ARKit
   iOS11͔Β౥ࡌ͞Ε͍ͯΔɺARઐ༻ϑϨʔϜϫʔΫ
   Device with A9 chip~
   Unity ARKit Plugin / Unity AR Foundation
   

   View Slide

6. ARKit
   Tracking Scene Understanding Rendering
   

   View Slide

7. AR Rendering
   SceneKit SpriteKit Metal RealityKit
   ARKit3
   

   View Slide

8. ARKit
   1.0 2017/06 ۭؒݕ஌ɺฏ໘ʢਫฏ໘ʣݕ஌ɺإද৘ݕ஌
   1.5 2018/01 ฏ໘ʢਨ௚໘ݕ஌ʣɺը૾ݕ஌
   2.0 2018/06
   ը૾ݕ஌Ϟʔυ௥ՃɺARঢ়ଶอଘରԠɺ
   Object SacnରԠɺࢹઢɾઉݕ஌
   3.0 2019/06 BodyScanɺPeople Occupation
   

   View Slide

9. What's new in ARKit3
   People Occlusion
   Body Detection
   and more...
   

   View Slide

10. People Occlusion
    

    View Slide

11. ͜͏͍͏എܠʹAR ObjectΛஔ͘ͱ……
    

    View Slide

12. ARKit2.0Ͱ͸͜͏ͳΓ·͢
    

    View Slide

13. ARKit3.0Ͱ͸͜͏ͳΓ·ͨ͠ɻ
    

    View Slide

14. ARKit2 ARKit3
    

    View Slide

15. View Slide

16. People Occlusion
    ը໘಺ʹෳ਺ਓͷਓ͕͍ͯ΋ରԠՄೳ
    ਓͷશମ૾Ͱ͸ͳ͘ɺख΍଍ͷҰ෦͕͏͍ͭͬͯΔ͚ͩͰ΋ Occulution
    Մೳɻ
    

    View Slide

17. People Occlusion
    set frameSemantics to ARWorldTrackingConfiguration
    

    View Slide

18. Frame semantics
    personSegmentation
    depthΛߟྀ͠ͳ͍
    ਓ͸ৗʹObjectͷલ໘
    

    View Slide

19. Frame semantics
    personSegmentationWithDepth
    depthΛߟྀ͢Δ
    ਓ͸depthʹԠͯ͡લ໘ɾഎ໘ͱͳΔ
    

    View Slide

20. AR FrameSemantics
    Configuration frameSemantics
    ARWorldTracking personSegmentation people always front
    ARWorldTracking personSegmentationWithDepth people with depth
    

    View Slide

21. SwiftStrike
    People Occlusion Λར༻ͨ͠
    ARήʔϜ
    WWDCձ৔Ͱৗʹ։࠵͞Εͯ
    ͓Γɺ੝Γ্͕͍ͬͯ·͠
    ͨɻ
    

    View Slide

22. View Slide

23. SwiftStrikeTabletop
    SwiftStrikeΛখ͍͞ςʔϒϧͰ༡΂Δαϯϓϧίʔυ
    https://developer.apple.com/documentation/realitykit/
    creating_a_game_with_reality_composer
    

    View Slide

24. Body Detection
    

    View Slide

25. Body Detection
    ARۭؒͷਓମΛೝࣝ͠ɺ࠲ඪΛऔಘ
    3DόʔνϟϧϞσϧΛਓମͷҐஔʹ
    Ԡͯ͡഑ஔ
    2D Body Detection / 3D Body
    Detection
    

    View Slide

26. 2D Body Detection
    

    View Slide

27. 3D Body Detection
    

    View Slide

28. 3D Skeleton
    93ͷjointʢؔઅʣ
    jointʢؔઅʣʹ͸ҐஔΛج४ͱͨ͠਌ࢠؔ܎͕͋Δ
    

    View Slide

29. 3D Skeleton
    index 0 : root
    index 1 : hip_joint
    index 2 : left_upLeg_joint
    index 3 : left_leg_joint
    ......
    index 91 : left_eyeLowerLid_joint
    index 92 : left_eyeBall_joint
    

    View Slide

30. 3D Skeleton
    index 0 : root
    index 1 : hip_joint
    index 2 : left_upLeg_joint
    index 3 : left_leg_joint
    ......
    index 91 : left_eyeLowerLid_joint
    index 92 : left_eyeBall_joint
    skeleton3D.definition.parentIndices[index]
    

    View Slide

31. 3D Skeleton
    index 0 : root
    index 1 : hip_joint
    index 2 : left_upLeg_joint
    index 3 : left_leg_joint
    ......
    index 91 : left_eyeLowerLid_joint
    index 92 : left_eyeBall_joint
    > parent : root
    > parent : hip_joint
    > parent : hip_upLeg_joint
    hips_joint
    root
    left_upLeg_joint
    left_leg_joint
    left_foot_joint
    parent
    

    View Slide

32. 3D Skeleton
    ਖ਼͍͠Ґஔ৘ใ͕͋Δjointͱɺͳ͍joint͕͋Δɻ
    ࢦɺ໨ඓɺͭ·ઌͳͲ͸ख଍ͷҐஔʹԊ͏Α͏
    ʹ഑ஔ͞Ε͓ͯΓɺඞͣ͠΋ਖ਼͍͠ҐஔͰ͸ͳ
    ͍ɻ
    

    View Slide

33. 3D Skeleton
    Local TransformͱModel Transform
    let jointLocalTransforms = skeleton3D.jointLocalTransforms
    let jointModelTransforms = skeleton3D.jointModelTransforms
    

    View Slide

34. localTransform
    

    View Slide

35. modelTransform
    

    View Slide

36. Load my own 3D model
    ࣗ෼Ͱ࡞ͬͨ3DϞσϧΛskeltonʹ
    ͋ΘͤͯϦΞϧλΠϜදࣔ͢Δ͜ͱ
    ΋Մೳɻ
    ϞσϧϑΥʔϚοτͳͲ͸͓ͬͯެ
    ։͞ΕΔΑ͏ɻ
    

    View Slide

37. AR Configuration Mode
    Camera iOS
    AROrientationTracking Back Camera iOS 11.0+
    ARWorldTracking Back Camera iOS 11.0+
    ARFaceTracking Front Camera iOS 11.0+
    ARImageTracking Back Camera iOS 12.0+
    ARObjectScanning Back Camera iOS 12.0+
    ARBodyTracking Back Camera iOS 13.0+
    ARPositionTracking Back Camera iOS 13.0+
    

    View Slide

38. AR FrameSemantics
    Configuration frameSemantics
    ARBodyTracking - 3D Body / 2D Body
    ARWorldTracking, ARImageTracking,
    AROrientationTracking
    bodyDetection 2D Body
    

    View Slide

39. RealityKit
    

    View Slide

40. RealityKit
    ARʹಛԽͨ͠3DඳըϥΠϒϥϦ
    iOS13 ~
    RealityKitͷΈͰαϙʔτ͍ͯ͠Δػೳ΋ɻ
    Collavotive rendering
    

    View Slide

41. Collaborative AR
    

    View Slide

42. Collaborative AR
    ARKit2.0Ͱαϙʔτ͞ΕͨWorldMapͷڞ༗ػೳ͕όʔδϣϯΞοϓ
    RealityKitͷ৔߹ʹ͸ɺࣗಈతʹϨϯμϦϯά·Ͱͯ͘͠ΕΔɻ
    ૬खͷҐஔ΋ೝࣝͰ͖Δ
    

    View Slide

43. View Slide

44. View Slide

45. AR Quick Look
    

    View Slide

46. USDZ
    USD : Universal Scene Description
    3D Model Format from Pixar
    

    View Slide

47. AR Quick Look
    SafariɺϝϞɺϝʔϧͳͲͰUSDZϑΝΠϧΛ։͘
    ͱɺࣗಈతʹΧϝϥө૾Λ༻͍ͨARͱͯ͠දࣔ͞
    ΕΔ
    ΞϓϦΛ࡞ΒͣʹARମݧΛఏڙͰ͖Δ
    iOS12 ~
    

    View Slide

48. ࣮ྫ
    NHK
    ౦ژిྗ ෱ౡୈҰݪࢠྗൃిॴAR
    https://www.nhk.or.jp/vr/AR/1fAR/
    

    View Slide

49. ࣮ྫ
    NHK
    ڪཽ௒AR
    ςΟϥϊα΢ϧεɺτϦέϥτϓεɺεϐϊα΢ϧε
    https://www.nhk.or.jp/special/dino/ar.html
    

    View Slide

50. What's new in iOS13 AR Quick Look
    λοϓͳͲͷΠϕϯτʹରԠ
    ΞχϝʔγϣϯʹରԠ
    ϢʔβʔΠϯλϥΫγϣϯΛؚΉ؆୯ͳήʔϜ͕USDZͷΈͰ࡞ΕΔΑ͏
    ʹɻ
    People OcclusionʹରԠ
    

    View Slide

51. RealityComposer
    

    View Slide

52. RealityComposer
    new 3D modeling tool for iPhone, iPad and Mac
    iOS version͸Test FlightͰ഑෍த (Apple Developer Accountඞਢ)
    

    View Slide

53. View Slide

54. View Slide

55. How to use RealityComposer
    Creating 3D Content with Reality Composer
    https://developer.apple.com/documentation/realitykit/
    creating_3d_content_with_reality_composer
    

    View Slide

56. and more on ARKit3...
    

    View Slide

57. Back and Front Camera
    

    View Slide

58. Implementing with AR
    

    View Slide

59. AR ࣮૷࣌ͷࠔ೉
    ޫྔ
    ޫྔෆ଍
    ԰֎ͰͷAR࣮ߦ
    ։ൃ࣌ͷ؀ڥͱϢʔβʔ࣮ߦ࣌ͷ 3Dঢ়گ͕ҟͳΔ
    

    View Slide

60. AR Coaching View
    ARKitͰ༻ҙ͞ΕͨɺϢʔβʔʹঢ়گΛද͢ϔϧϓϏϡʔ
    Ξχϝʔγϣϯ෇͖
    

    View Slide

61. Record and Replay
    σόοά࣌ͷMovieΛXcodeͰऩूͰ͖ΔΑ͏ʹɻ
    

    View Slide

62. Record and Replay
    Check : ARKit Replay data
    

    View Slide

63. Check Thermal State
    AR͸୺຤΁ͷॲཧෛ୲͕ߴ͍
    Թ౓มԽΛແࢹ͍ͯ͠ΔͱɺTemperature AlertʹͳΔ
    ڪΕ͕͋Δ
    Xcode11 ͷ Device Condition ը໘ͰσόοάՄೳ
    

    View Slide

64. Check Thermal State
    ProcessInfo.themalStateDidChangeNotification Λ؂ࢹ
    ProcessInfo.ThermalState ͷ஋ʹରԠ
    

    View Slide

65. View Slide

66. Recap
    ARKit3͸ࠓ·ͰͰ࠷େͷΞοϓσʔτɻ͍Ζ͍ΖͰ͖ΔͷͰɺ͍ͬͯ͡
    ͍Δ͚ͩͰָ͍͠ɻ
    iOS13ͷϩʔϯνͰҰ൪ Apple ͕ push͍ͯ͠Δͷ͸ AR !
    ळʹARମݧΛఏڙͰ͖ΔΑ͏ʹͳ͍ͬͯΕ͹ɺApple Store Top Feature !
    ARViewͰ࣮૷͠ͳͯ͘΋USDZϑΝΠϧΛஔ͍ͯ؆୯ʹ࣮૷͢Δख΋͋
    Γ·͢ɻ
    

    View Slide

67. Reference
    ARKit
    https://developer.apple.com/documentation/arkit
    AR Quick Look
    https://developer.apple.com/augmented-reality/quick-look/
    

    View Slide

68. Reference
    Introducing ARKit 3
    https://developer.apple.com/videos/play/wwdc2019/604
    Building Collaborative AR Experiences
    https://developer.apple.com/videos/play/wwdc2019/610
    Building AR Experiences with Reality Composer
    https://developer.apple.com/videos/play/wwdc2019/609
    Introducing RealityKit and Reality Composer
    https://developer.apple.com/videos/play/wwdc2019/603
    

    View Slide

69. Reference
    Working with USD
    https://developer.apple.com/videos/play/wwdc2019/602
    Building Apps with RealityKit
    https://developer.apple.com/videos/play/wwdc2019/605
    Advances in AR Quick Look
    https://developer.apple.com/videos/play/wwdc2019/612/
    Bringing People into AR
    https://developer.apple.com/videos/play/wwdc2019/607
    

    View Slide

70. Reference
    Creating Great Apps Using Core ML and ARKit
    https://developer.apple.com/videos/play/wwdc2019/228
    Designing for Adverse Network and Temperature Conditions
    https://developer.apple.com/videos/play/wwdc2019/422/
    

    View Slide