ARKit3

 ARKit3

61a68b2172503ecdf7a7f7757df56071?s=128

Tachibana Kaoru

June 12, 2019
Tweet

Transcript

  1. ARKit3 @TachibanaKaoru 2019/6/11 WWDC Catch-up

  2. About Me @TachibanaKaoru Freelance iOS Engineer झຯɿϘϧμϦϯάɺBotWͰཾͷྡྷङ ࠷ۙͷςʔϚɿཱྀΛ͠ͳ͕Β࢓ࣄΛ͢Δ όϦౡɺαϯτϦʔχౡɺόϯίΫɺόϧη ϩφɺΞΠεϥϯυ

  3. iOSDC 2019 2019/9/5 - 2019/9/7 At Waseda University Nishi-waseda campus

    CFP ืूதʂʢ6/17·Ͱʣ
  4. ARKit

  5. ARKit iOS11͔Β౥ࡌ͞Ε͍ͯΔɺARઐ༻ϑϨʔϜϫʔΫ Device with A9 chip~ Unity ARKit Plugin /

    Unity AR Foundation
  6. ARKit Tracking Scene Understanding Rendering

  7. AR Rendering SceneKit SpriteKit Metal RealityKit ARKit3

  8. ARKit 1.0 2017/06 ۭؒݕ஌ɺฏ໘ʢਫฏ໘ʣݕ஌ɺإද৘ݕ஌ 1.5 2018/01 ฏ໘ʢਨ௚໘ݕ஌ʣɺը૾ݕ஌ 2.0 2018/06 ը૾ݕ஌Ϟʔυ௥ՃɺARঢ়ଶอଘରԠɺ

    Object SacnରԠɺࢹઢɾઉݕ஌ 3.0 2019/06 BodyScanɺPeople Occupation
  9. What's new in ARKit3 People Occlusion Body Detection and more...

  10. People Occlusion

  11. ͜͏͍͏എܠʹAR ObjectΛஔ͘ͱ……

  12. ARKit2.0Ͱ͸͜͏ͳΓ·͢

  13. ARKit3.0Ͱ͸͜͏ͳΓ·ͨ͠ɻ

  14. ARKit2 ARKit3

  15. None
  16. People Occlusion ը໘಺ʹෳ਺ਓͷਓ͕͍ͯ΋ରԠՄೳ ਓͷશମ૾Ͱ͸ͳ͘ɺख΍଍ͷҰ෦͕͏͍ͭͬͯΔ͚ͩͰ΋ Occulution Մೳɻ

  17. People Occlusion set frameSemantics to ARWorldTrackingConfiguration

  18. Frame semantics personSegmentation depthΛߟྀ͠ͳ͍ ਓ͸ৗʹObjectͷલ໘

  19. Frame semantics personSegmentationWithDepth depthΛߟྀ͢Δ ਓ͸depthʹԠͯ͡લ໘ɾഎ໘ͱͳΔ

  20. AR FrameSemantics Configuration frameSemantics ARWorldTracking personSegmentation people always front ARWorldTracking

    personSegmentationWithDepth people with depth
  21. SwiftStrike People Occlusion Λར༻ͨ͠ ARήʔϜ WWDCձ৔Ͱৗʹ։࠵͞Εͯ ͓Γɺ੝Γ্͕͍ͬͯ·͠ ͨɻ

  22. None
  23. SwiftStrikeTabletop SwiftStrikeΛখ͍͞ςʔϒϧͰ༡΂Δαϯϓϧίʔυ https://developer.apple.com/documentation/realitykit/ creating_a_game_with_reality_composer

  24. Body Detection

  25. Body Detection ARۭؒͷਓମΛೝࣝ͠ɺ࠲ඪΛऔಘ 3DόʔνϟϧϞσϧΛਓମͷҐஔʹ Ԡͯ͡഑ஔ 2D Body Detection / 3D

    Body Detection
  26. 2D Body Detection

  27. 3D Body Detection

  28. 3D Skeleton 93ͷjointʢؔઅʣ jointʢؔઅʣʹ͸ҐஔΛج४ͱͨ͠਌ࢠؔ܎͕͋Δ

  29. 3D Skeleton index 0 : root index 1 : hip_joint

    index 2 : left_upLeg_joint index 3 : left_leg_joint ...... index 91 : left_eyeLowerLid_joint index 92 : left_eyeBall_joint
  30. 3D Skeleton index 0 : root index 1 : hip_joint

    index 2 : left_upLeg_joint index 3 : left_leg_joint ...... index 91 : left_eyeLowerLid_joint index 92 : left_eyeBall_joint skeleton3D.definition.parentIndices[index]
  31. 3D Skeleton index 0 : root index 1 : hip_joint

    index 2 : left_upLeg_joint index 3 : left_leg_joint ...... index 91 : left_eyeLowerLid_joint index 92 : left_eyeBall_joint > parent : root > parent : hip_joint > parent : hip_upLeg_joint hips_joint root left_upLeg_joint left_leg_joint left_foot_joint parent
  32. 3D Skeleton ਖ਼͍͠Ґஔ৘ใ͕͋Δjointͱɺͳ͍joint͕͋Δɻ ࢦɺ໨ඓɺͭ·ઌͳͲ͸ख଍ͷҐஔʹԊ͏Α͏ ʹ഑ஔ͞Ε͓ͯΓɺඞͣ͠΋ਖ਼͍͠ҐஔͰ͸ͳ ͍ɻ

  33. 3D Skeleton Local TransformͱModel Transform let jointLocalTransforms = skeleton3D.jointLocalTransforms let

    jointModelTransforms = skeleton3D.jointModelTransforms
  34. localTransform

  35. modelTransform

  36. Load my own 3D model ࣗ෼Ͱ࡞ͬͨ3DϞσϧΛskeltonʹ ͋ΘͤͯϦΞϧλΠϜදࣔ͢Δ͜ͱ ΋Մೳɻ ϞσϧϑΥʔϚοτͳͲ͸͓ͬͯެ ։͞ΕΔΑ͏ɻ

  37. AR Configuration Mode Camera iOS AROrientationTracking Back Camera iOS 11.0+

    ARWorldTracking Back Camera iOS 11.0+ ARFaceTracking Front Camera iOS 11.0+ ARImageTracking Back Camera iOS 12.0+ ARObjectScanning Back Camera iOS 12.0+ ARBodyTracking Back Camera iOS 13.0+ ARPositionTracking Back Camera iOS 13.0+
  38. AR FrameSemantics Configuration frameSemantics ARBodyTracking - 3D Body / 2D

    Body ARWorldTracking, ARImageTracking, AROrientationTracking bodyDetection 2D Body
  39. RealityKit

  40. RealityKit ARʹಛԽͨ͠3DඳըϥΠϒϥϦ iOS13 ~ RealityKitͷΈͰαϙʔτ͍ͯ͠Δػೳ΋ɻ Collavotive rendering

  41. Collaborative AR

  42. Collaborative AR ARKit2.0Ͱαϙʔτ͞ΕͨWorldMapͷڞ༗ػೳ͕όʔδϣϯΞοϓ RealityKitͷ৔߹ʹ͸ɺࣗಈతʹϨϯμϦϯά·Ͱͯ͘͠ΕΔɻ ૬खͷҐஔ΋ೝࣝͰ͖Δ

  43. None
  44. None
  45. AR Quick Look

  46. USDZ USD : Universal Scene Description 3D Model Format from

    Pixar
  47. AR Quick Look SafariɺϝϞɺϝʔϧͳͲͰUSDZϑΝΠϧΛ։͘ ͱɺࣗಈతʹΧϝϥө૾Λ༻͍ͨARͱͯ͠දࣔ͞ ΕΔ ΞϓϦΛ࡞ΒͣʹARମݧΛఏڙͰ͖Δ iOS12 ~

  48. ࣮ྫ NHK ౦ژిྗ ෱ౡୈҰݪࢠྗൃిॴAR https://www.nhk.or.jp/vr/AR/1fAR/

  49. ࣮ྫ NHK ڪཽ௒AR ςΟϥϊα΢ϧεɺτϦέϥτϓεɺεϐϊα΢ϧε https://www.nhk.or.jp/special/dino/ar.html

  50. What's new in iOS13 AR Quick Look λοϓͳͲͷΠϕϯτʹରԠ ΞχϝʔγϣϯʹରԠ ϢʔβʔΠϯλϥΫγϣϯΛؚΉ؆୯ͳήʔϜ͕USDZͷΈͰ࡞ΕΔΑ͏

    ʹɻ People OcclusionʹରԠ
  51. RealityComposer

  52. RealityComposer new 3D modeling tool for iPhone, iPad and Mac

    iOS version͸Test FlightͰ഑෍த (Apple Developer Accountඞਢ)
  53. None
  54. None
  55. How to use RealityComposer Creating 3D Content with Reality Composer

    https://developer.apple.com/documentation/realitykit/ creating_3d_content_with_reality_composer
  56. and more on ARKit3...

  57. Back and Front Camera

  58. Implementing with AR

  59. AR ࣮૷࣌ͷࠔ೉ ޫྔ ޫྔෆ଍ ԰֎ͰͷAR࣮ߦ ։ൃ࣌ͷ؀ڥͱϢʔβʔ࣮ߦ࣌ͷ 3Dঢ়گ͕ҟͳΔ

  60. AR Coaching View ARKitͰ༻ҙ͞ΕͨɺϢʔβʔʹঢ়گΛද͢ϔϧϓϏϡʔ Ξχϝʔγϣϯ෇͖

  61. Record and Replay σόοά࣌ͷMovieΛXcodeͰऩूͰ͖ΔΑ͏ʹɻ

  62. Record and Replay Check : ARKit Replay data

  63. Check Thermal State AR͸୺຤΁ͷॲཧෛ୲͕ߴ͍ Թ౓มԽΛແࢹ͍ͯ͠ΔͱɺTemperature AlertʹͳΔ ڪΕ͕͋Δ Xcode11 ͷ Device

    Condition ը໘ͰσόοάՄೳ
  64. Check Thermal State ProcessInfo.themalStateDidChangeNotification Λ؂ࢹ ProcessInfo.ThermalState ͷ஋ʹରԠ

  65. None
  66. Recap ARKit3͸ࠓ·ͰͰ࠷େͷΞοϓσʔτɻ͍Ζ͍ΖͰ͖ΔͷͰɺ͍ͬͯ͡ ͍Δ͚ͩͰָ͍͠ɻ iOS13ͷϩʔϯνͰҰ൪ Apple ͕ push͍ͯ͠Δͷ͸ AR ! ळʹARମݧΛఏڙͰ͖ΔΑ͏ʹͳ͍ͬͯΕ͹ɺApple

    Store Top Feature ! ARViewͰ࣮૷͠ͳͯ͘΋USDZϑΝΠϧΛஔ͍ͯ؆୯ʹ࣮૷͢Δख΋͋ Γ·͢ɻ
  67. Reference ARKit https://developer.apple.com/documentation/arkit AR Quick Look https://developer.apple.com/augmented-reality/quick-look/

  68. Reference Introducing ARKit 3 https://developer.apple.com/videos/play/wwdc2019/604 Building Collaborative AR Experiences https://developer.apple.com/videos/play/wwdc2019/610

    Building AR Experiences with Reality Composer https://developer.apple.com/videos/play/wwdc2019/609 Introducing RealityKit and Reality Composer https://developer.apple.com/videos/play/wwdc2019/603
  69. Reference Working with USD https://developer.apple.com/videos/play/wwdc2019/602 Building Apps with RealityKit https://developer.apple.com/videos/play/wwdc2019/605

    Advances in AR Quick Look https://developer.apple.com/videos/play/wwdc2019/612/ Bringing People into AR https://developer.apple.com/videos/play/wwdc2019/607
  70. Reference Creating Great Apps Using Core ML and ARKit https://developer.apple.com/videos/play/wwdc2019/228

    Designing for Adverse Network and Temperature Conditions https://developer.apple.com/videos/play/wwdc2019/422/