ARKit3

Transcript

1. ARKit3 @TachibanaKaoru 2019/6/11 WWDC Catch-up
2. About Me @TachibanaKaoru Freelance iOS Engineer झຯɿϘϧμϦϯάɺBotWͰཾͷྡྷङ ࠷ۙͷςʔϚɿཱྀΛ͠ͳ͕Β࢓ࣄΛ͢Δ όϦౡɺαϯτϦʔχౡɺόϯίΫɺόϧη ϩφɺΞΠεϥϯυ
3. iOSDC 2019 2019/9/5 - 2019/9/7 At Waseda University Nishi-waseda campus CFP ืूதʂʢ6/17·Ͱʣ
4. ARKit
5. ARKit iOS11͔Β౥ࡌ͞Ε͍ͯΔɺARઐ༻ϑϨʔϜϫʔΫ Device with A9 chip~ Unity ARKit Plugin / Unity AR Foundation
6. ARKit Tracking Scene Understanding Rendering
7. AR Rendering SceneKit SpriteKit Metal RealityKit ARKit3
8. ARKit 1.0 2017/06 ۭؒݕ஌ɺฏ໘ʢਫฏ໘ʣݕ஌ɺإද৘ݕ஌ 1.5 2018/01 ฏ໘ʢਨ௚໘ݕ஌ʣɺը૾ݕ஌ 2.0 2018/06 ը૾ݕ஌Ϟʔυ௥ՃɺARঢ়ଶอଘରԠɺ Object SacnରԠɺࢹઢɾઉݕ஌ 3.0 2019/06 BodyScanɺPeople Occupation
9. What's new in ARKit3 People Occlusion Body Detection and more...
10. People Occlusion
11. ͜͏͍͏എܠʹAR ObjectΛஔ͘ͱ……
12. ARKit2.0Ͱ͸͜͏ͳΓ·͢
13. ARKit3.0Ͱ͸͜͏ͳΓ·ͨ͠ɻ
14. ARKit2 ARKit3
16. People Occlusion ը໘಺ʹෳ਺ਓͷਓ͕͍ͯ΋ରԠՄೳ ਓͷશମ૾Ͱ͸ͳ͘ɺख΍଍ͷҰ෦͕͏͍ͭͬͯΔ͚ͩͰ΋ Occulution Մೳɻ
17. People Occlusion set frameSemantics to ARWorldTrackingConfiguration
18. Frame semantics personSegmentation depthΛߟྀ͠ͳ͍ ਓ͸ৗʹObjectͷલ໘
19. Frame semantics personSegmentationWithDepth depthΛߟྀ͢Δ ਓ͸depthʹԠͯ͡લ໘ɾഎ໘ͱͳΔ
20. AR FrameSemantics Configuration frameSemantics ARWorldTracking personSegmentation people always front ARWorldTracking personSegmentationWithDepth people with depth
21. SwiftStrike People Occlusion Λར༻ͨ͠ ARήʔϜ WWDCձ৔Ͱৗʹ։࠵͞Εͯ ͓Γɺ੝Γ্͕͍ͬͯ·͠ ͨɻ
23. SwiftStrikeTabletop SwiftStrikeΛখ͍͞ςʔϒϧͰ༡΂Δαϯϓϧίʔυ https://developer.apple.com/documentation/realitykit/ creating_a_game_with_reality_composer
24. Body Detection
25. Body Detection ARۭؒͷਓମΛೝࣝ͠ɺ࠲ඪΛऔಘ 3DόʔνϟϧϞσϧΛਓମͷҐஔʹ Ԡͯ͡഑ஔ 2D Body Detection / 3D Body Detection
26. 2D Body Detection
27. 3D Body Detection
28. 3D Skeleton 93ͷjointʢؔઅʣ jointʢؔઅʣʹ͸ҐஔΛج४ͱͨ͠਌ࢠؔ܎͕͋Δ
29. 3D Skeleton index 0 : root index 1 : hip_joint index 2 : left_upLeg_joint index 3 : left_leg_joint ...... index 91 : left_eyeLowerLid_joint index 92 : left_eyeBall_joint
30. 3D Skeleton index 0 : root index 1 : hip_joint index 2 : left_upLeg_joint index 3 : left_leg_joint ...... index 91 : left_eyeLowerLid_joint index 92 : left_eyeBall_joint skeleton3D.definition.parentIndices[index]
31. 3D Skeleton index 0 : root index 1 : hip_joint index 2 : left_upLeg_joint index 3 : left_leg_joint ...... index 91 : left_eyeLowerLid_joint index 92 : left_eyeBall_joint > parent : root > parent : hip_joint > parent : hip_upLeg_joint hips_joint root left_upLeg_joint left_leg_joint left_foot_joint parent
32. 3D Skeleton ਖ਼͍͠Ґஔ৘ใ͕͋Δjointͱɺͳ͍joint͕͋Δɻ ࢦɺ໨ඓɺͭ·ઌͳͲ͸ख଍ͷҐஔʹԊ͏Α͏ ʹ഑ஔ͞Ε͓ͯΓɺඞͣ͠΋ਖ਼͍͠ҐஔͰ͸ͳ ͍ɻ
33. 3D Skeleton Local TransformͱModel Transform let jointLocalTransforms = skeleton3D.jointLocalTransforms let jointModelTransforms = skeleton3D.jointModelTransforms
34. localTransform
35. modelTransform
36. Load my own 3D model ࣗ෼Ͱ࡞ͬͨ3DϞσϧΛskeltonʹ ͋ΘͤͯϦΞϧλΠϜදࣔ͢Δ͜ͱ ΋Մೳɻ ϞσϧϑΥʔϚοτͳͲ͸͓ͬͯެ ։͞ΕΔΑ͏ɻ
37. AR Configuration Mode Camera iOS AROrientationTracking Back Camera iOS 11.0+ ARWorldTracking Back Camera iOS 11.0+ ARFaceTracking Front Camera iOS 11.0+ ARImageTracking Back Camera iOS 12.0+ ARObjectScanning Back Camera iOS 12.0+ ARBodyTracking Back Camera iOS 13.0+ ARPositionTracking Back Camera iOS 13.0+
38. AR FrameSemantics Configuration frameSemantics ARBodyTracking - 3D Body / 2D Body ARWorldTracking, ARImageTracking, AROrientationTracking bodyDetection 2D Body
39. RealityKit
40. RealityKit ARʹಛԽͨ͠3DඳըϥΠϒϥϦ iOS13 ~ RealityKitͷΈͰαϙʔτ͍ͯ͠Δػೳ΋ɻ Collavotive rendering
41. Collaborative AR
42. Collaborative AR ARKit2.0Ͱαϙʔτ͞ΕͨWorldMapͷڞ༗ػೳ͕όʔδϣϯΞοϓ RealityKitͷ৔߹ʹ͸ɺࣗಈతʹϨϯμϦϯά·Ͱͯ͘͠ΕΔɻ ૬खͷҐஔ΋ೝࣝͰ͖Δ
45. AR Quick Look
46. USDZ USD : Universal Scene Description 3D Model Format from Pixar
47. AR Quick Look SafariɺϝϞɺϝʔϧͳͲͰUSDZϑΝΠϧΛ։͘ ͱɺࣗಈతʹΧϝϥө૾Λ༻͍ͨARͱͯ͠දࣔ͞ ΕΔ ΞϓϦΛ࡞ΒͣʹARମݧΛఏڙͰ͖Δ iOS12 ~
48. ࣮ྫ NHK ౦ژిྗ ෱ౡୈҰݪࢠྗൃిॴAR https://www.nhk.or.jp/vr/AR/1fAR/
49. ࣮ྫ NHK ڪཽ௒AR ςΟϥϊα΢ϧεɺτϦέϥτϓεɺεϐϊα΢ϧε https://www.nhk.or.jp/special/dino/ar.html
50. What's new in iOS13 AR Quick Look λοϓͳͲͷΠϕϯτʹରԠ ΞχϝʔγϣϯʹରԠ ϢʔβʔΠϯλϥΫγϣϯΛؚΉ؆୯ͳήʔϜ͕USDZͷΈͰ࡞ΕΔΑ͏ ʹɻ People OcclusionʹରԠ
51. RealityComposer
52. RealityComposer new 3D modeling tool for iPhone, iPad and Mac iOS version͸Test FlightͰ഑෍த (Apple Developer Accountඞਢ)
55. How to use RealityComposer Creating 3D Content with Reality Composer https://developer.apple.com/documentation/realitykit/ creating_3d_content_with_reality_composer
56. and more on ARKit3...
57. Back and Front Camera
58. Implementing with AR
59. AR ࣮૷࣌ͷࠔ೉ ޫྔ ޫྔෆ଍ ԰֎ͰͷAR࣮ߦ ։ൃ࣌ͷ؀ڥͱϢʔβʔ࣮ߦ࣌ͷ 3Dঢ়گ͕ҟͳΔ
60. AR Coaching View ARKitͰ༻ҙ͞ΕͨɺϢʔβʔʹঢ়گΛද͢ϔϧϓϏϡʔ Ξχϝʔγϣϯ෇͖
61. Record and Replay σόοά࣌ͷMovieΛXcodeͰऩूͰ͖ΔΑ͏ʹɻ
62. Record and Replay Check : ARKit Replay data
63. Check Thermal State AR͸୺຤΁ͷॲཧෛ୲͕ߴ͍ Թ౓มԽΛແࢹ͍ͯ͠ΔͱɺTemperature AlertʹͳΔ ڪΕ͕͋Δ Xcode11 ͷ Device Condition ը໘ͰσόοάՄೳ
64. Check Thermal State ProcessInfo.themalStateDidChangeNotification Λ؂ࢹ ProcessInfo.ThermalState ͷ஋ʹରԠ
66. Recap ARKit3͸ࠓ·ͰͰ࠷େͷΞοϓσʔτɻ͍Ζ͍ΖͰ͖ΔͷͰɺ͍ͬͯ͡ ͍Δ͚ͩͰָ͍͠ɻ iOS13ͷϩʔϯνͰҰ൪ Apple ͕ push͍ͯ͠Δͷ͸ AR ! ळʹARମݧΛఏڙͰ͖ΔΑ͏ʹͳ͍ͬͯΕ͹ɺApple Store Top Feature ! ARViewͰ࣮૷͠ͳͯ͘΋USDZϑΝΠϧΛஔ͍ͯ؆୯ʹ࣮૷͢Δख΋͋ Γ·͢ɻ
67. Reference ARKit https://developer.apple.com/documentation/arkit AR Quick Look https://developer.apple.com/augmented-reality/quick-look/
68. Reference Introducing ARKit 3 https://developer.apple.com/videos/play/wwdc2019/604 Building Collaborative AR Experiences https://developer.apple.com/videos/play/wwdc2019/610 Building AR Experiences with Reality Composer https://developer.apple.com/videos/play/wwdc2019/609 Introducing RealityKit and Reality Composer https://developer.apple.com/videos/play/wwdc2019/603
69. Reference Working with USD https://developer.apple.com/videos/play/wwdc2019/602 Building Apps with RealityKit https://developer.apple.com/videos/play/wwdc2019/605 Advances in AR Quick Look https://developer.apple.com/videos/play/wwdc2019/612/ Bringing People into AR https://developer.apple.com/videos/play/wwdc2019/607
70. Reference Creating Great Apps Using Core ML and ARKit https://developer.apple.com/videos/play/wwdc2019/228 Designing for Adverse Network and Temperature Conditions https://developer.apple.com/videos/play/wwdc2019/422/