Tweet
Tweet

Slide 1

Slide 1 text

ARKit3 @TachibanaKaoru 2019/6/11 WWDC Catch-up

Slide 2

Slide 2 text

About Me @TachibanaKaoru Freelance iOS Engineer झຯɿϘϧμϦϯάɺBotWͰཾͷྡྷङ ࠷ۙͷςʔϚɿཱྀΛ͠ͳ͕Β࢓ࣄΛ͢Δ όϦౡɺαϯτϦʔχౡɺόϯίΫɺόϧη ϩφɺΞΠεϥϯυ

Slide 3

Slide 3 text

iOSDC 2019 2019/9/5 - 2019/9/7 At Waseda University Nishi-waseda campus CFP ืूதʂʢ6/17·Ͱʣ

Slide 4

Slide 4 text

ARKit

Slide 5

Slide 5 text

ARKit iOS11͔Β౥ࡌ͞Ε͍ͯΔɺARઐ༻ϑϨʔϜϫʔΫ Device with A9 chip~ Unity ARKit Plugin / Unity AR Foundation

Slide 6

Slide 6 text

ARKit Tracking Scene Understanding Rendering

Slide 7

Slide 7 text

AR Rendering SceneKit SpriteKit Metal RealityKit ARKit3

Slide 8

Slide 8 text

ARKit 1.0 2017/06 ۭؒݕ஌ɺฏ໘ʢਫฏ໘ʣݕ஌ɺإද৘ݕ஌ 1.5 2018/01 ฏ໘ʢਨ௚໘ݕ஌ʣɺը૾ݕ஌ 2.0 2018/06 ը૾ݕ஌Ϟʔυ௥ՃɺARঢ়ଶอଘରԠɺ Object SacnରԠɺࢹઢɾઉݕ஌ 3.0 2019/06 BodyScanɺPeople Occupation

Slide 9

Slide 9 text

What's new in ARKit3 People Occlusion Body Detection and more...

Slide 10

Slide 10 text

People Occlusion

Slide 11

Slide 11 text

͜͏͍͏എܠʹAR ObjectΛஔ͘ͱ……

Slide 12

Slide 12 text

ARKit2.0Ͱ͸͜͏ͳΓ·͢

Slide 13

Slide 13 text

ARKit3.0Ͱ͸͜͏ͳΓ·ͨ͠ɻ

Slide 14

Slide 14 text

ARKit2 ARKit3

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

People Occlusion ը໘಺ʹෳ਺ਓͷਓ͕͍ͯ΋ରԠՄೳ ਓͷશମ૾Ͱ͸ͳ͘ɺख΍଍ͷҰ෦͕͏͍ͭͬͯΔ͚ͩͰ΋ Occulution Մೳɻ

Slide 17

Slide 17 text

People Occlusion set frameSemantics to ARWorldTrackingConfiguration

Slide 18

Slide 18 text

Frame semantics personSegmentation depthΛߟྀ͠ͳ͍ ਓ͸ৗʹObjectͷલ໘

Slide 19

Slide 19 text

Frame semantics personSegmentationWithDepth depthΛߟྀ͢Δ ਓ͸depthʹԠͯ͡લ໘ɾഎ໘ͱͳΔ

Slide 20

Slide 20 text

AR FrameSemantics Configuration frameSemantics ARWorldTracking personSegmentation people always front ARWorldTracking personSegmentationWithDepth people with depth

Slide 21

Slide 21 text

SwiftStrike People Occlusion Λར༻ͨ͠ ARήʔϜ WWDCձ৔Ͱৗʹ։࠵͞Εͯ ͓Γɺ੝Γ্͕͍ͬͯ·͠ ͨɻ

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

SwiftStrikeTabletop SwiftStrikeΛখ͍͞ςʔϒϧͰ༡΂Δαϯϓϧίʔυ https://developer.apple.com/documentation/realitykit/ creating_a_game_with_reality_composer

Slide 24

Slide 24 text

Body Detection

Slide 25

Slide 25 text

Body Detection ARۭؒͷਓମΛೝࣝ͠ɺ࠲ඪΛऔಘ 3DόʔνϟϧϞσϧΛਓମͷҐஔʹ Ԡͯ͡഑ஔ 2D Body Detection / 3D Body Detection

Slide 26

Slide 26 text

2D Body Detection

Slide 27

Slide 27 text

3D Body Detection

Slide 28

Slide 28 text

3D Skeleton 93ͷjointʢؔઅʣ jointʢؔઅʣʹ͸ҐஔΛج४ͱͨ͠਌ࢠؔ܎͕͋Δ

Slide 29

Slide 29 text

3D Skeleton index 0 : root index 1 : hip_joint index 2 : left_upLeg_joint index 3 : left_leg_joint ...... index 91 : left_eyeLowerLid_joint index 92 : left_eyeBall_joint

Slide 30

Slide 30 text

3D Skeleton index 0 : root index 1 : hip_joint index 2 : left_upLeg_joint index 3 : left_leg_joint ...... index 91 : left_eyeLowerLid_joint index 92 : left_eyeBall_joint skeleton3D.definition.parentIndices[index]

Slide 31

Slide 31 text

3D Skeleton index 0 : root index 1 : hip_joint index 2 : left_upLeg_joint index 3 : left_leg_joint ...... index 91 : left_eyeLowerLid_joint index 92 : left_eyeBall_joint > parent : root > parent : hip_joint > parent : hip_upLeg_joint hips_joint root left_upLeg_joint left_leg_joint left_foot_joint parent

Slide 32

Slide 32 text

3D Skeleton ਖ਼͍͠Ґஔ৘ใ͕͋Δjointͱɺͳ͍joint͕͋Δɻ ࢦɺ໨ඓɺͭ·ઌͳͲ͸ख଍ͷҐஔʹԊ͏Α͏ ʹ഑ஔ͞Ε͓ͯΓɺඞͣ͠΋ਖ਼͍͠ҐஔͰ͸ͳ ͍ɻ

Slide 33

Slide 33 text

3D Skeleton Local TransformͱModel Transform let jointLocalTransforms = skeleton3D.jointLocalTransforms let jointModelTransforms = skeleton3D.jointModelTransforms

Slide 34

Slide 34 text

localTransform

Slide 35

Slide 35 text

modelTransform

Slide 36

Slide 36 text

Load my own 3D model ࣗ෼Ͱ࡞ͬͨ3DϞσϧΛskeltonʹ ͋ΘͤͯϦΞϧλΠϜදࣔ͢Δ͜ͱ ΋Մೳɻ ϞσϧϑΥʔϚοτͳͲ͸͓ͬͯެ ։͞ΕΔΑ͏ɻ

Slide 37

Slide 37 text

AR Configuration Mode Camera iOS AROrientationTracking Back Camera iOS 11.0+ ARWorldTracking Back Camera iOS 11.0+ ARFaceTracking Front Camera iOS 11.0+ ARImageTracking Back Camera iOS 12.0+ ARObjectScanning Back Camera iOS 12.0+ ARBodyTracking Back Camera iOS 13.0+ ARPositionTracking Back Camera iOS 13.0+

Slide 38

Slide 38 text

AR FrameSemantics Configuration frameSemantics ARBodyTracking - 3D Body / 2D Body ARWorldTracking, ARImageTracking, AROrientationTracking bodyDetection 2D Body

Slide 39

Slide 39 text

RealityKit

Slide 40

Slide 40 text

RealityKit ARʹಛԽͨ͠3DඳըϥΠϒϥϦ iOS13 ~ RealityKitͷΈͰαϙʔτ͍ͯ͠Δػೳ΋ɻ Collavotive rendering

Slide 41

Slide 41 text

Collaborative AR

Slide 42

Slide 42 text

Collaborative AR ARKit2.0Ͱαϙʔτ͞ΕͨWorldMapͷڞ༗ػೳ͕όʔδϣϯΞοϓ RealityKitͷ৔߹ʹ͸ɺࣗಈతʹϨϯμϦϯά·Ͱͯ͘͠ΕΔɻ ૬खͷҐஔ΋ೝࣝͰ͖Δ

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

No content

Slide 45

Slide 45 text

AR Quick Look

Slide 46

Slide 46 text

USDZ USD : Universal Scene Description 3D Model Format from Pixar

Slide 47

Slide 47 text

AR Quick Look SafariɺϝϞɺϝʔϧͳͲͰUSDZϑΝΠϧΛ։͘ ͱɺࣗಈతʹΧϝϥө૾Λ༻͍ͨARͱͯ͠දࣔ͞ ΕΔ ΞϓϦΛ࡞ΒͣʹARମݧΛఏڙͰ͖Δ iOS12 ~

Slide 48

Slide 48 text

࣮ྫ NHK ౦ژిྗ ෱ౡୈҰݪࢠྗൃిॴAR https://www.nhk.or.jp/vr/AR/1fAR/

Slide 49

Slide 49 text

࣮ྫ NHK ڪཽ௒AR ςΟϥϊα΢ϧεɺτϦέϥτϓεɺεϐϊα΢ϧε https://www.nhk.or.jp/special/dino/ar.html

Slide 50

Slide 50 text

What's new in iOS13 AR Quick Look λοϓͳͲͷΠϕϯτʹରԠ ΞχϝʔγϣϯʹରԠ ϢʔβʔΠϯλϥΫγϣϯΛؚΉ؆୯ͳήʔϜ͕USDZͷΈͰ࡞ΕΔΑ͏ ʹɻ People OcclusionʹରԠ

Slide 51

Slide 51 text

RealityComposer

Slide 52

Slide 52 text

RealityComposer new 3D modeling tool for iPhone, iPad and Mac iOS version͸Test FlightͰ഑෍த (Apple Developer Accountඞਢ)

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

How to use RealityComposer Creating 3D Content with Reality Composer https://developer.apple.com/documentation/realitykit/ creating_3d_content_with_reality_composer

Slide 56

Slide 56 text

and more on ARKit3...

Slide 57

Slide 57 text

Back and Front Camera

Slide 58

Slide 58 text

Implementing with AR

Slide 59

Slide 59 text

AR ࣮૷࣌ͷࠔ೉ ޫྔ ޫྔෆ଍ ԰֎ͰͷAR࣮ߦ ։ൃ࣌ͷ؀ڥͱϢʔβʔ࣮ߦ࣌ͷ 3Dঢ়گ͕ҟͳΔ

Slide 60

Slide 60 text

AR Coaching View ARKitͰ༻ҙ͞ΕͨɺϢʔβʔʹঢ়گΛද͢ϔϧϓϏϡʔ Ξχϝʔγϣϯ෇͖

Slide 61

Slide 61 text

Record and Replay σόοά࣌ͷMovieΛXcodeͰऩूͰ͖ΔΑ͏ʹɻ

Slide 62

Slide 62 text

Record and Replay Check : ARKit Replay data

Slide 63

Slide 63 text

Check Thermal State AR͸୺຤΁ͷॲཧෛ୲͕ߴ͍ Թ౓มԽΛແࢹ͍ͯ͠ΔͱɺTemperature AlertʹͳΔ ڪΕ͕͋Δ Xcode11 ͷ Device Condition ը໘ͰσόοάՄೳ

Slide 64

Slide 64 text

Check Thermal State ProcessInfo.themalStateDidChangeNotification Λ؂ࢹ ProcessInfo.ThermalState ͷ஋ʹରԠ

Slide 65

Slide 65 text

No content

Slide 66

Slide 66 text

Recap ARKit3͸ࠓ·ͰͰ࠷େͷΞοϓσʔτɻ͍Ζ͍ΖͰ͖ΔͷͰɺ͍ͬͯ͡ ͍Δ͚ͩͰָ͍͠ɻ iOS13ͷϩʔϯνͰҰ൪ Apple ͕ push͍ͯ͠Δͷ͸ AR ! ळʹARମݧΛఏڙͰ͖ΔΑ͏ʹͳ͍ͬͯΕ͹ɺApple Store Top Feature ! ARViewͰ࣮૷͠ͳͯ͘΋USDZϑΝΠϧΛஔ͍ͯ؆୯ʹ࣮૷͢Δख΋͋ Γ·͢ɻ

Slide 67

Slide 67 text

Reference ARKit https://developer.apple.com/documentation/arkit AR Quick Look https://developer.apple.com/augmented-reality/quick-look/

Slide 68

Slide 68 text

Reference Introducing ARKit 3 https://developer.apple.com/videos/play/wwdc2019/604 Building Collaborative AR Experiences https://developer.apple.com/videos/play/wwdc2019/610 Building AR Experiences with Reality Composer https://developer.apple.com/videos/play/wwdc2019/609 Introducing RealityKit and Reality Composer https://developer.apple.com/videos/play/wwdc2019/603

Slide 69

Slide 69 text

Reference Working with USD https://developer.apple.com/videos/play/wwdc2019/602 Building Apps with RealityKit https://developer.apple.com/videos/play/wwdc2019/605 Advances in AR Quick Look https://developer.apple.com/videos/play/wwdc2019/612/ Bringing People into AR https://developer.apple.com/videos/play/wwdc2019/607

Slide 70

Slide 70 text

Reference Creating Great Apps Using Core ML and ARKit https://developer.apple.com/videos/play/wwdc2019/228 Designing for Adverse Network and Temperature Conditions https://developer.apple.com/videos/play/wwdc2019/422/