google/mediapipe で始めるARアプリ開発/iOSDC2020

Slide 1

Slide 1 text

google/mediapipe Ͱ࢝ ΊΔARΞϓϦ։ൃ ϨΪϡϥʔτʔΫʢ20෼ʣ noppe 1

Slide 2

Slide 2 text

ର৅ • ARΞϓϦʹڵຯ͕͋Δ • τϥοΩϯάʹڵຯ͕͋Δ • ػցֶश෼͔Βͳͯ͘΋໰୊͋Γ·ͤΜɻ 2

Slide 3

Slide 3 text

noppe • גࣜձࣾσΟʔɾΤψɾΤʔ • ιʔγϟϧϥΠϒΞϓϦ Pococha • ݸਓ։ൃऀ • vear ReplayKitͰεϚϗ͔Β഑৴Ͱ͖Δ VTuberΞϓϦ • iOSDC18,19 • ͖ͭͶ͕޷͖ 3

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Slide 6

Slide 6 text

noppe • גࣜձࣾσΟʔɾΤψɾΤʔ • ιʔγϟϧϥΠϒΞϓϦ Pococha • ݸਓ։ൃऀ • vear ReplayKitͰεϚϗ͔Β഑৴Ͱ͖Δ VTuberΞϓϦ • iOSDC18,19 • ͖ͭͶ͕޷͖ 6

Slide 7

Slide 7 text

mediapipe ͱ͸ʁ 7

Slide 8

Slide 8 text

mediapipe github.com/google/mediapipe • ॲཧ(Culculator)Λܨ͛ͯύΠϓϥΠϯʢGraphʣΛߏங͢Δ πʔϧ • ओʹը૾ॲཧͷύΠϓϥΠϯߏஙʹ࢖ΘΕΔ • MLॲཧΛࠞͥࠐΉࣄ͕ग़དྷΔ 8

Slide 9

Slide 9 text

Slide 10

Slide 10 text

Slide 11

Slide 11 text

mediapipe Culculator͸ɺ࣮૷ࡁΈͷ෺΋࢖͑Δ • LuminanceCalculator RGBΛड͚औΓɺً౓ը૾Λग़ྗ͢Δ • SobelEdgesCalculator SobelϑΟϧλΛ͔͚ͨը૾Λग़ྗ͢Δ ...etc 11

Slide 12

Slide 12 text

Slide 13

Slide 13 text

࣮૷ࡁΈͷGraph ͜ΕΒͷ࣮૷͸͙͢ʹ࢖͑Δios ࠓ೔͸͜ΕΛ࢖͏ྲྀΕΛղઆ ios Ұ෦iOSͰ͸࣮૷͕ͳ͍΋ͷ΋͋Δ 13

Slide 14

Slide 14 text

ΞδΣϯμ • άϥϑͷϏϧυ • ARKitͱͷ࿈ܞ • Ͱ͖ΔࣄɾͰ͖ͳ͍ࣄ 14

Slide 15

Slide 15 text

άϥϑͷϏϧυ 15

Slide 16

Slide 16 text

github.com/noppefoxwolf/HandTracker ࠓճ࢖͏ιʔείʔυ 16

Slide 17

Slide 17 text

Graph • ߏஙͨ͠Graph͸όΠφϦͱͯ͠Ϗϧυ ͢Δ͜ͱ͕Ͱ͖Δ • όΠφϦΛObjC͔Β࢖͏࣮૷͕͋Δͷ ͰɺͦΕΒΛ࢖ͬͯϑϨʔϜϫʔΫΛ࡞ Δࣄ͕Ͱ͖Δɻ • ࠓ೔͸σϞͰ঺հͨ͠ϋϯυτϥοΧʔ ΛϑϨʔϜϫʔΫͱͯ͠Ϗϧυ 17

Slide 18

Slide 18 text

Slide 19

Slide 19 text

άϥϑͷϏϧυ ϏϧυπʔϧΛ࢖ͬͯϏϧυΛߦ͏ • bazelbuild/bazel (ϕΠθϧ) • Kubernetes΍TensorFlowͰ࠾༻ Track E - Ryo Aoyama BazelΛར༻ͨ͠Micro Modular Architecture 19

Slide 20

Slide 20 text

άϥϑͷϏϧυ bazelͷΠϯετʔϧ͸εΫϦϓτམͱ͖࣮ͯͯ͠ߦ͢Δ͚ͩ curl -LO "https://.../bazel-3.2.0-installer-darwin-x86_64.sh" chmod +x "bazel-3.2.0-installer-darwin-x86_64.sh" ./bazel-3.2.0-installer-darwin-x86_64.sh 20

Slide 21

Slide 21 text

άϥϑͷϏϧυ XcodeͰ΋Ϗϧυ͕Ͱ͖Δtulsi - bazelbuild/tulsi tulsi BuildPhaseScriptͰbazelΛୟ͚ͩ͘ͳͷͰɺඞਢͰ͸ͳ͍ 21

Slide 22

Slide 22 text

άϥϑͷϏϧυ tulsi͸ϏϧυεΫϦϓτୟ͚ͩ͘ͰΠϯετʔϧͰ͖Δʢཁbazelʣ git clone [email protected]:bazelbuild/tulsi.git ./build_and_run.sh 22

Slide 23

Slide 23 text

ͦͷ΄͔ඞཁͳ΋ͷ • Xcode • ࣌ؒ • ωοτϫʔΫ 23

Slide 24

Slide 24 text

ϏϧυͷྲྀΕ 1. ϑϨʔϜϫʔΫͷίʔυΛॻ͘ 2. BUILDϑΝΠϧʹ੒Ռ෺ͷ৘ใ΍ґଘੑͳͲΛهड़͢Δ 3. bazelͰbuild͢Δ 4. framework͕ग़དྷ্͕Δ 24

Slide 25

Slide 25 text

ϑϨʔϜϫʔΫͷίʔυ Λॻ͘ 25

Slide 26

Slide 26 text

ϑϨʔϜϫʔΫͷίʔυΛॻ͘ 1. binarypbΛಡΈࠐΜͰMPPGraphΛ࡞Δ 2. MPPGraphΛ։࢝͢Δ 3. MPPGraphʹը૾ΛૹΔ 4. delegateͰ݁ՌΛड͚औΔ 26

Slide 27

Slide 27 text

Slide 28

Slide 28 text

binarypbΛಡΈࠐΜͰMPPGraphΛ࡞ Δ + (instanceType) init { NSURL* url = [NSURL ...@"hand_landmarks.binarypb"]; NSData* data = [NSData dataWithContentsOfURL:url options:0 error:nil]; mediapipe::CalculatorGraphConfig config; config.ParseFromArray(data.bytes, data.length); self.graph = [[MPPGraph alloc] initWithGraphConfig:config]; } 28

Slide 29

Slide 29 text

MPPGraphΛ։࢝͢Δ Graph಺Ͱར༻͢Δ֤छίϯϙʔωϯτͷॳظԽ͕ߦΘΕΔ - (void)start { [graph startWithError: nil]; } 29

Slide 30

Slide 30 text

MPPGraphʹը૾ΛૹΔ - (void)sendPixelBuffer:(CVPixelBufferRef)pixelBuffer { [self.mediapipeGraph sendPixelBuffer:pixelBuffer intoStream:@"input_video" packetType:MPPPacketTypePixelBuffer]; } 30

Slide 31

Slide 31 text

delegateͰ݁ՌΛड͚औΔ - (void)mediapipeGraph:(MPPGraph*)graph didOutputPacket:(const ::mediapipe::Packet&)packet fromStream:(const std::string&)streamName { const auto& timestamp = packet.Timestamp().Value(); const auto& landmarks = packet.Get<::mediapipe::NormalizedLandmarkList>(); NSArray *landmarkObjects = [self toObject: landmarks]; [delegate didOutputLandmarks: landmarkObjects]; } 31

Slide 32

Slide 32 text

BUILDϑΝΠϧॻ͘ 32

Slide 33

Slide 33 text

BUILDϑΝΠϧͱ͸ bazelͰϏϧυ͢ΔϑΝΠϧ΍ґଘؔ܎Λهड़ͨ͠ϑΝΠϧ 33

Slide 34

Slide 34 text

objc_library( name = "HandTrackerLibrary", hdrs = ["HandTracker.h"], srcs = ["HandTracker.mm"], data = [ "hand_tracking:hand_tracking_mobile_gpu_binary_graph", ... ], deps = [ "//mediapipe/objc:mediapipe_framework_ios", "//mediapipe/objc:mediapipe_input_sources_ios", "//mediapipe/graphs/hand_tracking:mobile_calculators", "//mediapipe/framework/formats:landmark_cc_proto", ], ), ) 34

Slide 35

Slide 35 text

Slide 36

Slide 36 text

bazelͰϏϧυ͢Δ $ bazel build mediapipe/iosdc:HandTracker 36

Slide 37

Slide 37 text

framework͕ग़དྷ্͕Δ githubʹஔ͍͓͖ͯ·ͨ͠ https://github.com/noppefoxwolf/HandTracker 37

Slide 38

Slide 38 text

ARKitͱͷ࿈ܞ 38

Slide 39

Slide 39 text

ARKitͱͷ࿈ܞ ARKitͷ࢓্༷ग़དྷͳ͍ࣄΛmediapipeͰิ͏͜ͱͰɺදݱྗΛղ ์Ͱ͖Δ • AR্ۭؒͷΦϒδΣΫτΛखͰૢ࡞͢Δ • ݕग़ͨ͠ΦϒδΣΫτͷԣʹΩϟϥΫλʔΛ഑ஔ͢Δ • إͷࠎ֨Λิਖ਼͠ͳ͕Β൅৭Λม͑Δhair hair Hair Segmentation͸ݱঢ়iOSඇରԠͰ͕͢ 39

Slide 40

Slide 40 text

ARKitͱͷ࿈ܞΞϓϦ։ൃ ͜Μͳ΋ͷΛ࡞Γ·͢ 40

Slide 41

Slide 41 text

Slide 42

Slide 42 text

ARKit࿈ܞͷྲྀΕ 1. ARFrame͔ΒcaptureImageΛऔΓग़͢ 2. PixelFormatΛม׵ 3. Tracker΁ૹΔ 4. DelegateͰ݁ՌΛड͚औΔ 5. RealityKitͷϘλϯΛ্ԼɾΧ΢ϯτΛ දࣔ 42

Slide 43

Slide 43 text

ARFrame͔ΒcaptureImageΛऔΓग़͢ • captureImage = Χϝϥ͔Βͷө૾ arSession.delegate = self ... // ARSessionDelegate func session(_ session: ARSession, didUpdate frame: ARFrame) { let captureImage: CVPixelBuffer = frame.capturedImage } 43

Slide 44

Slide 44 text

PixelFormatΛม׵ ը૾σʔλ͕ͲͷΑ͏ʹ֨ೲ͞Ε͍ͯΔ͔Λද͍ͯ͠Δ • ARKitͰ͸YCbCrܗࣜ • mediapipe͸BGRAͷΈΛड͚औΕΔ →ม׵ͷඞཁ͕͋Δ 44

Slide 45

Slide 45 text

kCVPixelFormatType_32BGRA 1ϐΫηϧ32bitͰදݱ͢ΔϑΥʔϚοτ B:FF G:00 R:00 A:FF → ੺ 45

Slide 46

Slide 46 text

kCVPixelFormatType_420YpCbCr8BiPl anarFullRange • 2ຕͷY(8bit)+CbCr(16bit)ͷ૊Έ߹ΘͤͰ1ϑϨʔϜΛදݱ͢Δ ϑΥʔϚοτ • ܰྔͳͷͰϏσΦσʔλͰ࢖ΘΕΔ͜ͱ͕ଟ͍ 46

Slide 47

Slide 47 text

YCbCr͔ΒBGRA΁ͷม׵ • Accelerate • vImage • Metal • MSL GPUΛ࢖͏෼Metalͷํ͕ߴ଎ 47

Slide 48

Slide 48 text

YCbCr͔ΒBGRA΁ͷม׵ ࣍ͷܭࢉࣜͰม׵͕Ͱ͖Δ R = Y + 1.402 × Cr G = Y - 0.344136 × Cb - 0.714136 × Cr B = Y + 1.772 × Cb A = 1 48

Slide 49

Slide 49 text

BlueDress YCbCr͔ΒBGRAʹߴ଎ίϯόʔτ͢ΔϥΠϒϥϦ https://github.com/noppefoxwolf/BlueDress 49

Slide 50

Slide 50 text

Tracker΁ૹΔ let captureImage = frame.capturedImage let bgraCaptureImage = converter.convertToBGRA(captureImage) handTracker.send(bgraCaptureImage) 50

Slide 51

Slide 51 text

DelegateͰ݁ՌΛड͚औΔ ؔઅͷҐஔ͸X,Y,ZͰऔΕ·͢…͕ɺੈք࠲ඪܥͰ͸ແ͘ X,Y: Screen࠲ඪܥ Z: खट͔ΒͷԞߦ͖ ͔͠औΕ·ͤΜɻ࢒೦ʂ 51

Slide 52

Slide 52 text

Slide 53

Slide 53 text

RealityKitͷϘλϯΛ্ԼɾΧ΢ϯτΛ දࣔ ࠓճ͸ਓࠩ͠ࢦͷԆ௕ઢ্ʹEntity͕͋Ε͹ɺisPressΛtrueʹ͢Δ func handTracker(_ handTracker: HandTracker!, didOutputLandmarks landmarks: [Landmark]!) { let indexFinderPosition = landmarks[8] let screenLocation: CGPoint = ...(indexFinderPosition) self.isPress = arView .entities(at: screenLocation) .contains(where: { $0.id == self.buttonEntity.id }) } 53

Slide 54

Slide 54 text

RealityKitͷϘλϯΛ্ԼɾΧ΢ϯτΛ දࣔ private func press() { self.scene.notifications.press.post() count += 1 changeText("\(count)") } 54

Slide 55

Slide 55 text

Slide 56

Slide 56 text

Slide 57

Slide 57 text

Ͱ͖ΔࣄͰ͖ͳ͍ࣄ 57

Slide 58

Slide 58 text

ؾʹͳΔͱ͜Ζ • HandTracker͸Χϝϥ͔Βͷڑ཭͕औΕΔΘ͚Ͱ͸ͳ͍ࣄʹ஫ҙ →IrisͰ͸Ԟߦ͖͕औΕΔ • ֤ؔઅͷճస΋औΕͳ͍ →ࣗ෼Ͱܭࢉͯࣗ͠વʹݟͤΔ޻෉͕ඞཁ • ϑϨʔϜϫʔΫαΠζ͕·͊·͊Ͱ͔͍ →100MB͘Β͍ 58

Slide 59

Slide 59 text

͍͍ͱ͜Ζ • ࣮૷ࡁΈͷάϥϑ͕ඇৗʹ࣭͕ߴ͍ • OSS • ϚϧνϓϥοτϑΥʔϜ 59

Slide 60

Slide 60 text

΍ͬͯΈ͍ͨ͜ͱ • HairSegmentationͳͲΛiOSʹରԠ͍ͯ͠ͳ͍ͷ͸Կނͳͷ͔ௐ ΂͍ͨ • NSObjectΛܦ༝͠ͳ͚Ε͹ߴ଎Խ͕๬ΊΔͷͰ͸ • ϥϯυϚʔΫ͚ͩΛग़ྗ͢ΔάϥϑΛ࡞ͬͯΈ͍ͨ 60

Slide 61

Slide 61 text

͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝ ·ͨ͠ 61