google/mediapipe で始めるARアプリ開発/iOSDC2020

Transcript

1. google/mediapipe Ͱ࢝ ΊΔARΞϓϦ։ൃ ϨΪϡϥʔτʔΫʢ20෼ʣ noppe 1

2. ର৅ • ARΞϓϦʹڵຯ͕͋Δ • τϥοΩϯάʹڵຯ͕͋Δ • ػցֶश෼͔Βͳͯ͘΋໰୊͋Γ·ͤΜɻ 2

3. noppe • גࣜձࣾσΟʔɾΤψɾΤʔ • ιʔγϟϧϥΠϒΞϓϦ Pococha • ݸਓ։ൃऀ • vear

   ReplayKitͰεϚϗ͔Β഑৴Ͱ͖Δ VTuberΞϓϦ • iOSDC18,19 • ͖ͭͶ͕޷͖ 3

4. 4

5. 5

6. noppe • גࣜձࣾσΟʔɾΤψɾΤʔ • ιʔγϟϧϥΠϒΞϓϦ Pococha • ݸਓ։ൃऀ • vear

   ReplayKitͰεϚϗ͔Β഑৴Ͱ͖Δ VTuberΞϓϦ • iOSDC18,19 • ͖ͭͶ͕޷͖ 6

7. mediapipe ͱ͸ʁ 7

8. mediapipe github.com/google/mediapipe • ॲཧ(Culculator)Λܨ͛ͯύΠϓϥΠϯʢGraphʣΛߏங͢Δ πʔϧ • ओʹը૾ॲཧͷύΠϓϥΠϯߏஙʹ࢖ΘΕΔ • MLॲཧΛࠞͥࠐΉࣄ͕ग़དྷΔ 8

9. 9

10. 10

11. mediapipe Culculator͸ɺ࣮૷ࡁΈͷ෺΋࢖͑Δ • LuminanceCalculator RGBΛड͚औΓɺً౓ը૾Λग़ྗ͢Δ • SobelEdgesCalculator SobelϑΟϧλΛ͔͚ͨը૾Λग़ྗ͢Δ ...etc 11

12. 12

13. ࣮૷ࡁΈͷGraph ͜ΕΒͷ࣮૷͸͙͢ʹ࢖͑Δios ࠓ೔͸͜ΕΛ࢖͏ྲྀΕΛղઆ ios Ұ෦iOSͰ͸࣮૷͕ͳ͍΋ͷ΋͋Δ 13

14. ΞδΣϯμ • άϥϑͷϏϧυ • ARKitͱͷ࿈ܞ • Ͱ͖ΔࣄɾͰ͖ͳ͍ࣄ 14

15. άϥϑͷϏϧυ 15

16. github.com/noppefoxwolf/HandTracker ࠓճ࢖͏ιʔείʔυ 16

17. Graph • ߏஙͨ͠Graph͸όΠφϦͱͯ͠Ϗϧυ ͢Δ͜ͱ͕Ͱ͖Δ • όΠφϦΛObjC͔Β࢖͏࣮૷͕͋Δͷ ͰɺͦΕΒΛ࢖ͬͯϑϨʔϜϫʔΫΛ࡞ Δࣄ͕Ͱ͖Δɻ • ࠓ೔͸σϞͰ঺հͨ͠ϋϯυτϥοΧʔ

    ΛϑϨʔϜϫʔΫͱͯ͠Ϗϧυ 17

18. 18

19. άϥϑͷϏϧυ ϏϧυπʔϧΛ࢖ͬͯϏϧυΛߦ͏ • bazelbuild/bazel (ϕΠθϧ) • Kubernetes΍TensorFlowͰ࠾༻ Track E -

    Ryo Aoyama BazelΛར༻ͨ͠Micro Modular Architecture 19

20. άϥϑͷϏϧυ bazelͷΠϯετʔϧ͸εΫϦϓτམͱ͖࣮ͯͯ͠ߦ͢Δ͚ͩ curl -LO "https://.../bazel-3.2.0-installer-darwin-x86_64.sh" chmod +x "bazel-3.2.0-installer-darwin-x86_64.sh" ./bazel-3.2.0-installer-darwin-x86_64.sh 20

21. άϥϑͷϏϧυ XcodeͰ΋Ϗϧυ͕Ͱ͖Δtulsi - bazelbuild/tulsi tulsi BuildPhaseScriptͰbazelΛୟ͚ͩ͘ͳͷͰɺඞਢͰ͸ͳ͍ 21

22. άϥϑͷϏϧυ tulsi͸ϏϧυεΫϦϓτୟ͚ͩ͘ͰΠϯετʔϧͰ͖Δʢཁbazelʣ git clone git@github.com:bazelbuild/tulsi.git ./build_and_run.sh 22

23. ͦͷ΄͔ඞཁͳ΋ͷ • Xcode • ࣌ؒ • ωοτϫʔΫ 23

24. ϏϧυͷྲྀΕ 1. ϑϨʔϜϫʔΫͷίʔυΛॻ͘ 2. BUILDϑΝΠϧʹ੒Ռ෺ͷ৘ใ΍ґଘੑͳͲΛهड़͢Δ 3. bazelͰbuild͢Δ 4. framework͕ग़དྷ্͕Δ 24

25. ϑϨʔϜϫʔΫͷίʔυ Λॻ͘ 25

26. ϑϨʔϜϫʔΫͷίʔυΛॻ͘ 1. binarypbΛಡΈࠐΜͰMPPGraphΛ࡞Δ 2. MPPGraphΛ։࢝͢Δ 3. MPPGraphʹը૾ΛૹΔ 4. delegateͰ݁ՌΛड͚औΔ 26

27. 27

28. binarypbΛಡΈࠐΜͰMPPGraphΛ࡞ Δ + (instanceType) init { NSURL* url = [NSURL

    ...@"hand_landmarks.binarypb"]; NSData* data = [NSData dataWithContentsOfURL:url options:0 error:nil]; mediapipe::CalculatorGraphConfig config; config.ParseFromArray(data.bytes, data.length); self.graph = [[MPPGraph alloc] initWithGraphConfig:config]; } 28

29. MPPGraphΛ։࢝͢Δ Graph಺Ͱར༻͢Δ֤छίϯϙʔωϯτͷॳظԽ͕ߦΘΕΔ - (void)start { [graph startWithError: nil]; } 29

30. MPPGraphʹը૾ΛૹΔ - (void)sendPixelBuffer:(CVPixelBufferRef)pixelBuffer { [self.mediapipeGraph sendPixelBuffer:pixelBuffer intoStream:@"input_video" packetType:MPPPacketTypePixelBuffer]; } 30

31. delegateͰ݁ՌΛड͚औΔ - (void)mediapipeGraph:(MPPGraph*)graph didOutputPacket:(const ::mediapipe::Packet&)packet fromStream:(const std::string&)streamName { const auto&

    timestamp = packet.Timestamp().Value(); const auto& landmarks = packet.Get<::mediapipe::NormalizedLandmarkList>(); NSArray *landmarkObjects = [self toObject: landmarks]; [delegate didOutputLandmarks: landmarkObjects]; } 31

32. BUILDϑΝΠϧॻ͘ 32

33. BUILDϑΝΠϧͱ͸ bazelͰϏϧυ͢ΔϑΝΠϧ΍ґଘؔ܎Λهड़ͨ͠ϑΝΠϧ 33

34. objc_library( name = "HandTrackerLibrary", hdrs = ["HandTracker.h"], srcs = ["HandTracker.mm"],

    data = [ "hand_tracking:hand_tracking_mobile_gpu_binary_graph", ... ], deps = [ "//mediapipe/objc:mediapipe_framework_ios", "//mediapipe/objc:mediapipe_input_sources_ios", "//mediapipe/graphs/hand_tracking:mobile_calculators", "//mediapipe/framework/formats:landmark_cc_proto", ], ), ) 34

35. 35

36. bazelͰϏϧυ͢Δ $ bazel build mediapipe/iosdc:HandTracker 36

37. framework͕ग़དྷ্͕Δ githubʹஔ͍͓͖ͯ·ͨ͠ https://github.com/noppefoxwolf/HandTracker 37

38. ARKitͱͷ࿈ܞ 38

39. ARKitͱͷ࿈ܞ ARKitͷ࢓্༷ग़དྷͳ͍ࣄΛmediapipeͰิ͏͜ͱͰɺදݱྗΛղ ์Ͱ͖Δ • AR্ۭؒͷΦϒδΣΫτΛखͰૢ࡞͢Δ • ݕग़ͨ͠ΦϒδΣΫτͷԣʹΩϟϥΫλʔΛ഑ஔ͢Δ • إͷࠎ֨Λิਖ਼͠ͳ͕Β൅৭Λม͑Δhair hair

    Hair Segmentation͸ݱঢ়iOSඇରԠͰ͕͢ 39

40. ARKitͱͷ࿈ܞΞϓϦ։ൃ ͜Μͳ΋ͷΛ࡞Γ·͢ 40

41. 41

42. ARKit࿈ܞͷྲྀΕ 1. ARFrame͔ΒcaptureImageΛऔΓग़͢ 2. PixelFormatΛม׵ 3. Tracker΁ૹΔ 4. DelegateͰ݁ՌΛड͚औΔ 5.

    RealityKitͷϘλϯΛ্ԼɾΧ΢ϯτΛ දࣔ 42

43. ARFrame͔ΒcaptureImageΛऔΓग़͢ • captureImage = Χϝϥ͔Βͷө૾ arSession.delegate = self ... //

    ARSessionDelegate func session(_ session: ARSession, didUpdate frame: ARFrame) { let captureImage: CVPixelBuffer = frame.capturedImage } 43

44. PixelFormatΛม׵ ը૾σʔλ͕ͲͷΑ͏ʹ֨ೲ͞Ε͍ͯΔ͔Λද͍ͯ͠Δ • ARKitͰ͸YCbCrܗࣜ • mediapipe͸BGRAͷΈΛड͚औΕΔ →ม׵ͷඞཁ͕͋Δ 44

45. kCVPixelFormatType_32BGRA 1ϐΫηϧ32bitͰදݱ͢ΔϑΥʔϚοτ B:FF G:00 R:00 A:FF → ੺ 45

46. kCVPixelFormatType_420YpCbCr8BiPl anarFullRange • 2ຕͷY(8bit)+CbCr(16bit)ͷ૊Έ߹ΘͤͰ1ϑϨʔϜΛදݱ͢Δ ϑΥʔϚοτ • ܰྔͳͷͰϏσΦσʔλͰ࢖ΘΕΔ͜ͱ͕ଟ͍ 46

47. YCbCr͔ΒBGRA΁ͷม׵ • Accelerate • vImage • Metal • MSL GPUΛ࢖͏෼Metalͷํ͕ߴ଎

    47

48. YCbCr͔ΒBGRA΁ͷม׵ ࣍ͷܭࢉࣜͰม׵͕Ͱ͖Δ R = Y + 1.402 × Cr G

    = Y - 0.344136 × Cb - 0.714136 × Cr B = Y + 1.772 × Cb A = 1 48

49. BlueDress YCbCr͔ΒBGRAʹߴ଎ίϯόʔτ͢ΔϥΠϒϥϦ https://github.com/noppefoxwolf/BlueDress 49

50. Tracker΁ૹΔ let captureImage = frame.capturedImage let bgraCaptureImage = converter.convertToBGRA(captureImage) handTracker.send(bgraCaptureImage)

    50

51. DelegateͰ݁ՌΛड͚औΔ ؔઅͷҐஔ͸X,Y,ZͰऔΕ·͢…͕ɺੈք࠲ඪܥͰ͸ແ͘ X,Y: Screen࠲ඪܥ Z: खट͔ΒͷԞߦ͖ ͔͠औΕ·ͤΜɻ࢒೦ʂ 51

52. 52

53. RealityKitͷϘλϯΛ্ԼɾΧ΢ϯτΛ දࣔ ࠓճ͸ਓࠩ͠ࢦͷԆ௕ઢ্ʹEntity͕͋Ε͹ɺisPressΛtrueʹ͢Δ func handTracker(_ handTracker: HandTracker!, didOutputLandmarks landmarks: [Landmark]!)

    { let indexFinderPosition = landmarks[8] let screenLocation: CGPoint = ...(indexFinderPosition) self.isPress = arView .entities(at: screenLocation) .contains(where: { $0.id == self.buttonEntity.id }) } 53

54. RealityKitͷϘλϯΛ্ԼɾΧ΢ϯτΛ දࣔ private func press() { self.scene.notifications.press.post() count += 1

    changeText("\(count)") } 54

55. 55

56. 56

57. Ͱ͖ΔࣄͰ͖ͳ͍ࣄ 57

58. ؾʹͳΔͱ͜Ζ • HandTracker͸Χϝϥ͔Βͷڑ཭͕औΕΔΘ͚Ͱ͸ͳ͍ࣄʹ஫ҙ →IrisͰ͸Ԟߦ͖͕औΕΔ • ֤ؔઅͷճస΋औΕͳ͍ →ࣗ෼Ͱܭࢉͯࣗ͠વʹݟͤΔ޻෉͕ඞཁ • ϑϨʔϜϫʔΫαΠζ͕·͊·͊Ͱ͔͍ →100MB͘Β͍

    58

59. ͍͍ͱ͜Ζ • ࣮૷ࡁΈͷάϥϑ͕ඇৗʹ࣭͕ߴ͍ • OSS • ϚϧνϓϥοτϑΥʔϜ 59

60. ΍ͬͯΈ͍ͨ͜ͱ • HairSegmentationͳͲΛiOSʹରԠ͍ͯ͠ͳ͍ͷ͸Կނͳͷ͔ௐ ΂͍ͨ • NSObjectΛܦ༝͠ͳ͚Ε͹ߴ଎Խ͕๬ΊΔͷͰ͸ • ϥϯυϚʔΫ͚ͩΛग़ྗ͢ΔάϥϑΛ࡞ͬͯΈ͍ͨ 60

61. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝ ·ͨ͠ 61