google/mediapipe で始めるARアプリ開発/iOSDC2020

3e66b3ccfa617b78c59d9dec865aa29f?s=47 noppefoxwolf
September 21, 2020

google/mediapipe で始めるARアプリ開発/iOSDC2020

3e66b3ccfa617b78c59d9dec865aa29f?s=128

noppefoxwolf

September 21, 2020
Tweet

Transcript

  1. google/mediapipe Ͱ࢝ ΊΔARΞϓϦ։ൃ ϨΪϡϥʔτʔΫʢ20෼ʣ noppe 1

  2. ର৅ • ARΞϓϦʹڵຯ͕͋Δ • τϥοΩϯάʹڵຯ͕͋Δ • ػցֶश෼͔Βͳͯ͘΋໰୊͋Γ·ͤΜɻ 2

  3. noppe • גࣜձࣾσΟʔɾΤψɾΤʔ • ιʔγϟϧϥΠϒΞϓϦ Pococha • ݸਓ։ൃऀ • vear

    ReplayKitͰεϚϗ͔Β഑৴Ͱ͖Δ VTuberΞϓϦ • iOSDC18,19 • ͖ͭͶ͕޷͖ 3
  4. 4

  5. 5

  6. noppe • גࣜձࣾσΟʔɾΤψɾΤʔ • ιʔγϟϧϥΠϒΞϓϦ Pococha • ݸਓ։ൃऀ • vear

    ReplayKitͰεϚϗ͔Β഑৴Ͱ͖Δ VTuberΞϓϦ • iOSDC18,19 • ͖ͭͶ͕޷͖ 6
  7. mediapipe ͱ͸ʁ 7

  8. mediapipe github.com/google/mediapipe • ॲཧ(Culculator)Λܨ͛ͯύΠϓϥΠϯʢGraphʣΛߏங͢Δ πʔϧ • ओʹը૾ॲཧͷύΠϓϥΠϯߏஙʹ࢖ΘΕΔ • MLॲཧΛࠞͥࠐΉࣄ͕ग़དྷΔ 8

  9. 9

  10. 10

  11. mediapipe Culculator͸ɺ࣮૷ࡁΈͷ෺΋࢖͑Δ • LuminanceCalculator RGBΛड͚औΓɺً౓ը૾Λग़ྗ͢Δ • SobelEdgesCalculator SobelϑΟϧλΛ͔͚ͨը૾Λग़ྗ͢Δ ...etc 11

  12. 12

  13. ࣮૷ࡁΈͷGraph ͜ΕΒͷ࣮૷͸͙͢ʹ࢖͑Δios ࠓ೔͸͜ΕΛ࢖͏ྲྀΕΛղઆ ios Ұ෦iOSͰ͸࣮૷͕ͳ͍΋ͷ΋͋Δ 13

  14. ΞδΣϯμ • άϥϑͷϏϧυ • ARKitͱͷ࿈ܞ • Ͱ͖ΔࣄɾͰ͖ͳ͍ࣄ 14

  15. άϥϑͷϏϧυ 15

  16. github.com/noppefoxwolf/HandTracker ࠓճ࢖͏ιʔείʔυ 16

  17. Graph • ߏஙͨ͠Graph͸όΠφϦͱͯ͠Ϗϧυ ͢Δ͜ͱ͕Ͱ͖Δ • όΠφϦΛObjC͔Β࢖͏࣮૷͕͋Δͷ ͰɺͦΕΒΛ࢖ͬͯϑϨʔϜϫʔΫΛ࡞ Δࣄ͕Ͱ͖Δɻ • ࠓ೔͸σϞͰ঺հͨ͠ϋϯυτϥοΧʔ

    ΛϑϨʔϜϫʔΫͱͯ͠Ϗϧυ 17
  18. 18

  19. άϥϑͷϏϧυ ϏϧυπʔϧΛ࢖ͬͯϏϧυΛߦ͏ • bazelbuild/bazel (ϕΠθϧ) • Kubernetes΍TensorFlowͰ࠾༻ Track E -

    Ryo Aoyama BazelΛར༻ͨ͠Micro Modular Architecture 19
  20. άϥϑͷϏϧυ bazelͷΠϯετʔϧ͸εΫϦϓτམͱ͖࣮ͯͯ͠ߦ͢Δ͚ͩ curl -LO "https://.../bazel-3.2.0-installer-darwin-x86_64.sh" chmod +x "bazel-3.2.0-installer-darwin-x86_64.sh" ./bazel-3.2.0-installer-darwin-x86_64.sh 20

  21. άϥϑͷϏϧυ XcodeͰ΋Ϗϧυ͕Ͱ͖Δtulsi - bazelbuild/tulsi tulsi BuildPhaseScriptͰbazelΛୟ͚ͩ͘ͳͷͰɺඞਢͰ͸ͳ͍ 21

  22. άϥϑͷϏϧυ tulsi͸ϏϧυεΫϦϓτୟ͚ͩ͘ͰΠϯετʔϧͰ͖Δʢཁbazelʣ git clone git@github.com:bazelbuild/tulsi.git ./build_and_run.sh 22

  23. ͦͷ΄͔ඞཁͳ΋ͷ • Xcode • ࣌ؒ • ωοτϫʔΫ 23

  24. ϏϧυͷྲྀΕ 1. ϑϨʔϜϫʔΫͷίʔυΛॻ͘ 2. BUILDϑΝΠϧʹ੒Ռ෺ͷ৘ใ΍ґଘੑͳͲΛهड़͢Δ 3. bazelͰbuild͢Δ 4. framework͕ग़དྷ্͕Δ 24

  25. ϑϨʔϜϫʔΫͷίʔυ Λॻ͘ 25

  26. ϑϨʔϜϫʔΫͷίʔυΛॻ͘ 1. binarypbΛಡΈࠐΜͰMPPGraphΛ࡞Δ 2. MPPGraphΛ։࢝͢Δ 3. MPPGraphʹը૾ΛૹΔ 4. delegateͰ݁ՌΛड͚औΔ 26

  27. 27

  28. binarypbΛಡΈࠐΜͰMPPGraphΛ࡞ Δ + (instanceType) init { NSURL* url = [NSURL

    ...@"hand_landmarks.binarypb"]; NSData* data = [NSData dataWithContentsOfURL:url options:0 error:nil]; mediapipe::CalculatorGraphConfig config; config.ParseFromArray(data.bytes, data.length); self.graph = [[MPPGraph alloc] initWithGraphConfig:config]; } 28
  29. MPPGraphΛ։࢝͢Δ Graph಺Ͱར༻͢Δ֤छίϯϙʔωϯτͷॳظԽ͕ߦΘΕΔ - (void)start { [graph startWithError: nil]; } 29

  30. MPPGraphʹը૾ΛૹΔ - (void)sendPixelBuffer:(CVPixelBufferRef)pixelBuffer { [self.mediapipeGraph sendPixelBuffer:pixelBuffer intoStream:@"input_video" packetType:MPPPacketTypePixelBuffer]; } 30

  31. delegateͰ݁ՌΛड͚औΔ - (void)mediapipeGraph:(MPPGraph*)graph didOutputPacket:(const ::mediapipe::Packet&)packet fromStream:(const std::string&)streamName { const auto&

    timestamp = packet.Timestamp().Value(); const auto& landmarks = packet.Get<::mediapipe::NormalizedLandmarkList>(); NSArray *landmarkObjects = [self toObject: landmarks]; [delegate didOutputLandmarks: landmarkObjects]; } 31
  32. BUILDϑΝΠϧॻ͘ 32

  33. BUILDϑΝΠϧͱ͸ bazelͰϏϧυ͢ΔϑΝΠϧ΍ґଘؔ܎Λهड़ͨ͠ϑΝΠϧ 33

  34. objc_library( name = "HandTrackerLibrary", hdrs = ["HandTracker.h"], srcs = ["HandTracker.mm"],

    data = [ "hand_tracking:hand_tracking_mobile_gpu_binary_graph", ... ], deps = [ "//mediapipe/objc:mediapipe_framework_ios", "//mediapipe/objc:mediapipe_input_sources_ios", "//mediapipe/graphs/hand_tracking:mobile_calculators", "//mediapipe/framework/formats:landmark_cc_proto", ], ), ) 34
  35. 35

  36. bazelͰϏϧυ͢Δ $ bazel build mediapipe/iosdc:HandTracker 36

  37. framework͕ग़དྷ্͕Δ githubʹஔ͍͓͖ͯ·ͨ͠ https://github.com/noppefoxwolf/HandTracker 37

  38. ARKitͱͷ࿈ܞ 38

  39. ARKitͱͷ࿈ܞ ARKitͷ࢓্༷ग़དྷͳ͍ࣄΛmediapipeͰิ͏͜ͱͰɺදݱྗΛղ ์Ͱ͖Δ • AR্ۭؒͷΦϒδΣΫτΛखͰૢ࡞͢Δ • ݕग़ͨ͠ΦϒδΣΫτͷԣʹΩϟϥΫλʔΛ഑ஔ͢Δ • إͷࠎ֨Λิਖ਼͠ͳ͕Β൅৭Λม͑Δhair hair

    Hair Segmentation͸ݱঢ়iOSඇରԠͰ͕͢ 39
  40. ARKitͱͷ࿈ܞΞϓϦ։ൃ ͜Μͳ΋ͷΛ࡞Γ·͢ 40

  41. 41

  42. ARKit࿈ܞͷྲྀΕ 1. ARFrame͔ΒcaptureImageΛऔΓग़͢ 2. PixelFormatΛม׵ 3. Tracker΁ૹΔ 4. DelegateͰ݁ՌΛड͚औΔ 5.

    RealityKitͷϘλϯΛ্ԼɾΧ΢ϯτΛ දࣔ 42
  43. ARFrame͔ΒcaptureImageΛऔΓग़͢ • captureImage = Χϝϥ͔Βͷө૾ arSession.delegate = self ... //

    ARSessionDelegate func session(_ session: ARSession, didUpdate frame: ARFrame) { let captureImage: CVPixelBuffer = frame.capturedImage } 43
  44. PixelFormatΛม׵ ը૾σʔλ͕ͲͷΑ͏ʹ֨ೲ͞Ε͍ͯΔ͔Λද͍ͯ͠Δ • ARKitͰ͸YCbCrܗࣜ • mediapipe͸BGRAͷΈΛड͚औΕΔ →ม׵ͷඞཁ͕͋Δ 44

  45. kCVPixelFormatType_32BGRA 1ϐΫηϧ32bitͰදݱ͢ΔϑΥʔϚοτ B:FF G:00 R:00 A:FF → ੺ 45

  46. kCVPixelFormatType_420YpCbCr8BiPl anarFullRange • 2ຕͷY(8bit)+CbCr(16bit)ͷ૊Έ߹ΘͤͰ1ϑϨʔϜΛදݱ͢Δ ϑΥʔϚοτ • ܰྔͳͷͰϏσΦσʔλͰ࢖ΘΕΔ͜ͱ͕ଟ͍ 46

  47. YCbCr͔ΒBGRA΁ͷม׵ • Accelerate • vImage • Metal • MSL GPUΛ࢖͏෼Metalͷํ͕ߴ଎

    47
  48. YCbCr͔ΒBGRA΁ͷม׵ ࣍ͷܭࢉࣜͰม׵͕Ͱ͖Δ R = Y + 1.402 × Cr G

    = Y - 0.344136 × Cb - 0.714136 × Cr B = Y + 1.772 × Cb A = 1 48
  49. BlueDress YCbCr͔ΒBGRAʹߴ଎ίϯόʔτ͢ΔϥΠϒϥϦ https://github.com/noppefoxwolf/BlueDress 49

  50. Tracker΁ૹΔ let captureImage = frame.capturedImage let bgraCaptureImage = converter.convertToBGRA(captureImage) handTracker.send(bgraCaptureImage)

    50
  51. DelegateͰ݁ՌΛड͚औΔ ؔઅͷҐஔ͸X,Y,ZͰऔΕ·͢…͕ɺੈք࠲ඪܥͰ͸ແ͘ X,Y: Screen࠲ඪܥ Z: खट͔ΒͷԞߦ͖ ͔͠औΕ·ͤΜɻ࢒೦ʂ 51

  52. 52

  53. RealityKitͷϘλϯΛ্ԼɾΧ΢ϯτΛ දࣔ ࠓճ͸ਓࠩ͠ࢦͷԆ௕ઢ্ʹEntity͕͋Ε͹ɺisPressΛtrueʹ͢Δ func handTracker(_ handTracker: HandTracker!, didOutputLandmarks landmarks: [Landmark]!)

    { let indexFinderPosition = landmarks[8] let screenLocation: CGPoint = ...(indexFinderPosition) self.isPress = arView .entities(at: screenLocation) .contains(where: { $0.id == self.buttonEntity.id }) } 53
  54. RealityKitͷϘλϯΛ্ԼɾΧ΢ϯτΛ දࣔ private func press() { self.scene.notifications.press.post() count += 1

    changeText("\(count)") } 54
  55. 55

  56. 56

  57. Ͱ͖ΔࣄͰ͖ͳ͍ࣄ 57

  58. ؾʹͳΔͱ͜Ζ • HandTracker͸Χϝϥ͔Βͷڑ཭͕औΕΔΘ͚Ͱ͸ͳ͍ࣄʹ஫ҙ →IrisͰ͸Ԟߦ͖͕औΕΔ • ֤ؔઅͷճస΋औΕͳ͍ →ࣗ෼Ͱܭࢉͯࣗ͠વʹݟͤΔ޻෉͕ඞཁ • ϑϨʔϜϫʔΫαΠζ͕·͊·͊Ͱ͔͍ →100MB͘Β͍

    58
  59. ͍͍ͱ͜Ζ • ࣮૷ࡁΈͷάϥϑ͕ඇৗʹ࣭͕ߴ͍ • OSS • ϚϧνϓϥοτϑΥʔϜ 59

  60. ΍ͬͯΈ͍ͨ͜ͱ • HairSegmentationͳͲΛiOSʹରԠ͍ͯ͠ͳ͍ͷ͸Կނͳͷ͔ௐ ΂͍ͨ • NSObjectΛܦ༝͠ͳ͚Ε͹ߴ଎Խ͕๬ΊΔͷͰ͸ • ϥϯυϚʔΫ͚ͩΛग़ྗ͢ΔάϥϑΛ࡞ͬͯΈ͍ͨ 60

  61. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝ ·ͨ͠ 61