google/mediapipe で始めるARアプリ開発/iOSDC2020

Transcript

1. google/mediapipe Ͱ࢝
   ΊΔARΞϓϦ։ൃ
   ϨΪϡϥʔτʔΫʢ20෼ʣ
   noppe
   1
   

   View Slide

2. ର৅
   • ARΞϓϦʹڵຯ͕͋Δ
   • τϥοΩϯάʹڵຯ͕͋Δ
   • ػցֶश෼͔Βͳͯ͘΋໰୊͋Γ·ͤΜɻ
   2
   

   View Slide

3. noppe
   • גࣜձࣾσΟʔɾΤψɾΤʔ
   • ιʔγϟϧϥΠϒΞϓϦ Pococha
   • ݸਓ։ൃऀ
   • vear
   ReplayKitͰεϚϗ͔Β഑৴Ͱ͖Δ
   VTuberΞϓϦ
   • iOSDC18,19
   • ͖ͭͶ͕޷͖
   3
   

   View Slide

4. View Slide

5. View Slide

6. noppe
   • גࣜձࣾσΟʔɾΤψɾΤʔ
   • ιʔγϟϧϥΠϒΞϓϦ Pococha
   • ݸਓ։ൃऀ
   • vear
   ReplayKitͰεϚϗ͔Β഑৴Ͱ͖Δ
   VTuberΞϓϦ
   • iOSDC18,19
   • ͖ͭͶ͕޷͖
   6
   

   View Slide

7. mediapipe ͱ͸ʁ
   7
   

   View Slide

8. mediapipe
   github.com/google/mediapipe
   • ॲཧ(Culculator)Λܨ͛ͯύΠϓϥΠϯʢGraphʣΛߏங͢Δ
   πʔϧ
   • ओʹը૾ॲཧͷύΠϓϥΠϯߏஙʹ࢖ΘΕΔ
   • MLॲཧΛࠞͥࠐΉࣄ͕ग़དྷΔ
   8
   

   View Slide

9. View Slide

10. View Slide

11. mediapipe
    Culculator͸ɺ࣮૷ࡁΈͷ෺΋࢖͑Δ
    • LuminanceCalculator
    RGBΛड͚औΓɺً౓ը૾Λग़ྗ͢Δ
    • SobelEdgesCalculator
    SobelϑΟϧλΛ͔͚ͨը૾Λग़ྗ͢Δ
    ...etc
    11
    

    View Slide

12. View Slide

13. ࣮૷ࡁΈͷGraph
    ͜ΕΒͷ࣮૷͸͙͢ʹ࢖͑Δios
    ࠓ೔͸͜ΕΛ࢖͏ྲྀΕΛղઆ
    ios Ұ෦iOSͰ͸࣮૷͕ͳ͍΋ͷ΋͋Δ
    13
    

    View Slide

14. ΞδΣϯμ
    • άϥϑͷϏϧυ
    • ARKitͱͷ࿈ܞ
    • Ͱ͖ΔࣄɾͰ͖ͳ͍ࣄ
    14
    

    View Slide

15. άϥϑͷϏϧυ
    15
    

    View Slide

16. github.com/noppefoxwolf/HandTracker
    ࠓճ࢖͏ιʔείʔυ
    16
    

    View Slide

17. Graph
    • ߏஙͨ͠Graph͸όΠφϦͱͯ͠Ϗϧυ
    ͢Δ͜ͱ͕Ͱ͖Δ
    • όΠφϦΛObjC͔Β࢖͏࣮૷͕͋Δͷ
    ͰɺͦΕΒΛ࢖ͬͯϑϨʔϜϫʔΫΛ࡞
    Δࣄ͕Ͱ͖Δɻ
    • ࠓ೔͸σϞͰ঺հͨ͠ϋϯυτϥοΧʔ
    ΛϑϨʔϜϫʔΫͱͯ͠Ϗϧυ
    17
    

    View Slide

18. View Slide

19. άϥϑͷϏϧυ
    ϏϧυπʔϧΛ࢖ͬͯϏϧυΛߦ͏
    • bazelbuild/bazel (ϕΠθϧ)
    • Kubernetes΍TensorFlowͰ࠾༻
    Track E - Ryo Aoyama
    BazelΛར༻ͨ͠Micro Modular Architecture
    19
    

    View Slide

20. άϥϑͷϏϧυ
    bazelͷΠϯετʔϧ͸εΫϦϓτམͱ͖࣮ͯͯ͠ߦ͢Δ͚ͩ
    curl -LO "https://.../bazel-3.2.0-installer-darwin-x86_64.sh"
    chmod +x "bazel-3.2.0-installer-darwin-x86_64.sh"
    ./bazel-3.2.0-installer-darwin-x86_64.sh
    20
    

    View Slide

21. άϥϑͷϏϧυ
    XcodeͰ΋Ϗϧυ͕Ͱ͖Δtulsi
    - bazelbuild/tulsi
    tulsi BuildPhaseScriptͰbazelΛୟ͚ͩ͘ͳͷͰɺඞਢͰ͸ͳ͍
    21
    

    View Slide

22. άϥϑͷϏϧυ
    tulsi͸ϏϧυεΫϦϓτୟ͚ͩ͘ͰΠϯετʔϧͰ͖Δʢཁbazelʣ
    git clone git@github.com:bazelbuild/tulsi.git
    ./build_and_run.sh
    22
    

    View Slide

23. ͦͷ΄͔ඞཁͳ΋ͷ
    • Xcode
    • ࣌ؒ
    • ωοτϫʔΫ
    23
    

    View Slide

24. ϏϧυͷྲྀΕ
    1. ϑϨʔϜϫʔΫͷίʔυΛॻ͘
    2. BUILDϑΝΠϧʹ੒Ռ෺ͷ৘ใ΍ґଘੑͳͲΛهड़͢Δ
    3. bazelͰbuild͢Δ
    4. framework͕ग़དྷ্͕Δ
    24
    

    View Slide

25. ϑϨʔϜϫʔΫͷίʔυ
    Λॻ͘
    25
    

    View Slide

26. ϑϨʔϜϫʔΫͷίʔυΛॻ͘
    1. binarypbΛಡΈࠐΜͰMPPGraphΛ࡞Δ
    2. MPPGraphΛ։࢝͢Δ
    3. MPPGraphʹը૾ΛૹΔ
    4. delegateͰ݁ՌΛड͚औΔ
    26
    

    View Slide

27. View Slide

28. binarypbΛಡΈࠐΜͰMPPGraphΛ࡞
    Δ
    + (instanceType) init {
    NSURL* url = [NSURL ...@"hand_landmarks.binarypb"];
    NSData* data = [NSData dataWithContentsOfURL:url options:0 error:nil];
    mediapipe::CalculatorGraphConfig config;
    config.ParseFromArray(data.bytes, data.length);
    self.graph = [[MPPGraph alloc] initWithGraphConfig:config];
    }
    28
    

    View Slide

29. MPPGraphΛ։࢝͢Δ
    Graph಺Ͱར༻͢Δ֤छίϯϙʔωϯτͷॳظԽ͕ߦΘΕΔ
    - (void)start {
    [graph startWithError: nil];
    }
    29
    

    View Slide

30. MPPGraphʹը૾ΛૹΔ
    - (void)sendPixelBuffer:(CVPixelBufferRef)pixelBuffer {
    [self.mediapipeGraph sendPixelBuffer:pixelBuffer
    intoStream:@"input_video"
    packetType:MPPPacketTypePixelBuffer];
    }
    30
    

    View Slide

31. delegateͰ݁ՌΛड͚औΔ
    - (void)mediapipeGraph:(MPPGraph*)graph
    didOutputPacket:(const ::mediapipe::Packet&)packet
    fromStream:(const std::string&)streamName
    {
    const auto& timestamp = packet.Timestamp().Value();
    const auto& landmarks = packet.Get<::mediapipe::NormalizedLandmarkList>();
    NSArray *landmarkObjects = [self toObject: landmarks];
    [delegate didOutputLandmarks: landmarkObjects];
    }
    31
    

    View Slide

32. BUILDϑΝΠϧॻ͘
    32
    

    View Slide

33. BUILDϑΝΠϧͱ͸
    bazelͰϏϧυ͢ΔϑΝΠϧ΍ґଘؔ܎Λهड़ͨ͠ϑΝΠϧ
    33
    

    View Slide

34. objc_library(
    name = "HandTrackerLibrary",
    hdrs = ["HandTracker.h"],
    srcs = ["HandTracker.mm"],
    data = [
    "hand_tracking:hand_tracking_mobile_gpu_binary_graph",
    ...
    ],
    deps = [
    "//mediapipe/objc:mediapipe_framework_ios",
    "//mediapipe/objc:mediapipe_input_sources_ios",
    "//mediapipe/graphs/hand_tracking:mobile_calculators",
    "//mediapipe/framework/formats:landmark_cc_proto",
    ],
    ),
    )
    34
    

    View Slide

35. View Slide

36. bazelͰϏϧυ͢Δ
    $ bazel build mediapipe/iosdc:HandTracker
    36
    

    View Slide

37. framework͕ग़དྷ্͕Δ
    githubʹஔ͍͓͖ͯ·ͨ͠
    https://github.com/noppefoxwolf/HandTracker
    37
    

    View Slide

38. ARKitͱͷ࿈ܞ
    38
    

    View Slide

39. ARKitͱͷ࿈ܞ
    ARKitͷ࢓্༷ग़དྷͳ͍ࣄΛmediapipeͰิ͏͜ͱͰɺදݱྗΛղ
    ์Ͱ͖Δ
    • AR্ۭؒͷΦϒδΣΫτΛखͰૢ࡞͢Δ
    • ݕग़ͨ͠ΦϒδΣΫτͷԣʹΩϟϥΫλʔΛ഑ஔ͢Δ
    • إͷࠎ֨Λิਖ਼͠ͳ͕Β൅৭Λม͑Δhair
    hair Hair Segmentation͸ݱঢ়iOSඇରԠͰ͕͢
    39
    

    View Slide

40. ARKitͱͷ࿈ܞΞϓϦ։ൃ
    ͜Μͳ΋ͷΛ࡞Γ·͢
    40
    

    View Slide

41. View Slide

42. ARKit࿈ܞͷྲྀΕ
    1. ARFrame͔ΒcaptureImageΛऔΓग़͢
    2. PixelFormatΛม׵
    3. Tracker΁ૹΔ
    4. DelegateͰ݁ՌΛड͚औΔ
    5. RealityKitͷϘλϯΛ্ԼɾΧ΢ϯτΛ
    දࣔ
    42
    

    View Slide

43. ARFrame͔ΒcaptureImageΛऔΓग़͢
    • captureImage = Χϝϥ͔Βͷө૾
    arSession.delegate = self
    ...
    // ARSessionDelegate
    func session(_ session: ARSession, didUpdate frame: ARFrame) {
    let captureImage: CVPixelBuffer = frame.capturedImage
    }
    43
    

    View Slide

44. PixelFormatΛม׵
    ը૾σʔλ͕ͲͷΑ͏ʹ֨ೲ͞Ε͍ͯΔ͔Λද͍ͯ͠Δ
    • ARKitͰ͸YCbCrܗࣜ
    • mediapipe͸BGRAͷΈΛड͚औΕΔ
    →ม׵ͷඞཁ͕͋Δ
    44
    

    View Slide

45. kCVPixelFormatType_32BGRA
    1ϐΫηϧ32bitͰදݱ͢ΔϑΥʔϚοτ
    B:FF G:00 R:00 A:FF → ੺
    45
    

    View Slide

46. kCVPixelFormatType_420YpCbCr8BiPl
    anarFullRange
    • 2ຕͷY(8bit)+CbCr(16bit)ͷ૊Έ߹ΘͤͰ1ϑϨʔϜΛදݱ͢Δ
    ϑΥʔϚοτ
    • ܰྔͳͷͰϏσΦσʔλͰ࢖ΘΕΔ͜ͱ͕ଟ͍
    46
    

    View Slide

47. YCbCr͔ΒBGRA΁ͷม׵
    • Accelerate
    • vImage
    • Metal
    • MSL
    GPUΛ࢖͏෼Metalͷํ͕ߴ଎
    47
    

    View Slide

48. YCbCr͔ΒBGRA΁ͷม׵
    ࣍ͷܭࢉࣜͰม׵͕Ͱ͖Δ
    R = Y + 1.402 × Cr
    G = Y - 0.344136 × Cb - 0.714136 × Cr
    B = Y + 1.772 × Cb
    A = 1
    48
    

    View Slide

49. BlueDress
    YCbCr͔ΒBGRAʹߴ଎ίϯόʔτ͢ΔϥΠϒϥϦ
    https://github.com/noppefoxwolf/BlueDress
    49
    

    View Slide

50. Tracker΁ૹΔ
    let captureImage = frame.capturedImage
    let bgraCaptureImage = converter.convertToBGRA(captureImage)
    handTracker.send(bgraCaptureImage)
    50
    

    View Slide

51. DelegateͰ݁ՌΛड͚औΔ
    ؔઅͷҐஔ͸X,Y,ZͰऔΕ·͢…͕ɺੈք࠲ඪܥͰ͸ແ͘
    X,Y: Screen࠲ඪܥ
    Z: खट͔ΒͷԞߦ͖
    ͔͠औΕ·ͤΜɻ࢒೦ʂ
    51
    

    View Slide

52. View Slide

53. RealityKitͷϘλϯΛ্ԼɾΧ΢ϯτΛ
    දࣔ
    ࠓճ͸ਓࠩ͠ࢦͷԆ௕ઢ্ʹEntity͕͋Ε͹ɺisPressΛtrueʹ͢Δ
    func handTracker(_ handTracker: HandTracker!, didOutputLandmarks landmarks: [Landmark]!) {
    let indexFinderPosition = landmarks[8]
    let screenLocation: CGPoint = ...(indexFinderPosition)
    self.isPress = arView
    .entities(at: screenLocation)
    .contains(where: { $0.id == self.buttonEntity.id })
    }
    53
    

    View Slide

54. RealityKitͷϘλϯΛ্ԼɾΧ΢ϯτΛ
    දࣔ
    private func press() {
    self.scene.notifications.press.post()
    count += 1
    changeText("\(count)")
    }
    54
    

    View Slide

55. View Slide

56. View Slide

57. Ͱ͖ΔࣄͰ͖ͳ͍ࣄ
    57
    

    View Slide

58. ؾʹͳΔͱ͜Ζ
    • HandTracker͸Χϝϥ͔Βͷڑ཭͕औΕΔΘ͚Ͱ͸ͳ͍ࣄʹ஫ҙ
    →IrisͰ͸Ԟߦ͖͕औΕΔ
    • ֤ؔઅͷճస΋औΕͳ͍
    →ࣗ෼Ͱܭࢉͯࣗ͠વʹݟͤΔ޻෉͕ඞཁ
    • ϑϨʔϜϫʔΫαΠζ͕·͊·͊Ͱ͔͍
    →100MB͘Β͍
    58
    

    View Slide

59. ͍͍ͱ͜Ζ
    • ࣮૷ࡁΈͷάϥϑ͕ඇৗʹ࣭͕ߴ͍
    • OSS
    • ϚϧνϓϥοτϑΥʔϜ
    59
    

    View Slide

60. ΍ͬͯΈ͍ͨ͜ͱ
    • HairSegmentationͳͲΛiOSʹରԠ͍ͯ͠ͳ͍ͷ͸Կނͳͷ͔ௐ
    ΂͍ͨ
    • NSObjectΛܦ༝͠ͳ͚Ε͹ߴ଎Խ͕๬ΊΔͷͰ͸
    • ϥϯυϚʔΫ͚ͩΛग़ྗ͢ΔάϥϑΛ࡞ͬͯΈ͍ͨ
    60
    

    View Slide

61. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝
    ·ͨ͠
    61
    

    View Slide