Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
google/mediapipe で始めるARアプリ開発/iOSDC2020
noppefoxwolf
September 21, 2020
Programming
1
710
google/mediapipe で始めるARアプリ開発/iOSDC2020
noppefoxwolf
September 21, 2020
Tweet
Share
More Decks by noppefoxwolf
See All by noppefoxwolf
ランタイムデバッグのススメ/iOSDC21
noppefoxwolf
1
2.8k
モバイルファーストなアプリを作るためにvearがしたこと/xRDCC
noppefoxwolf
0
24
ソーシャルライブサービスにおけるデジタル化粧の仕組みと実装/iOSDC19
noppefoxwolf
4
4.2k
Limited import clarification and its effect/tryswift2019
noppefoxwolf
2
910
立ち上げ時のライブ配信アプリに最適な開発環境・技術的ノウハウとは/PocoDevMeetup-1
noppefoxwolf
0
820
SceneKitでふんわりした影を描画する/potatotips58
noppefoxwolf
2
1.1k
What is CVBuffer/ROPPONGI.swift
noppefoxwolf
0
250
UILayoutGuideを活用する/love_swift36
noppefoxwolf
1
130
CodableでカオスなJSONに対応していく/Otemachi02
noppefoxwolf
0
110
Other Decks in Programming
See All in Programming
近況PHP / PHP in now a days
uzulla
0
410
設計とテストの必要性について考える
akeno
0
140
Reactでアプリケーションを構築する多様化
sakito
4
2.5k
TechFeed Conference 2022 - Kotlin Experimental
jmatsu
0
310
Angular's new Standalone Components: How Will They Affect My Architecture? @iJS London 2022
manfredsteyer
PRO
0
370
職場にPythonistaを増やす方法
soogie
0
170
Cloud Bigtable を使いこなす秘訣 2022
kusahana
0
220
クリエイティブ系のウェブサイト制作で役立つCSS技法 / CSS for develop creative website
clockmaker
1
890
Kotlin KSP - Intro
taehwandev
0
370
型パラメータが使えるようになったのでLINQを実装してみた
makki_d
2
500
microCMS × Shopifyで、ECサイトがリニューアル後急成長した話
microcms
0
440
Githubのアクティビティ履歴からチームの健康状態を知る(Findy Teams使ってみた)
kumagoro95
0
120
Featured
See All Featured
The MySQL Ecosystem @ GitHub 2015
samlambert
238
11k
Facilitating Awesome Meetings
lara
29
3.9k
Ruby is Unlike a Banana
tanoku
91
9.2k
Writing Fast Ruby
sferik
612
57k
Fontdeck: Realign not Redesign
paulrobertlloyd
73
4.1k
What’s in a name? Adding method to the madness
productmarketing
11
1.5k
Making the Leap to Tech Lead
cromwellryan
113
6.9k
Web Components: a chance to create the future
zenorocha
303
40k
No one is an island. Learnings from fostering a developers community.
thoeni
9
1.1k
Debugging Ruby Performance
tmm1
65
10k
Art, The Web, and Tiny UX
lynnandtonic
280
17k
Keith and Marios Guide to Fast Websites
keithpitt
404
21k
Transcript
google/mediapipe Ͱ࢝ ΊΔARΞϓϦ։ൃ ϨΪϡϥʔτʔΫʢ20ʣ noppe 1
ର • ARΞϓϦʹڵຯ͕͋Δ • τϥοΩϯάʹڵຯ͕͋Δ • ػցֶश͔Βͳͯ͋͘Γ·ͤΜɻ 2
noppe • גࣜձࣾσΟʔɾΤψɾΤʔ • ιʔγϟϧϥΠϒΞϓϦ Pococha • ݸਓ։ൃऀ • vear
ReplayKitͰεϚϗ͔Β৴Ͱ͖Δ VTuberΞϓϦ • iOSDC18,19 • ͖ͭͶ͕͖ 3
4
5
noppe • גࣜձࣾσΟʔɾΤψɾΤʔ • ιʔγϟϧϥΠϒΞϓϦ Pococha • ݸਓ։ൃऀ • vear
ReplayKitͰεϚϗ͔Β৴Ͱ͖Δ VTuberΞϓϦ • iOSDC18,19 • ͖ͭͶ͕͖ 6
mediapipe ͱʁ 7
mediapipe github.com/google/mediapipe • ॲཧ(Culculator)Λܨ͛ͯύΠϓϥΠϯʢGraphʣΛߏங͢Δ πʔϧ • ओʹը૾ॲཧͷύΠϓϥΠϯߏஙʹΘΕΔ • MLॲཧΛࠞͥࠐΉࣄ͕ग़དྷΔ 8
9
10
mediapipe Culculatorɺ࣮ࡁΈͷ͑Δ • LuminanceCalculator RGBΛड͚औΓɺًը૾Λग़ྗ͢Δ • SobelEdgesCalculator SobelϑΟϧλΛ͔͚ͨը૾Λग़ྗ͢Δ ...etc 11
12
࣮ࡁΈͷGraph ͜ΕΒͷ࣮͙͢ʹ͑Δios ࠓ͜ΕΛ͏ྲྀΕΛղઆ ios Ұ෦iOSͰ࣮͕ͳ͍ͷ͋Δ 13
ΞδΣϯμ • άϥϑͷϏϧυ • ARKitͱͷ࿈ܞ • Ͱ͖ΔࣄɾͰ͖ͳ͍ࣄ 14
άϥϑͷϏϧυ 15
github.com/noppefoxwolf/HandTracker ࠓճ͏ιʔείʔυ 16
Graph • ߏஙͨ͠GraphόΠφϦͱͯ͠Ϗϧυ ͢Δ͜ͱ͕Ͱ͖Δ • όΠφϦΛObjC͔Β͏࣮͕͋Δͷ ͰɺͦΕΒΛͬͯϑϨʔϜϫʔΫΛ࡞ Δࣄ͕Ͱ͖Δɻ • ࠓσϞͰհͨ͠ϋϯυτϥοΧʔ
ΛϑϨʔϜϫʔΫͱͯ͠Ϗϧυ 17
18
άϥϑͷϏϧυ ϏϧυπʔϧΛͬͯϏϧυΛߦ͏ • bazelbuild/bazel (ϕΠθϧ) • KubernetesTensorFlowͰ࠾༻ Track E -
Ryo Aoyama BazelΛར༻ͨ͠Micro Modular Architecture 19
άϥϑͷϏϧυ bazelͷΠϯετʔϧεΫϦϓτམͱ͖࣮ͯͯ͠ߦ͢Δ͚ͩ curl -LO "https://.../bazel-3.2.0-installer-darwin-x86_64.sh" chmod +x "bazel-3.2.0-installer-darwin-x86_64.sh" ./bazel-3.2.0-installer-darwin-x86_64.sh 20
άϥϑͷϏϧυ XcodeͰϏϧυ͕Ͱ͖Δtulsi - bazelbuild/tulsi tulsi BuildPhaseScriptͰbazelΛୟ͚ͩ͘ͳͷͰɺඞਢͰͳ͍ 21
άϥϑͷϏϧυ tulsiϏϧυεΫϦϓτୟ͚ͩ͘ͰΠϯετʔϧͰ͖Δʢཁbazelʣ git clone git@github.com:bazelbuild/tulsi.git ./build_and_run.sh 22
ͦͷ΄͔ඞཁͳͷ • Xcode • ࣌ؒ • ωοτϫʔΫ 23
ϏϧυͷྲྀΕ 1. ϑϨʔϜϫʔΫͷίʔυΛॻ͘ 2. BUILDϑΝΠϧʹՌͷใґଘੑͳͲΛهड़͢Δ 3. bazelͰbuild͢Δ 4. framework͕ग़དྷ্͕Δ 24
ϑϨʔϜϫʔΫͷίʔυ Λॻ͘ 25
ϑϨʔϜϫʔΫͷίʔυΛॻ͘ 1. binarypbΛಡΈࠐΜͰMPPGraphΛ࡞Δ 2. MPPGraphΛ։࢝͢Δ 3. MPPGraphʹը૾ΛૹΔ 4. delegateͰ݁ՌΛड͚औΔ 26
27
binarypbΛಡΈࠐΜͰMPPGraphΛ࡞ Δ + (instanceType) init { NSURL* url = [NSURL
...@"hand_landmarks.binarypb"]; NSData* data = [NSData dataWithContentsOfURL:url options:0 error:nil]; mediapipe::CalculatorGraphConfig config; config.ParseFromArray(data.bytes, data.length); self.graph = [[MPPGraph alloc] initWithGraphConfig:config]; } 28
MPPGraphΛ։࢝͢Δ GraphͰར༻͢Δ֤छίϯϙʔωϯτͷॳظԽ͕ߦΘΕΔ - (void)start { [graph startWithError: nil]; } 29
MPPGraphʹը૾ΛૹΔ - (void)sendPixelBuffer:(CVPixelBufferRef)pixelBuffer { [self.mediapipeGraph sendPixelBuffer:pixelBuffer intoStream:@"input_video" packetType:MPPPacketTypePixelBuffer]; } 30
delegateͰ݁ՌΛड͚औΔ - (void)mediapipeGraph:(MPPGraph*)graph didOutputPacket:(const ::mediapipe::Packet&)packet fromStream:(const std::string&)streamName { const auto&
timestamp = packet.Timestamp().Value(); const auto& landmarks = packet.Get<::mediapipe::NormalizedLandmarkList>(); NSArray *landmarkObjects = [self toObject: landmarks]; [delegate didOutputLandmarks: landmarkObjects]; } 31
BUILDϑΝΠϧॻ͘ 32
BUILDϑΝΠϧͱ bazelͰϏϧυ͢ΔϑΝΠϧґଘؔΛهड़ͨ͠ϑΝΠϧ 33
objc_library( name = "HandTrackerLibrary", hdrs = ["HandTracker.h"], srcs = ["HandTracker.mm"],
data = [ "hand_tracking:hand_tracking_mobile_gpu_binary_graph", ... ], deps = [ "//mediapipe/objc:mediapipe_framework_ios", "//mediapipe/objc:mediapipe_input_sources_ios", "//mediapipe/graphs/hand_tracking:mobile_calculators", "//mediapipe/framework/formats:landmark_cc_proto", ], ), ) 34
35
bazelͰϏϧυ͢Δ $ bazel build mediapipe/iosdc:HandTracker 36
framework͕ग़དྷ্͕Δ githubʹஔ͍͓͖ͯ·ͨ͠ https://github.com/noppefoxwolf/HandTracker 37
ARKitͱͷ࿈ܞ 38
ARKitͱͷ࿈ܞ ARKitͷ্༷ग़དྷͳ͍ࣄΛmediapipeͰิ͏͜ͱͰɺදݱྗΛղ ์Ͱ͖Δ • AR্ۭؒͷΦϒδΣΫτΛखͰૢ࡞͢Δ • ݕग़ͨ͠ΦϒδΣΫτͷԣʹΩϟϥΫλʔΛஔ͢Δ • إͷࠎ֨Λิਖ਼͠ͳ͕Β৭Λม͑Δhair hair
Hair Segmentationݱঢ়iOSඇରԠͰ͕͢ 39
ARKitͱͷ࿈ܞΞϓϦ։ൃ ͜ΜͳͷΛ࡞Γ·͢ 40
41
ARKit࿈ܞͷྲྀΕ 1. ARFrame͔ΒcaptureImageΛऔΓग़͢ 2. PixelFormatΛม 3. TrackerૹΔ 4. DelegateͰ݁ՌΛड͚औΔ 5.
RealityKitͷϘλϯΛ্ԼɾΧϯτΛ දࣔ 42
ARFrame͔ΒcaptureImageΛऔΓग़͢ • captureImage = Χϝϥ͔Βͷө૾ arSession.delegate = self ... //
ARSessionDelegate func session(_ session: ARSession, didUpdate frame: ARFrame) { let captureImage: CVPixelBuffer = frame.capturedImage } 43
PixelFormatΛม ը૾σʔλ͕ͲͷΑ͏ʹ֨ೲ͞Ε͍ͯΔ͔Λද͍ͯ͠Δ • ARKitͰYCbCrܗࣜ • mediapipeBGRAͷΈΛड͚औΕΔ →มͷඞཁ͕͋Δ 44
kCVPixelFormatType_32BGRA 1ϐΫηϧ32bitͰදݱ͢ΔϑΥʔϚοτ B:FF G:00 R:00 A:FF → 45
kCVPixelFormatType_420YpCbCr8BiPl anarFullRange • 2ຕͷY(8bit)+CbCr(16bit)ͷΈ߹ΘͤͰ1ϑϨʔϜΛදݱ͢Δ ϑΥʔϚοτ • ܰྔͳͷͰϏσΦσʔλͰΘΕΔ͜ͱ͕ଟ͍ 46
YCbCr͔ΒBGRAͷม • Accelerate • vImage • Metal • MSL GPUΛ͏Metalͷํ͕ߴ
47
YCbCr͔ΒBGRAͷม ࣍ͷܭࢉࣜͰม͕Ͱ͖Δ R = Y + 1.402 × Cr G
= Y - 0.344136 × Cb - 0.714136 × Cr B = Y + 1.772 × Cb A = 1 48
BlueDress YCbCr͔ΒBGRAʹߴίϯόʔτ͢ΔϥΠϒϥϦ https://github.com/noppefoxwolf/BlueDress 49
TrackerૹΔ let captureImage = frame.capturedImage let bgraCaptureImage = converter.convertToBGRA(captureImage) handTracker.send(bgraCaptureImage)
50
DelegateͰ݁ՌΛड͚औΔ ؔઅͷҐஔX,Y,ZͰऔΕ·͢…͕ɺੈք࠲ඪܥͰແ͘ X,Y: Screen࠲ඪܥ Z: खट͔ΒͷԞߦ͖ ͔͠औΕ·ͤΜɻ೦ʂ 51
52
RealityKitͷϘλϯΛ্ԼɾΧϯτΛ දࣔ ࠓճਓࠩ͠ࢦͷԆઢ্ʹEntity͕͋ΕɺisPressΛtrueʹ͢Δ func handTracker(_ handTracker: HandTracker!, didOutputLandmarks landmarks: [Landmark]!)
{ let indexFinderPosition = landmarks[8] let screenLocation: CGPoint = ...(indexFinderPosition) self.isPress = arView .entities(at: screenLocation) .contains(where: { $0.id == self.buttonEntity.id }) } 53
RealityKitͷϘλϯΛ্ԼɾΧϯτΛ දࣔ private func press() { self.scene.notifications.press.post() count += 1
changeText("\(count)") } 54
55
56
Ͱ͖ΔࣄͰ͖ͳ͍ࣄ 57
ؾʹͳΔͱ͜Ζ • HandTrackerΧϝϥ͔Βͷڑ͕औΕΔΘ͚Ͱͳ͍ࣄʹҙ →IrisͰԞߦ͖͕औΕΔ • ֤ؔઅͷճసऔΕͳ͍ →ࣗͰܭࢉͯࣗ͠વʹݟͤΔ͕ඞཁ • ϑϨʔϜϫʔΫαΠζ͕·͊·͊Ͱ͔͍ →100MB͘Β͍
58
͍͍ͱ͜Ζ • ࣮ࡁΈͷάϥϑ͕ඇৗʹ࣭͕ߴ͍ • OSS • ϚϧνϓϥοτϑΥʔϜ 59
ͬͯΈ͍ͨ͜ͱ • HairSegmentationͳͲΛiOSʹରԠ͍ͯ͠ͳ͍ͷԿނͳͷ͔ௐ ͍ͨ • NSObjectΛܦ༝͠ͳ͚ΕߴԽ͕ΊΔͷͰ • ϥϯυϚʔΫ͚ͩΛग़ྗ͢ΔάϥϑΛ࡞ͬͯΈ͍ͨ 60
͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝ ·ͨ͠ 61