Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Detect Body and Hand Pose with Vision
Koki Hirokawa
July 01, 2020
Technology
0
640
Detect Body and Hand Pose with Vision
Koki Hirokawa
July 01, 2020
Tweet
Share
More Decks by Koki Hirokawa
See All by Koki Hirokawa
ABEMAのリアーキテクチャ
kokihirokawa
5
2.7k
Other Decks in Technology
See All in Technology
サイボウズの アジャイル・クオリティ / Agile Quality at Cybozu
cybozuinsideout
PRO
4
2.5k
【toranoana.deno#7】Denoからwasmを呼び出す基礎
toranoana
0
130
UIKitのアップデート #WWDC22
akatsuki174
4
350
Strategyパターン
hankehly
0
160
データエンジニアと作るデータ文化
yuki_saito
4
1.7k
Google Cloud Updates 2022/05/16-05/31
no24oka
2
110
アーキテクチャを明文化して開発に臨んだ話
akkie76
0
360
ノーコードで Stripeを使いこなす3つの方法 / jp-stripes-online-vol-4
stripehideokamoto
0
320
20220622_FinJAWS_あのときにAWSがあったらこうできた
taketakekaho
0
120
【配布資料】複数データソースのデータを仮想化してまとめて分析してみよう(AWS、IBM Cloud)
tkhresk
0
100
HoloLens2とMetaQuest2どちらも動くWebXRアプリをBabylon.jsで作る
iwaken71
0
210
FoodTechにおける商流・金流・物流の進化/Evolution of Commercial, Financial, and Logistics in FoodTech
dskst
0
410
Featured
See All Featured
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
119
28k
Pencils Down: Stop Designing & Start Developing
hursman
112
9.8k
For a Future-Friendly Web
brad_frost
166
7.4k
The Web Native Designer (August 2011)
paulrobertlloyd
74
1.9k
Art Directing for the Web. Five minutes with CSS Template Areas
malarkey
196
9.4k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
15
950
A Tale of Four Properties
chriscoyier
149
21k
Clear Off the Table
cherdarchuk
79
280k
Code Reviewing Like a Champion
maltzj
506
37k
How To Stay Up To Date on Web Technology
chriscoyier
780
250k
The MySQL Ecosystem @ GitHub 2015
samlambert
238
11k
The Power of CSS Pseudo Elements
geoffreycrofte
47
3.9k
Transcript
Koki Hirokawa Detect Body and Hand Pose With Vision CA.swift
#12
ኍ߉ل / Koki Hirokawa / ͽʔͻΖ GitHub -> KokiHirokawa Twitter
-> @pihero13
ਓؒͷࢹ֮γεςϜʹ͓͍ͯߦ͑ΔλεΫΛίϯϐϡʔλͰ࣮ݱ͢Δ͜ͱΛٻ͢Δݚڀ Computer Vision ͱʁ
iOS 11 Ͱಋೖ͞ΕͨϑϨʔϜϫʔΫ إͷϥϯυϚʔΫݕग़ɾςΩετݕग़ɾόʔίʔυೝࣝɾը૾ೝࣝ ྨମݕग़ʹಠࣗͷ Core ML ϞσϧΛ༻͢Δ͜ͱՄೳ iOS /
Vision ͱʁ
ମͱखͷϙʔζΛݕͰ͖ΔΑ͏ʹͳͬͨ Detect Body and Hand Pose with Vision
͏ࣗࡱΓ͍Βͳ͍ ಛఆͷδΣενϟʔΛݕͯ͠γϟολʔΛΔ ׆༻ྫᶃ
δΣενϟʔͷྨ ֆจࣈΛΦʔόʔϨΠ ׆༻ྫᶄ
࣮ࡍͲͷ͘Β͍ͷύϑΥʔϚϯε͕ ग़ͤΔͷʁ
ͪ͜ΒαϯϓϧΞϓϦͱͯ͠͞Ε͍ͯ·͢ ۭதͰจࣈΛॻ͚ΔΞϓϦͰݕূ
ྲྀੴʹֆݫ͍͠Μ͡Όͳ͍ʁ
ԿΛඳ͍͍ͯΔ͔νϟοτͰͯͯΈͯԼ͍͞
ࢥͬͨΑΓઢ͕៉ྷʹॻ͚Δʂ ࢦΛ͍Α͘ಈ͔ͯ͠ݕͯ͘͠ΕΔʂ Beta ൛͚ͩͲɺ҆ఆ͍ͯ͠Δɻ ͨͩɺશͯͷࢦ͕ݟ͑ΔΑ͏ʹ͠ͳ͍ͱ্ख͘ݕͯ͘͠Εͳ͍ɻ ମݧͯ͠Έͨॴײ
ը໘ͷͰ্ख͘ػೳ͠ͳ͍͜ͱ͕͋Δ Χϝϥͷࢹํʹରͯ͠ฒߦͳखݕ͕͍͠ खାΛ͍ͯ͠Δ߹ݕ͕͍͠ Λखͱצҧ͍ͯ͠͠·͏͜ͱ͕͋Δ σόΠεݻఆ͞ΕΔඞཁ͕͋Δ ਫ਼ΛߴΊΔͨΊͷߟྀࣄ߲
࣮໘ͷʹҠΓ·͢
Request ͷ࡞ Handler ͷ࡞ & Handler ʹ Request Λ͢ ݁Ռͱͯ͠
Observation Λऔಘ͢Δ λεΫͷྲྀΕ
var handPoseRequest = VNDetectHumanHandPoseRequest() let handler = VNImageRequestHandler( cmSampleBuffer: sampleBuffer,
orientation: .up, options: [:] ) try! handler.perform([handPoseRequest]) let observation = handPoseRequest.results?.first as! VNRecognizedPointsObservation
var handPoseRequest = VNDetectHumanHandPoseRequest() let handler = VNImageRequestHandler( cmSampleBuffer: sampleBuffer,
orientation: .up, options: [:] ) try! handler.perform([handPoseRequest]) let observation = handPoseRequest.results?.first as! VNRecognizedPointsObservation
var handPoseRequest = VNDetectHumanHandPoseRequest() let handler = VNImageRequestHandler( cmSampleBuffer: sampleBuffer,
orientation: .up, options: [:] ) try! handler.perform([handPoseRequest]) let observation = handPoseRequest.results?.first as! VNRecognizedPointsObservation
var handPoseRequest = VNDetectHumanHandPoseRequest() let handler = VNImageRequestHandler( cmSampleBuffer: sampleBuffer,
orientation: .up, options: [:] ) try! handler.perform([handPoseRequest]) let observation = handPoseRequest.results?.first as! VNRecognizedPointsObservation
public func captureOutput( _ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from
connection: AVCaptureConnection ) { var handPoseRequest = VNDetectHumanHandPoseRequest() let handler = VNImageRequestHandler(cmSampleBuffer: sampleBuffer, …) try handler.perform([handPoseRequest]) let observation = handPoseRequest.results?.first as! VNRecognizedPointsObservation } AVCaptureVideoDataOutputSampleBufferDelegate AVCaptureSession AVCaptureVideoDataOutput
֤ϥϯυϚʔΫͷऔಘ - VNRecognizedPointGroupKey Thumb Index Finger Middle Finger Ring Finger
Little Finger
֤ϥϯυϚʔΫͷऔಘ - VNRecognizedPoint
VNRecognizedPoint • ࠲ඪͷ৴པ • x ࠲ඪ • y ࠲ඪ •
ϥϕϧ
let thumbPoints = try observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyThumb) let thumbTipPoints = thumbPoints[.handLandmarkKeyThumbTIP]!
let x = thumbTipPoints.location.x let y = thumbTipPoints.location.y
let thumbPoints = try observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyThumb) let thumbTipPoints = thumbPoints[.handLandmarkKeyThumbTIP]!
let x = thumbTipPoints.location.x let y = thumbTipPoints.location.y
let thumbPoints = try observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyThumb) let thumbTipPoints = thumbPoints[.handLandmarkKeyThumbTIP]!
let x = thumbTipPoints.location.x let y = thumbTipPoints.location.y
None
VNDetectHumanHandPoseRequest ͷ maximumHandCount Λௐ͢Δ͜ͱͰෳͷख Λݕ͢Δ͜ͱ͕Մೳ σϑΥϧτ 2 ύϥϝʔλͷΛେ͖͘͢Εͦͷ͚ͩϨΠςϯγʹӨڹ͕ੜ͡Δ ͘ઃఆ͢ΕɺͦͷҎ্ͷखݕग़͞Εͳ͘ͳΓύϑΥʔϚϯε্͕Δ ϨΠςϯγʹؔ͢Δҙࣄ߲
BODY POSE ΄΅ಉ͡ॲཧʹͳΓ·͢
ARKit ͷϞʔγϣϯΩϟϓνϟʔͱͷҧ͍
͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠
ग़య ʮDetect Body and Hand Pose with Visionʯ ؔ࿈ηογϣϯ ʮCreate
ML Λ༻ͨ͠ΞΫγϣϯྨࢠͷ࡞ʯ ʮComputer Vision API ͷ୳ٻʯ