Koki Hirokawa Detect Body and Hand Pose With Vision CA.swift #12 

ኍ઒߉ل / Koki Hirokawa / ͽʔͻΖ GitHub -> KokiHirokawa Twitter -> @pihero13 

ਓؒͷࢹ֮γεςϜʹ͓͍ͯߦ͑ΔλεΫΛίϯϐϡʔλͰ࣮ݱ͢Δ͜ͱΛ௥ٻ͢Δݚڀ෼໺ Computer Vision ͱ͸ʁ 

iOS 11 Ͱಋೖ͞ΕͨϑϨʔϜϫʔΫ إͷϥϯυϚʔΫݕग़ɾςΩετݕग़ɾόʔίʔυೝࣝɾը૾ೝࣝ ෼ྨ΍෺ମݕग़ʹಠࣗͷ Core ML ϞσϧΛ࢖༻͢Δ͜ͱ΋Մೳ iOS / Vision ͱ͸ʁ 

ମͱखͷϙʔζΛݕ஌Ͱ͖ΔΑ͏ʹͳͬͨ Detect Body and Hand Pose with Vision 

΋͏ࣗࡱΓ๮͸͍Βͳ͍ ಛఆͷδΣενϟʔΛݕ஌ͯ͠γϟολʔΛ੾Δ ׆༻ྫᶃ 

δΣενϟʔͷ෼ྨ ֆจࣈΛΦʔόʔϨΠ ׆༻ྫᶄ 

࣮ࡍͲͷ͘Β͍ͷύϑΥʔϚϯε͕ ग़ͤΔͷʁ 

ͪ͜Β͸αϯϓϧΞϓϦͱͯ͠഑෍͞Ε͍ͯ·͢ ۭதͰจࣈΛॻ͚ΔΞϓϦͰݕূ 

ྲྀੴʹֆ͸ݫ͍͠Μ͡Όͳ͍ʁ 

ԿΛඳ͍͍ͯΔ͔νϟοτͰ౰ͯͯΈͯԼ͍͞ 

ࢥͬͨΑΓ΋ઢ͕៉ྷʹॻ͚Δʂ ࢦΛ੎͍Α͘ಈ͔ͯ͠΋ݕ஌ͯ͘͠ΕΔʂ Beta ൛͚ͩͲɺ҆ఆ͍ͯ͠Δɻ ͨͩɺશͯͷࢦ͕ݟ͑ΔΑ͏ʹ͠ͳ͍ͱ্ख͘ݕ஌ͯ͘͠Εͳ͍ɻ ମݧͯ͠Έͨॴײ 

ը໘ͷ୺Ͱ͸্ख͘ػೳ͠ͳ͍͜ͱ͕͋Δ Χϝϥͷࢹ໺ํ޲ʹରͯ͠ฒߦͳख͸ݕ஌͕೉͍͠ खାΛ͍ͯ͠Δ৔߹͸ݕ஌͕೉͍͠ ଍Λखͱצҧ͍ͯ͠͠·͏͜ͱ͕͋Δ σόΠε͸ݻఆ͞ΕΔඞཁ͕͋Δ ਫ਼౓ΛߴΊΔͨΊͷߟྀࣄ߲ 

࣮૷໘ͷ࿩ʹҠΓ·͢ 

Request ͷ࡞੒ Handler ͷ࡞੒ & Handler ʹ Request Λ౉͢ ݁Ռͱͯ͠ Observation Λऔಘ͢Δ λεΫͷྲྀΕ 

var handPoseRequest = VNDetectHumanHandPoseRequest() let handler = VNImageRequestHandler( cmSampleBuffer: sampleBuffer, orientation: .up, options: [:] ) try! handler.perform([handPoseRequest]) let observation = handPoseRequest.results?.first as! VNRecognizedPointsObservation 

var handPoseRequest = VNDetectHumanHandPoseRequest() let handler = VNImageRequestHandler( cmSampleBuffer: sampleBuffer, orientation: .up, options: [:] ) try! handler.perform([handPoseRequest]) let observation = handPoseRequest.results?.first as! VNRecognizedPointsObservation 

var handPoseRequest = VNDetectHumanHandPoseRequest() let handler = VNImageRequestHandler( cmSampleBuffer: sampleBuffer, orientation: .up, options: [:] ) try! handler.perform([handPoseRequest]) let observation = handPoseRequest.results?.first as! VNRecognizedPointsObservation 

var handPoseRequest = VNDetectHumanHandPoseRequest() let handler = VNImageRequestHandler( cmSampleBuffer: sampleBuffer, orientation: .up, options: [:] ) try! handler.perform([handPoseRequest]) let observation = handPoseRequest.results?.first as! VNRecognizedPointsObservation 

public func captureOutput( _ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection ) { var handPoseRequest = VNDetectHumanHandPoseRequest() let handler = VNImageRequestHandler(cmSampleBuffer: sampleBuffer, …) try handler.perform([handPoseRequest]) let observation = handPoseRequest.results?.first as! VNRecognizedPointsObservation } AVCaptureVideoDataOutputSampleBufferDelegate AVCaptureSession AVCaptureVideoDataOutput 

֤ϥϯυϚʔΫͷऔಘ - VNRecognizedPointGroupKey Thumb Index Finger Middle Finger Ring Finger Little Finger 

֤ϥϯυϚʔΫͷऔಘ - VNRecognizedPoint 

VNRecognizedPoint • ࠲ඪͷ৴པ౓ • x ࠲ඪ • y ࠲ඪ • ϥϕϧ 

let thumbPoints = try observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyThumb) let thumbTipPoints = thumbPoints[.handLandmarkKeyThumbTIP]! let x = thumbTipPoints.location.x let y = thumbTipPoints.location.y 

let thumbPoints = try observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyThumb) let thumbTipPoints = thumbPoints[.handLandmarkKeyThumbTIP]! let x = thumbTipPoints.location.x let y = thumbTipPoints.location.y 

let thumbPoints = try observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyThumb) let thumbTipPoints = thumbPoints[.handLandmarkKeyThumbTIP]! let x = thumbTipPoints.location.x let y = thumbTipPoints.location.y 

No content

VNDetectHumanHandPoseRequest ͷ maximumHandCount Λௐ੔͢Δ͜ͱͰෳ਺ͷख Λݕ஌͢Δ͜ͱ͕Մೳ σϑΥϧτ͸ 2 ύϥϝʔλͷ஋Λେ͖͘͢Ε͹ͦͷ෼͚ͩϨΠςϯγʹӨڹ͕ੜ͡Δ ௿͘ઃఆ͢Ε͹ɺͦͷ਺Ҏ্ͷख͸ݕग़͞Εͳ͘ͳΓύϑΥʔϚϯε͸্͕Δ ϨΠςϯγʹؔ͢Δ஫ҙࣄ߲ 

BODY POSE ΋΄΅ಉ͡ॲཧʹͳΓ·͢ 

ARKit ͷϞʔγϣϯΩϟϓνϟʔͱͷҧ͍ 

͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ 

ग़య ʮDetect Body and Hand Pose with Visionʯ ؔ࿈ηογϣϯ ʮCreate ML Λ࢖༻ͨ͠ΞΫγϣϯ෼ྨࢠͷ࡞੒ʯ ʮComputer Vision API ͷ୳ٻʯ