Slide 1

Slide 1 text

Koki Hirokawa Detect Body and Hand Pose With Vision CA.swift #12

Slide 2

Slide 2 text

ኍ઒߉ل / Koki Hirokawa / ͽʔͻΖ GitHub -> KokiHirokawa Twitter -> @pihero13

Slide 3

Slide 3 text

ਓؒͷࢹ֮γεςϜʹ͓͍ͯߦ͑ΔλεΫΛίϯϐϡʔλͰ࣮ݱ͢Δ͜ͱΛ௥ٻ͢Δݚڀ෼໺ Computer Vision ͱ͸ʁ

Slide 4

Slide 4 text

iOS 11 Ͱಋೖ͞ΕͨϑϨʔϜϫʔΫ إͷϥϯυϚʔΫݕग़ɾςΩετݕग़ɾόʔίʔυೝࣝɾը૾ೝࣝ ෼ྨ΍෺ମݕग़ʹಠࣗͷ Core ML ϞσϧΛ࢖༻͢Δ͜ͱ΋Մೳ iOS / Vision ͱ͸ʁ

Slide 5

Slide 5 text

ମͱखͷϙʔζΛݕ஌Ͱ͖ΔΑ͏ʹͳͬͨ Detect Body and Hand Pose with Vision

Slide 6

Slide 6 text

΋͏ࣗࡱΓ๮͸͍Βͳ͍ ಛఆͷδΣενϟʔΛݕ஌ͯ͠γϟολʔΛ੾Δ ׆༻ྫᶃ

Slide 7

Slide 7 text

δΣενϟʔͷ෼ྨ ֆจࣈΛΦʔόʔϨΠ ׆༻ྫᶄ

Slide 8

Slide 8 text

࣮ࡍͲͷ͘Β͍ͷύϑΥʔϚϯε͕ ग़ͤΔͷʁ

Slide 9

Slide 9 text

ͪ͜Β͸αϯϓϧΞϓϦͱͯ͠഑෍͞Ε͍ͯ·͢ ۭதͰจࣈΛॻ͚ΔΞϓϦͰݕূ

Slide 10

Slide 10 text

ྲྀੴʹֆ͸ݫ͍͠Μ͡Όͳ͍ʁ

Slide 11

Slide 11 text

ԿΛඳ͍͍ͯΔ͔νϟοτͰ౰ͯͯΈͯԼ͍͞

Slide 12

Slide 12 text

ࢥͬͨΑΓ΋ઢ͕៉ྷʹॻ͚Δʂ ࢦΛ੎͍Α͘ಈ͔ͯ͠΋ݕ஌ͯ͘͠ΕΔʂ Beta ൛͚ͩͲɺ҆ఆ͍ͯ͠Δɻ ͨͩɺશͯͷࢦ͕ݟ͑ΔΑ͏ʹ͠ͳ͍ͱ্ख͘ݕ஌ͯ͘͠Εͳ͍ɻ ମݧͯ͠Έͨॴײ

Slide 13

Slide 13 text

ը໘ͷ୺Ͱ͸্ख͘ػೳ͠ͳ͍͜ͱ͕͋Δ Χϝϥͷࢹ໺ํ޲ʹରͯ͠ฒߦͳख͸ݕ஌͕೉͍͠ खାΛ͍ͯ͠Δ৔߹͸ݕ஌͕೉͍͠ ଍Λखͱצҧ͍ͯ͠͠·͏͜ͱ͕͋Δ σόΠε͸ݻఆ͞ΕΔඞཁ͕͋Δ ਫ਼౓ΛߴΊΔͨΊͷߟྀࣄ߲

Slide 14

Slide 14 text

࣮૷໘ͷ࿩ʹҠΓ·͢

Slide 15

Slide 15 text

Request ͷ࡞੒ Handler ͷ࡞੒ & Handler ʹ Request Λ౉͢ ݁Ռͱͯ͠ Observation Λऔಘ͢Δ λεΫͷྲྀΕ

Slide 16

Slide 16 text

var handPoseRequest = VNDetectHumanHandPoseRequest() let handler = VNImageRequestHandler( cmSampleBuffer: sampleBuffer, orientation: .up, options: [:] ) try! handler.perform([handPoseRequest]) let observation = handPoseRequest.results?.first as! VNRecognizedPointsObservation

Slide 17

Slide 17 text

var handPoseRequest = VNDetectHumanHandPoseRequest() let handler = VNImageRequestHandler( cmSampleBuffer: sampleBuffer, orientation: .up, options: [:] ) try! handler.perform([handPoseRequest]) let observation = handPoseRequest.results?.first as! VNRecognizedPointsObservation

Slide 18

Slide 18 text

var handPoseRequest = VNDetectHumanHandPoseRequest() let handler = VNImageRequestHandler( cmSampleBuffer: sampleBuffer, orientation: .up, options: [:] ) try! handler.perform([handPoseRequest]) let observation = handPoseRequest.results?.first as! VNRecognizedPointsObservation

Slide 19

Slide 19 text

var handPoseRequest = VNDetectHumanHandPoseRequest() let handler = VNImageRequestHandler( cmSampleBuffer: sampleBuffer, orientation: .up, options: [:] ) try! handler.perform([handPoseRequest]) let observation = handPoseRequest.results?.first as! VNRecognizedPointsObservation

Slide 20

Slide 20 text

public func captureOutput( _ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection ) { var handPoseRequest = VNDetectHumanHandPoseRequest() let handler = VNImageRequestHandler(cmSampleBuffer: sampleBuffer, …) try handler.perform([handPoseRequest]) let observation = handPoseRequest.results?.first as! VNRecognizedPointsObservation } AVCaptureVideoDataOutputSampleBufferDelegate AVCaptureSession AVCaptureVideoDataOutput

Slide 21

Slide 21 text

֤ϥϯυϚʔΫͷऔಘ - VNRecognizedPointGroupKey Thumb Index Finger Middle Finger Ring Finger Little Finger

Slide 22

Slide 22 text

֤ϥϯυϚʔΫͷऔಘ - VNRecognizedPoint

Slide 23

Slide 23 text

VNRecognizedPoint • ࠲ඪͷ৴པ౓ • x ࠲ඪ • y ࠲ඪ • ϥϕϧ

Slide 24

Slide 24 text

let thumbPoints = try observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyThumb) let thumbTipPoints = thumbPoints[.handLandmarkKeyThumbTIP]! let x = thumbTipPoints.location.x let y = thumbTipPoints.location.y

Slide 25

Slide 25 text

let thumbPoints = try observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyThumb) let thumbTipPoints = thumbPoints[.handLandmarkKeyThumbTIP]! let x = thumbTipPoints.location.x let y = thumbTipPoints.location.y

Slide 26

Slide 26 text

let thumbPoints = try observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyThumb) let thumbTipPoints = thumbPoints[.handLandmarkKeyThumbTIP]! let x = thumbTipPoints.location.x let y = thumbTipPoints.location.y

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

VNDetectHumanHandPoseRequest ͷ maximumHandCount Λௐ੔͢Δ͜ͱͰෳ਺ͷख Λݕ஌͢Δ͜ͱ͕Մೳ σϑΥϧτ͸ 2 ύϥϝʔλͷ஋Λେ͖͘͢Ε͹ͦͷ෼͚ͩϨΠςϯγʹӨڹ͕ੜ͡Δ ௿͘ઃఆ͢Ε͹ɺͦͷ਺Ҏ্ͷख͸ݕग़͞Εͳ͘ͳΓύϑΥʔϚϯε͸্͕Δ ϨΠςϯγʹؔ͢Δ஫ҙࣄ߲

Slide 29

Slide 29 text

BODY POSE ΋΄΅ಉ͡ॲཧʹͳΓ·͢

Slide 30

Slide 30 text

ARKit ͷϞʔγϣϯΩϟϓνϟʔͱͷҧ͍

Slide 31

Slide 31 text

͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠

Slide 32

Slide 32 text

ग़య ʮDetect Body and Hand Pose with Visionʯ ؔ࿈ηογϣϯ ʮCreate ML Λ࢖༻ͨ͠ΞΫγϣϯ෼ྨࢠͷ࡞੒ʯ ʮComputer Vision API ͷ୳ٻʯ