Detect Body and Hand Pose with Vision

Transcript

1. Koki Hirokawa Detect Body and Hand Pose With Vision CA.swift

   #12

2. ኍ઒߉ل / Koki Hirokawa / ͽʔͻΖ GitHub -> KokiHirokawa Twitter

   -> @pihero13

3. ਓؒͷࢹ֮γεςϜʹ͓͍ͯߦ͑ΔλεΫΛίϯϐϡʔλͰ࣮ݱ͢Δ͜ͱΛ௥ٻ͢Δݚڀ෼໺ Computer Vision ͱ͸ʁ

4. iOS 11 Ͱಋೖ͞ΕͨϑϨʔϜϫʔΫ إͷϥϯυϚʔΫݕग़ɾςΩετݕग़ɾόʔίʔυೝࣝɾը૾ೝࣝ ෼ྨ΍෺ମݕग़ʹಠࣗͷ Core ML ϞσϧΛ࢖༻͢Δ͜ͱ΋Մೳ iOS /

   Vision ͱ͸ʁ

5. ମͱखͷϙʔζΛݕ஌Ͱ͖ΔΑ͏ʹͳͬͨ Detect Body and Hand Pose with Vision

6. ΋͏ࣗࡱΓ๮͸͍Βͳ͍ ಛఆͷδΣενϟʔΛݕ஌ͯ͠γϟολʔΛ੾Δ ׆༻ྫᶃ

7. δΣενϟʔͷ෼ྨ ֆจࣈΛΦʔόʔϨΠ ׆༻ྫᶄ

8. ࣮ࡍͲͷ͘Β͍ͷύϑΥʔϚϯε͕ ग़ͤΔͷʁ

9. ͪ͜Β͸αϯϓϧΞϓϦͱͯ͠഑෍͞Ε͍ͯ·͢ ۭதͰจࣈΛॻ͚ΔΞϓϦͰݕূ

10. ྲྀੴʹֆ͸ݫ͍͠Μ͡Όͳ͍ʁ

11. ԿΛඳ͍͍ͯΔ͔νϟοτͰ౰ͯͯΈͯԼ͍͞

12. ࢥͬͨΑΓ΋ઢ͕៉ྷʹॻ͚Δʂ ࢦΛ੎͍Α͘ಈ͔ͯ͠΋ݕ஌ͯ͘͠ΕΔʂ Beta ൛͚ͩͲɺ҆ఆ͍ͯ͠Δɻ ͨͩɺશͯͷࢦ͕ݟ͑ΔΑ͏ʹ͠ͳ͍ͱ্ख͘ݕ஌ͯ͘͠Εͳ͍ɻ ମݧͯ͠Έͨॴײ

13. ը໘ͷ୺Ͱ͸্ख͘ػೳ͠ͳ͍͜ͱ͕͋Δ Χϝϥͷࢹ໺ํ޲ʹରͯ͠ฒߦͳख͸ݕ஌͕೉͍͠ खାΛ͍ͯ͠Δ৔߹͸ݕ஌͕೉͍͠ ଍Λखͱצҧ͍ͯ͠͠·͏͜ͱ͕͋Δ σόΠε͸ݻఆ͞ΕΔඞཁ͕͋Δ ਫ਼౓ΛߴΊΔͨΊͷߟྀࣄ߲

14. ࣮૷໘ͷ࿩ʹҠΓ·͢

15. Request ͷ࡞੒ Handler ͷ࡞੒ & Handler ʹ Request Λ౉͢ ݁Ռͱͯ͠

    Observation Λऔಘ͢Δ λεΫͷྲྀΕ

16. var handPoseRequest = VNDetectHumanHandPoseRequest() let handler = VNImageRequestHandler( cmSampleBuffer: sampleBuffer,

    orientation: .up, options: [:] ) try! handler.perform([handPoseRequest]) let observation = handPoseRequest.results?.first as! VNRecognizedPointsObservation

17. var handPoseRequest = VNDetectHumanHandPoseRequest() let handler = VNImageRequestHandler( cmSampleBuffer: sampleBuffer,

    orientation: .up, options: [:] ) try! handler.perform([handPoseRequest]) let observation = handPoseRequest.results?.first as! VNRecognizedPointsObservation

18. var handPoseRequest = VNDetectHumanHandPoseRequest() let handler = VNImageRequestHandler( cmSampleBuffer: sampleBuffer,

    orientation: .up, options: [:] ) try! handler.perform([handPoseRequest]) let observation = handPoseRequest.results?.first as! VNRecognizedPointsObservation

19. var handPoseRequest = VNDetectHumanHandPoseRequest() let handler = VNImageRequestHandler( cmSampleBuffer: sampleBuffer,

    orientation: .up, options: [:] ) try! handler.perform([handPoseRequest]) let observation = handPoseRequest.results?.first as! VNRecognizedPointsObservation

20. public func captureOutput( _ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from

    connection: AVCaptureConnection ) { var handPoseRequest = VNDetectHumanHandPoseRequest() let handler = VNImageRequestHandler(cmSampleBuffer: sampleBuffer, …) try handler.perform([handPoseRequest]) let observation = handPoseRequest.results?.first as! VNRecognizedPointsObservation } AVCaptureVideoDataOutputSampleBufferDelegate AVCaptureSession AVCaptureVideoDataOutput

21. ֤ϥϯυϚʔΫͷऔಘ - VNRecognizedPointGroupKey Thumb Index Finger Middle Finger Ring Finger

    Little Finger

22. ֤ϥϯυϚʔΫͷऔಘ - VNRecognizedPoint

23. VNRecognizedPoint • ࠲ඪͷ৴པ౓ • x ࠲ඪ • y ࠲ඪ •

    ϥϕϧ

24. let thumbPoints = try observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyThumb) let thumbTipPoints = thumbPoints[.handLandmarkKeyThumbTIP]!

    let x = thumbTipPoints.location.x let y = thumbTipPoints.location.y

25. let thumbPoints = try observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyThumb) let thumbTipPoints = thumbPoints[.handLandmarkKeyThumbTIP]!

    let x = thumbTipPoints.location.x let y = thumbTipPoints.location.y

26. let thumbPoints = try observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyThumb) let thumbTipPoints = thumbPoints[.handLandmarkKeyThumbTIP]!

    let x = thumbTipPoints.location.x let y = thumbTipPoints.location.y
27. None

28. VNDetectHumanHandPoseRequest ͷ maximumHandCount Λௐ੔͢Δ͜ͱͰෳ਺ͷख Λݕ஌͢Δ͜ͱ͕Մೳ σϑΥϧτ͸ 2 ύϥϝʔλͷ஋Λେ͖͘͢Ε͹ͦͷ෼͚ͩϨΠςϯγʹӨڹ͕ੜ͡Δ ௿͘ઃఆ͢Ε͹ɺͦͷ਺Ҏ্ͷख͸ݕग़͞Εͳ͘ͳΓύϑΥʔϚϯε͸্͕Δ ϨΠςϯγʹؔ͢Δ஫ҙࣄ߲

29. BODY POSE ΋΄΅ಉ͡ॲཧʹͳΓ·͢

30. ARKit ͷϞʔγϣϯΩϟϓνϟʔͱͷҧ͍

31. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠

32. ग़య ʮDetect Body and Hand Pose with Visionʯ ؔ࿈ηογϣϯ ʮCreate

    ML Λ࢖༻ͨ͠ΞΫγϣϯ෼ྨࢠͷ࡞੒ʯ ʮComputer Vision API ͷ୳ٻʯ