Detect Body and Hand Pose with Vision

Detect Body and Hand Pose with Vision

3057ed2fe304f016e9b6d1840a33c093?s=128

Koki Hirokawa

July 01, 2020
Tweet

Transcript

  1. Koki Hirokawa Detect Body and Hand Pose With Vision CA.swift

    #12
  2. ኍ઒߉ل / Koki Hirokawa / ͽʔͻΖ GitHub -> KokiHirokawa Twitter

    -> @pihero13
  3. ਓؒͷࢹ֮γεςϜʹ͓͍ͯߦ͑ΔλεΫΛίϯϐϡʔλͰ࣮ݱ͢Δ͜ͱΛ௥ٻ͢Δݚڀ෼໺ Computer Vision ͱ͸ʁ

  4. iOS 11 Ͱಋೖ͞ΕͨϑϨʔϜϫʔΫ إͷϥϯυϚʔΫݕग़ɾςΩετݕग़ɾόʔίʔυೝࣝɾը૾ೝࣝ ෼ྨ΍෺ମݕग़ʹಠࣗͷ Core ML ϞσϧΛ࢖༻͢Δ͜ͱ΋Մೳ iOS /

    Vision ͱ͸ʁ
  5. ମͱखͷϙʔζΛݕ஌Ͱ͖ΔΑ͏ʹͳͬͨ Detect Body and Hand Pose with Vision

  6. ΋͏ࣗࡱΓ๮͸͍Βͳ͍ ಛఆͷδΣενϟʔΛݕ஌ͯ͠γϟολʔΛ੾Δ ׆༻ྫᶃ

  7. δΣενϟʔͷ෼ྨ ֆจࣈΛΦʔόʔϨΠ ׆༻ྫᶄ

  8. ࣮ࡍͲͷ͘Β͍ͷύϑΥʔϚϯε͕ ग़ͤΔͷʁ

  9. ͪ͜Β͸αϯϓϧΞϓϦͱͯ͠഑෍͞Ε͍ͯ·͢ ۭதͰจࣈΛॻ͚ΔΞϓϦͰݕূ

  10. ྲྀੴʹֆ͸ݫ͍͠Μ͡Όͳ͍ʁ

  11. ԿΛඳ͍͍ͯΔ͔νϟοτͰ౰ͯͯΈͯԼ͍͞

  12. ࢥͬͨΑΓ΋ઢ͕៉ྷʹॻ͚Δʂ ࢦΛ੎͍Α͘ಈ͔ͯ͠΋ݕ஌ͯ͘͠ΕΔʂ Beta ൛͚ͩͲɺ҆ఆ͍ͯ͠Δɻ ͨͩɺશͯͷࢦ͕ݟ͑ΔΑ͏ʹ͠ͳ͍ͱ্ख͘ݕ஌ͯ͘͠Εͳ͍ɻ ମݧͯ͠Έͨॴײ

  13. ը໘ͷ୺Ͱ͸্ख͘ػೳ͠ͳ͍͜ͱ͕͋Δ Χϝϥͷࢹ໺ํ޲ʹରͯ͠ฒߦͳख͸ݕ஌͕೉͍͠ खାΛ͍ͯ͠Δ৔߹͸ݕ஌͕೉͍͠ ଍Λखͱצҧ͍ͯ͠͠·͏͜ͱ͕͋Δ σόΠε͸ݻఆ͞ΕΔඞཁ͕͋Δ ਫ਼౓ΛߴΊΔͨΊͷߟྀࣄ߲

  14. ࣮૷໘ͷ࿩ʹҠΓ·͢

  15. Request ͷ࡞੒ Handler ͷ࡞੒ & Handler ʹ Request Λ౉͢ ݁Ռͱͯ͠

    Observation Λऔಘ͢Δ λεΫͷྲྀΕ
  16. var handPoseRequest = VNDetectHumanHandPoseRequest() let handler = VNImageRequestHandler( cmSampleBuffer: sampleBuffer,

    orientation: .up, options: [:] ) try! handler.perform([handPoseRequest]) let observation = handPoseRequest.results?.first as! VNRecognizedPointsObservation
  17. var handPoseRequest = VNDetectHumanHandPoseRequest() let handler = VNImageRequestHandler( cmSampleBuffer: sampleBuffer,

    orientation: .up, options: [:] ) try! handler.perform([handPoseRequest]) let observation = handPoseRequest.results?.first as! VNRecognizedPointsObservation
  18. var handPoseRequest = VNDetectHumanHandPoseRequest() let handler = VNImageRequestHandler( cmSampleBuffer: sampleBuffer,

    orientation: .up, options: [:] ) try! handler.perform([handPoseRequest]) let observation = handPoseRequest.results?.first as! VNRecognizedPointsObservation
  19. var handPoseRequest = VNDetectHumanHandPoseRequest() let handler = VNImageRequestHandler( cmSampleBuffer: sampleBuffer,

    orientation: .up, options: [:] ) try! handler.perform([handPoseRequest]) let observation = handPoseRequest.results?.first as! VNRecognizedPointsObservation
  20. public func captureOutput( _ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from

    connection: AVCaptureConnection ) { var handPoseRequest = VNDetectHumanHandPoseRequest() let handler = VNImageRequestHandler(cmSampleBuffer: sampleBuffer, …) try handler.perform([handPoseRequest]) let observation = handPoseRequest.results?.first as! VNRecognizedPointsObservation } AVCaptureVideoDataOutputSampleBufferDelegate AVCaptureSession AVCaptureVideoDataOutput
  21. ֤ϥϯυϚʔΫͷऔಘ - VNRecognizedPointGroupKey Thumb Index Finger Middle Finger Ring Finger

    Little Finger
  22. ֤ϥϯυϚʔΫͷऔಘ - VNRecognizedPoint

  23. VNRecognizedPoint • ࠲ඪͷ৴པ౓ • x ࠲ඪ • y ࠲ඪ •

    ϥϕϧ
  24. let thumbPoints = try observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyThumb) let thumbTipPoints = thumbPoints[.handLandmarkKeyThumbTIP]!

    let x = thumbTipPoints.location.x let y = thumbTipPoints.location.y
  25. let thumbPoints = try observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyThumb) let thumbTipPoints = thumbPoints[.handLandmarkKeyThumbTIP]!

    let x = thumbTipPoints.location.x let y = thumbTipPoints.location.y
  26. let thumbPoints = try observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyThumb) let thumbTipPoints = thumbPoints[.handLandmarkKeyThumbTIP]!

    let x = thumbTipPoints.location.x let y = thumbTipPoints.location.y
  27. None
  28. VNDetectHumanHandPoseRequest ͷ maximumHandCount Λௐ੔͢Δ͜ͱͰෳ਺ͷख Λݕ஌͢Δ͜ͱ͕Մೳ σϑΥϧτ͸ 2 ύϥϝʔλͷ஋Λେ͖͘͢Ε͹ͦͷ෼͚ͩϨΠςϯγʹӨڹ͕ੜ͡Δ ௿͘ઃఆ͢Ε͹ɺͦͷ਺Ҏ্ͷख͸ݕग़͞Εͳ͘ͳΓύϑΥʔϚϯε͸্͕Δ ϨΠςϯγʹؔ͢Δ஫ҙࣄ߲

  29. BODY POSE ΋΄΅ಉ͡ॲཧʹͳΓ·͢

  30. ARKit ͷϞʔγϣϯΩϟϓνϟʔͱͷҧ͍

  31. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠

  32. ग़య ʮDetect Body and Hand Pose with Visionʯ ؔ࿈ηογϣϯ ʮCreate

    ML Λ࢖༻ͨ͠ΞΫγϣϯ෼ྨࢠͷ࡞੒ʯ ʮComputer Vision API ͷ୳ٻʯ