Detect Body and Hand Pose with Vision

Transcript

1. Koki Hirokawa Detect Body and Hand Pose With Vision CA.swift #12
2. ኍ઒߉ل / Koki Hirokawa / ͽʔͻΖ GitHub -> KokiHirokawa Twitter -> @pihero13
3. ਓؒͷࢹ֮γεςϜʹ͓͍ͯߦ͑ΔλεΫΛίϯϐϡʔλͰ࣮ݱ͢Δ͜ͱΛ௥ٻ͢Δݚڀ෼໺ Computer Vision ͱ͸ʁ
4. iOS 11 Ͱಋೖ͞ΕͨϑϨʔϜϫʔΫ إͷϥϯυϚʔΫݕग़ɾςΩετݕग़ɾόʔίʔυೝࣝɾը૾ೝࣝ ෼ྨ΍෺ମݕग़ʹಠࣗͷ Core ML ϞσϧΛ࢖༻͢Δ͜ͱ΋Մೳ iOS / Vision ͱ͸ʁ
5. ମͱखͷϙʔζΛݕ஌Ͱ͖ΔΑ͏ʹͳͬͨ Detect Body and Hand Pose with Vision
6. ΋͏ࣗࡱΓ๮͸͍Βͳ͍ ಛఆͷδΣενϟʔΛݕ஌ͯ͠γϟολʔΛ੾Δ ׆༻ྫᶃ
7. δΣενϟʔͷ෼ྨ ֆจࣈΛΦʔόʔϨΠ ׆༻ྫᶄ
8. ࣮ࡍͲͷ͘Β͍ͷύϑΥʔϚϯε͕ ग़ͤΔͷʁ
9. ͪ͜Β͸αϯϓϧΞϓϦͱͯ͠഑෍͞Ε͍ͯ·͢ ۭதͰจࣈΛॻ͚ΔΞϓϦͰݕূ
10. ྲྀੴʹֆ͸ݫ͍͠Μ͡Όͳ͍ʁ
11. ԿΛඳ͍͍ͯΔ͔νϟοτͰ౰ͯͯΈͯԼ͍͞
12. ࢥͬͨΑΓ΋ઢ͕៉ྷʹॻ͚Δʂ ࢦΛ੎͍Α͘ಈ͔ͯ͠΋ݕ஌ͯ͘͠ΕΔʂ Beta ൛͚ͩͲɺ҆ఆ͍ͯ͠Δɻ ͨͩɺશͯͷࢦ͕ݟ͑ΔΑ͏ʹ͠ͳ͍ͱ্ख͘ݕ஌ͯ͘͠Εͳ͍ɻ ମݧͯ͠Έͨॴײ
13. ը໘ͷ୺Ͱ͸্ख͘ػೳ͠ͳ͍͜ͱ͕͋Δ Χϝϥͷࢹ໺ํ޲ʹରͯ͠ฒߦͳख͸ݕ஌͕೉͍͠ खାΛ͍ͯ͠Δ৔߹͸ݕ஌͕೉͍͠ ଍Λखͱצҧ͍ͯ͠͠·͏͜ͱ͕͋Δ σόΠε͸ݻఆ͞ΕΔඞཁ͕͋Δ ਫ਼౓ΛߴΊΔͨΊͷߟྀࣄ߲
14. ࣮૷໘ͷ࿩ʹҠΓ·͢
15. Request ͷ࡞੒ Handler ͷ࡞੒ & Handler ʹ Request Λ౉͢ ݁Ռͱͯ͠ Observation Λऔಘ͢Δ λεΫͷྲྀΕ
16. var handPoseRequest = VNDetectHumanHandPoseRequest() let handler = VNImageRequestHandler( cmSampleBuffer: sampleBuffer, orientation: .up, options: [:] ) try! handler.perform([handPoseRequest]) let observation = handPoseRequest.results?.first as! VNRecognizedPointsObservation
17. var handPoseRequest = VNDetectHumanHandPoseRequest() let handler = VNImageRequestHandler( cmSampleBuffer: sampleBuffer, orientation: .up, options: [:] ) try! handler.perform([handPoseRequest]) let observation = handPoseRequest.results?.first as! VNRecognizedPointsObservation
18. var handPoseRequest = VNDetectHumanHandPoseRequest() let handler = VNImageRequestHandler( cmSampleBuffer: sampleBuffer, orientation: .up, options: [:] ) try! handler.perform([handPoseRequest]) let observation = handPoseRequest.results?.first as! VNRecognizedPointsObservation
19. var handPoseRequest = VNDetectHumanHandPoseRequest() let handler = VNImageRequestHandler( cmSampleBuffer: sampleBuffer, orientation: .up, options: [:] ) try! handler.perform([handPoseRequest]) let observation = handPoseRequest.results?.first as! VNRecognizedPointsObservation
20. public func captureOutput( _ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection ) { var handPoseRequest = VNDetectHumanHandPoseRequest() let handler = VNImageRequestHandler(cmSampleBuffer: sampleBuffer, …) try handler.perform([handPoseRequest]) let observation = handPoseRequest.results?.first as! VNRecognizedPointsObservation } AVCaptureVideoDataOutputSampleBufferDelegate AVCaptureSession AVCaptureVideoDataOutput
21. ֤ϥϯυϚʔΫͷऔಘ - VNRecognizedPointGroupKey Thumb Index Finger Middle Finger Ring Finger Little Finger
22. ֤ϥϯυϚʔΫͷऔಘ - VNRecognizedPoint
23. VNRecognizedPoint • ࠲ඪͷ৴པ౓ • x ࠲ඪ • y ࠲ඪ • ϥϕϧ
24. let thumbPoints = try observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyThumb) let thumbTipPoints = thumbPoints[.handLandmarkKeyThumbTIP]! let x = thumbTipPoints.location.x let y = thumbTipPoints.location.y
25. let thumbPoints = try observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyThumb) let thumbTipPoints = thumbPoints[.handLandmarkKeyThumbTIP]! let x = thumbTipPoints.location.x let y = thumbTipPoints.location.y
26. let thumbPoints = try observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyThumb) let thumbTipPoints = thumbPoints[.handLandmarkKeyThumbTIP]! let x = thumbTipPoints.location.x let y = thumbTipPoints.location.y
28. VNDetectHumanHandPoseRequest ͷ maximumHandCount Λௐ੔͢Δ͜ͱͰෳ਺ͷख Λݕ஌͢Δ͜ͱ͕Մೳ σϑΥϧτ͸ 2 ύϥϝʔλͷ஋Λେ͖͘͢Ε͹ͦͷ෼͚ͩϨΠςϯγʹӨڹ͕ੜ͡Δ ௿͘ઃఆ͢Ε͹ɺͦͷ਺Ҏ্ͷख͸ݕग़͞Εͳ͘ͳΓύϑΥʔϚϯε͸্͕Δ ϨΠςϯγʹؔ͢Δ஫ҙࣄ߲
29. BODY POSE ΋΄΅ಉ͡ॲཧʹͳΓ·͢
30. ARKit ͷϞʔγϣϯΩϟϓνϟʔͱͷҧ͍
31. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠
32. ग़య ʮDetect Body and Hand Pose with Visionʯ ؔ࿈ηογϣϯ ʮCreate ML Λ࢖༻ͨ͠ΞΫγϣϯ෼ྨࢠͷ࡞੒ʯ ʮComputer Vision API ͷ୳ٻʯ