Providing Better Feedback in Real-time Object Detection Apps

lightning talk at #tryswiftconf

Transcript

1. Providing Be,er Feedback in Real-&me Object Detec&on Apps try! Swi) Tokyo 2017 Lightning Talk shingt (Shinichi Goto) @ Wantedly, Inc. ɹ
2. Object Detec)on
3. Wantedly People
4. Goal and Problem
5. Assump&on: We already have detec&on logic
6. struct CardFeature { let coordinates: [CGPoint] ... }
7. extension DetectionViewController: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput( _ captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, ...) { ... guard let frame = sampleBuffer.toUIImage() else { return } let cardFeatures = detector.cardFeatures(in: frame) updateCircleLayersView(with: cardFeatures) ... } }
8. extension DetectionViewController: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput( _ captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, ...) { ... guard let frame = sampleBuffer.toUIImage() else { return } let cardFeatures = detector.cardFeatures(in: frame) ! // Detect cards updateCircleLayersView(with: cardFeatures) ... } }
9. extension DetectionViewController: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput( _ captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, ...) { ... guard let frame = sampleBuffer.toUIImage() else { return } let cardFeatures = detector.cardFeatures(in: frame) updateCircleLayersView(with: cardFeatures) ! // Add CALayers ... } }
11. One object
12. One object
13. Mul$ple objects
14. Mul$ple objects
15. What we want
16. Tracking • Assigning consistent labels to objects in videos • Generic trackers available in OpenCV • e.g. TLD, KCF, etc. • Performance issues on older iOS devices
17. Tracking (our case) • Combina)on of lightweight informa)on: • dHash (perceptual hash based on gradient) • Posi)on • Op)cal flow
18. struct CardFeature { let coordinates: [CGPoint] let trackingID: Int ! ... }
19. extension DetectionViewController: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput( _ captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, ...) { ... guard let frame = sampleBuffer.toUIImage() else { return } let cardFeatures = detector.cardFeatures(in: frame) // // Now we can animate circle layers // from previous positions to current positions // updateCircleLayersView(with: cardFeatures) ! ... } }
23. Face Tracking using CoreImage open class CIFaceFeature : CIFeature { open var bounds: CGRect { get } open var trackingID: Int32 { get } ! ... }
24. Face Tracking using CoreImage open class CIFaceFeature : CIFeature { open var bounds: CGRect { get } open var trackingID: Int32 { get } ! ... } Sample including anima.on: h2ps:/ /github.com/shingt/FaceTracker- sample (Instead of our product code)
25. Summary • In real-*me object detec*on apps • Difficulty in giving feedback differs depending a situa*on • Tracking can improve feedback in some cases
26. Thank you!