Providing Better Feedback in Real-time Object Detection Apps

Providing Better Feedback in Real-time Object Detection Apps

lightning talk at #tryswiftconf

3652e140db258288623a0adf248baec0?s=128

Shinichi Goto

March 02, 2017
Tweet

Transcript

  1. Providing Be,er Feedback in Real-&me Object Detec&on Apps try! Swi)

    Tokyo 2017 Lightning Talk shingt (Shinichi Goto) @ Wantedly, Inc. ɹ
  2. Object Detec)on

  3. Wantedly People

  4. Goal and Problem

  5. Assump&on: We already have detec&on logic

  6. struct CardFeature { let coordinates: [CGPoint] ... }

  7. extension DetectionViewController: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput( _ captureOutput: AVCaptureOutput!, didOutputSampleBuffer

    sampleBuffer: CMSampleBuffer!, ...) { ... guard let frame = sampleBuffer.toUIImage() else { return } let cardFeatures = detector.cardFeatures(in: frame) updateCircleLayersView(with: cardFeatures) ... } }
  8. extension DetectionViewController: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput( _ captureOutput: AVCaptureOutput!, didOutputSampleBuffer

    sampleBuffer: CMSampleBuffer!, ...) { ... guard let frame = sampleBuffer.toUIImage() else { return } let cardFeatures = detector.cardFeatures(in: frame) ! // Detect cards updateCircleLayersView(with: cardFeatures) ... } }
  9. extension DetectionViewController: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput( _ captureOutput: AVCaptureOutput!, didOutputSampleBuffer

    sampleBuffer: CMSampleBuffer!, ...) { ... guard let frame = sampleBuffer.toUIImage() else { return } let cardFeatures = detector.cardFeatures(in: frame) updateCircleLayersView(with: cardFeatures) ! // Add CALayers ... } }
  10. None
  11. One object

  12. One object

  13. Mul$ple objects

  14. Mul$ple objects

  15. What we want

  16. Tracking • Assigning consistent labels to objects in videos •

    Generic trackers available in OpenCV • e.g. TLD, KCF, etc. • Performance issues on older iOS devices
  17. Tracking (our case) • Combina)on of lightweight informa)on: • dHash

    (perceptual hash based on gradient) • Posi)on • Op)cal flow
  18. struct CardFeature { let coordinates: [CGPoint] let trackingID: Int !

    ... }
  19. extension DetectionViewController: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput( _ captureOutput: AVCaptureOutput!, didOutputSampleBuffer

    sampleBuffer: CMSampleBuffer!, ...) { ... guard let frame = sampleBuffer.toUIImage() else { return } let cardFeatures = detector.cardFeatures(in: frame) // // Now we can animate circle layers // from previous positions to current positions // updateCircleLayersView(with: cardFeatures) ! ... } }
  20. None
  21. None
  22. None
  23. Face Tracking using CoreImage open class CIFaceFeature : CIFeature {

    open var bounds: CGRect { get } open var trackingID: Int32 { get } ! ... }
  24. Face Tracking using CoreImage open class CIFaceFeature : CIFeature {

    open var bounds: CGRect { get } open var trackingID: Int32 { get } ! ... } Sample including anima.on: h2ps:/ /github.com/shingt/FaceTracker- sample (Instead of our product code)
  25. Summary • In real-*me object detec*on apps • Difficulty in

    giving feedback differs depending a situa*on • Tracking can improve feedback in some cases
  26. Thank you!