Providing Better Feedback in Real-time Object Detection Apps

Providing Better Feedback in Real-time Object Detection Apps

lightning talk at #tryswiftconf

3652e140db258288623a0adf248baec0?s=128

Shinichi Goto

March 02, 2017
Tweet

Transcript

  1. 1.

    Providing Be,er Feedback in Real-&me Object Detec&on Apps try! Swi)

    Tokyo 2017 Lightning Talk shingt (Shinichi Goto) @ Wantedly, Inc. ɹ
  2. 7.

    extension DetectionViewController: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput( _ captureOutput: AVCaptureOutput!, didOutputSampleBuffer

    sampleBuffer: CMSampleBuffer!, ...) { ... guard let frame = sampleBuffer.toUIImage() else { return } let cardFeatures = detector.cardFeatures(in: frame) updateCircleLayersView(with: cardFeatures) ... } }
  3. 8.

    extension DetectionViewController: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput( _ captureOutput: AVCaptureOutput!, didOutputSampleBuffer

    sampleBuffer: CMSampleBuffer!, ...) { ... guard let frame = sampleBuffer.toUIImage() else { return } let cardFeatures = detector.cardFeatures(in: frame) ! // Detect cards updateCircleLayersView(with: cardFeatures) ... } }
  4. 9.

    extension DetectionViewController: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput( _ captureOutput: AVCaptureOutput!, didOutputSampleBuffer

    sampleBuffer: CMSampleBuffer!, ...) { ... guard let frame = sampleBuffer.toUIImage() else { return } let cardFeatures = detector.cardFeatures(in: frame) updateCircleLayersView(with: cardFeatures) ! // Add CALayers ... } }
  5. 10.
  6. 16.

    Tracking • Assigning consistent labels to objects in videos •

    Generic trackers available in OpenCV • e.g. TLD, KCF, etc. • Performance issues on older iOS devices
  7. 17.

    Tracking (our case) • Combina)on of lightweight informa)on: • dHash

    (perceptual hash based on gradient) • Posi)on • Op)cal flow
  8. 19.

    extension DetectionViewController: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput( _ captureOutput: AVCaptureOutput!, didOutputSampleBuffer

    sampleBuffer: CMSampleBuffer!, ...) { ... guard let frame = sampleBuffer.toUIImage() else { return } let cardFeatures = detector.cardFeatures(in: frame) // // Now we can animate circle layers // from previous positions to current positions // updateCircleLayersView(with: cardFeatures) ! ... } }
  9. 20.
  10. 21.
  11. 22.
  12. 23.

    Face Tracking using CoreImage open class CIFaceFeature : CIFeature {

    open var bounds: CGRect { get } open var trackingID: Int32 { get } ! ... }
  13. 24.

    Face Tracking using CoreImage open class CIFaceFeature : CIFeature {

    open var bounds: CGRect { get } open var trackingID: Int32 { get } ! ... } Sample including anima.on: h2ps:/ /github.com/shingt/FaceTracker- sample (Instead of our product code)
  14. 25.

    Summary • In real-*me object detec*on apps • Difficulty in

    giving feedback differs depending a situa*on • Tracking can improve feedback in some cases