Providing Better Feedback in Real-time Object Detection Apps

lightning talk at #tryswiftconf

Transcript

1. Providing Be,er Feedback in
   Real-&me Object Detec&on Apps
   try! Swi) Tokyo 2017 Lightning Talk
   shingt (Shinichi Goto) @ Wantedly, Inc.
   ɹ
   

   View Slide

2. Object Detec)on
   

   View Slide

3. Wantedly People
   

   View Slide

4. Goal and Problem
   

   View Slide

5. Assump&on: We already have detec&on logic
   

   View Slide

6. struct CardFeature {
   let coordinates: [CGPoint]
   ...
   }
   

   View Slide

7. extension DetectionViewController: AVCaptureVideoDataOutputSampleBufferDelegate {
   func captureOutput(
   _ captureOutput: AVCaptureOutput!,
   didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, ...) {
   ...
   guard let frame = sampleBuffer.toUIImage() else { return }
   let cardFeatures = detector.cardFeatures(in: frame)
   updateCircleLayersView(with: cardFeatures)
   ...
   }
   }
   

   View Slide

8. extension DetectionViewController: AVCaptureVideoDataOutputSampleBufferDelegate {
   func captureOutput(
   _ captureOutput: AVCaptureOutput!,
   didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, ...) {
   ...
   guard let frame = sampleBuffer.toUIImage() else { return }
   let cardFeatures = detector.cardFeatures(in: frame) ! // Detect cards
   updateCircleLayersView(with: cardFeatures)
   ...
   }
   }
   

   View Slide

9. extension DetectionViewController: AVCaptureVideoDataOutputSampleBufferDelegate {
   func captureOutput(
   _ captureOutput: AVCaptureOutput!,
   didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, ...) {
   ...
   guard let frame = sampleBuffer.toUIImage() else { return }
   let cardFeatures = detector.cardFeatures(in: frame)
   updateCircleLayersView(with: cardFeatures) ! // Add CALayers
   ...
   }
   }
   

   View Slide

10. View Slide

11. One object
    

    View Slide

12. One object
    

    View Slide

13. Mul$ple objects
    

    View Slide

14. Mul$ple objects
    

    View Slide

15. What we want
    

    View Slide

16. Tracking
    • Assigning consistent labels to objects in videos
    • Generic trackers available in OpenCV
    • e.g. TLD, KCF, etc.
    • Performance issues on older iOS devices
    

    View Slide

17. Tracking (our case)
    • Combina)on of lightweight informa)on:
    • dHash (perceptual hash based on gradient)
    • Posi)on
    • Op)cal flow
    

    View Slide

18. struct CardFeature {
    let coordinates: [CGPoint]
    let trackingID: Int !
    ...
    }
    

    View Slide

19. extension DetectionViewController: AVCaptureVideoDataOutputSampleBufferDelegate {
    func captureOutput(
    _ captureOutput: AVCaptureOutput!,
    didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, ...) {
    ...
    guard let frame = sampleBuffer.toUIImage() else { return }
    let cardFeatures = detector.cardFeatures(in: frame)
    //
    // Now we can animate circle layers
    // from previous positions to current positions
    //
    updateCircleLayersView(with: cardFeatures) !
    ...
    }
    }
    

    View Slide

20. View Slide

21. View Slide

22. View Slide

23. Face Tracking using CoreImage
    open class CIFaceFeature : CIFeature {
    open var bounds: CGRect { get }
    open var trackingID: Int32 { get } !
    ...
    }
    

    View Slide

24. Face Tracking using CoreImage
    open class CIFaceFeature : CIFeature {
    open var bounds: CGRect { get }
    open var trackingID: Int32 { get } !
    ...
    }
    Sample including anima.on: h2ps:/
    /github.com/shingt/FaceTracker-
    sample
    (Instead of our product code)
    

    View Slide

25. Summary
    • In real-*me object detec*on apps
    • Difficulty in giving feedback differs depending a situa*on
    • Tracking can improve feedback in some cases
    

    View Slide

26. Thank you!
    

    View Slide