Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Providing Better Feedback in Real-time Object Detection Apps

Providing Better Feedback in Real-time Object Detection Apps

lightning talk at #tryswiftconf

Shinichi Goto

March 02, 2017
Tweet

More Decks by Shinichi Goto

Other Decks in Technology

Transcript

  1. Providing Be,er Feedback in
    Real-&me Object Detec&on Apps
    try! Swi) Tokyo 2017 Lightning Talk
    shingt (Shinichi Goto) @ Wantedly, Inc.
    ɹ

    View full-size slide

  2. Object Detec)on

    View full-size slide

  3. Wantedly People

    View full-size slide

  4. Goal and Problem

    View full-size slide

  5. Assump&on: We already have detec&on logic

    View full-size slide

  6. struct CardFeature {
    let coordinates: [CGPoint]
    ...
    }

    View full-size slide

  7. extension DetectionViewController: AVCaptureVideoDataOutputSampleBufferDelegate {
    func captureOutput(
    _ captureOutput: AVCaptureOutput!,
    didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, ...) {
    ...
    guard let frame = sampleBuffer.toUIImage() else { return }
    let cardFeatures = detector.cardFeatures(in: frame)
    updateCircleLayersView(with: cardFeatures)
    ...
    }
    }

    View full-size slide

  8. extension DetectionViewController: AVCaptureVideoDataOutputSampleBufferDelegate {
    func captureOutput(
    _ captureOutput: AVCaptureOutput!,
    didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, ...) {
    ...
    guard let frame = sampleBuffer.toUIImage() else { return }
    let cardFeatures = detector.cardFeatures(in: frame) ! // Detect cards
    updateCircleLayersView(with: cardFeatures)
    ...
    }
    }

    View full-size slide

  9. extension DetectionViewController: AVCaptureVideoDataOutputSampleBufferDelegate {
    func captureOutput(
    _ captureOutput: AVCaptureOutput!,
    didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, ...) {
    ...
    guard let frame = sampleBuffer.toUIImage() else { return }
    let cardFeatures = detector.cardFeatures(in: frame)
    updateCircleLayersView(with: cardFeatures) ! // Add CALayers
    ...
    }
    }

    View full-size slide

  10. Mul$ple objects

    View full-size slide

  11. Mul$ple objects

    View full-size slide

  12. What we want

    View full-size slide

  13. Tracking
    • Assigning consistent labels to objects in videos
    • Generic trackers available in OpenCV
    • e.g. TLD, KCF, etc.
    • Performance issues on older iOS devices

    View full-size slide

  14. Tracking (our case)
    • Combina)on of lightweight informa)on:
    • dHash (perceptual hash based on gradient)
    • Posi)on
    • Op)cal flow

    View full-size slide

  15. struct CardFeature {
    let coordinates: [CGPoint]
    let trackingID: Int !
    ...
    }

    View full-size slide

  16. extension DetectionViewController: AVCaptureVideoDataOutputSampleBufferDelegate {
    func captureOutput(
    _ captureOutput: AVCaptureOutput!,
    didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, ...) {
    ...
    guard let frame = sampleBuffer.toUIImage() else { return }
    let cardFeatures = detector.cardFeatures(in: frame)
    //
    // Now we can animate circle layers
    // from previous positions to current positions
    //
    updateCircleLayersView(with: cardFeatures) !
    ...
    }
    }

    View full-size slide

  17. Face Tracking using CoreImage
    open class CIFaceFeature : CIFeature {
    open var bounds: CGRect { get }
    open var trackingID: Int32 { get } !
    ...
    }

    View full-size slide

  18. Face Tracking using CoreImage
    open class CIFaceFeature : CIFeature {
    open var bounds: CGRect { get }
    open var trackingID: Int32 { get } !
    ...
    }
    Sample including anima.on: h2ps:/
    /github.com/shingt/FaceTracker-
    sample
    (Instead of our product code)

    View full-size slide

  19. Summary
    • In real-*me object detec*on apps
    • Difficulty in giving feedback differs depending a situa*on
    • Tracking can improve feedback in some cases

    View full-size slide