Providing Be,er Feedback in Real-&me Object Detec&on Apps try! Swi) Tokyo 2017 Lightning Talk shingt (Shinichi Goto) @ Wantedly, Inc. ɹ 

Object Detec)on 

Wantedly People 

Goal and Problem 

Assump&on: We already have detec&on logic 

struct CardFeature { let coordinates: [CGPoint] ... } 

extension DetectionViewController: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput( _ captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, ...) { ... guard let frame = sampleBuffer.toUIImage() else { return } let cardFeatures = detector.cardFeatures(in: frame) updateCircleLayersView(with: cardFeatures) ... } } 

extension DetectionViewController: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput( _ captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, ...) { ... guard let frame = sampleBuffer.toUIImage() else { return } let cardFeatures = detector.cardFeatures(in: frame) ! // Detect cards updateCircleLayersView(with: cardFeatures) ... } } 

extension DetectionViewController: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput( _ captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, ...) { ... guard let frame = sampleBuffer.toUIImage() else { return } let cardFeatures = detector.cardFeatures(in: frame) updateCircleLayersView(with: cardFeatures) ! // Add CALayers ... } } 

No content

One object 

One object 

Mul$ple objects 

Mul$ple objects 

What we want 

Tracking • Assigning consistent labels to objects in videos • Generic trackers available in OpenCV • e.g. TLD, KCF, etc. • Performance issues on older iOS devices 

Tracking (our case) • Combina)on of lightweight informa)on: • dHash (perceptual hash based on gradient) • Posi)on • Op)cal flow 

struct CardFeature { let coordinates: [CGPoint] let trackingID: Int ! ... } 

extension DetectionViewController: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput( _ captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, ...) { ... guard let frame = sampleBuffer.toUIImage() else { return } let cardFeatures = detector.cardFeatures(in: frame) // // Now we can animate circle layers // from previous positions to current positions // updateCircleLayersView(with: cardFeatures) ! ... } } 

No content

No content

No content

Face Tracking using CoreImage open class CIFaceFeature : CIFeature { open var bounds: CGRect { get } open var trackingID: Int32 { get } ! ... } 

Face Tracking using CoreImage open class CIFaceFeature : CIFeature { open var bounds: CGRect { get } open var trackingID: Int32 { get } ! ... } Sample including anima.on: h2ps:/ /github.com/shingt/FaceTracker- sample (Instead of our product code) 

Summary • In real-*me object detec*on apps • Difficulty in giving feedback differs depending a situa*on • Tracking can improve feedback in some cases 

Thank you!