OpenTalks.AI - Борис Лесцов, Детектирование людей в толпе

Transcript

1. Person detection in crowds Boris Lestsov Mail.Ru Group
2. Computer Vision Team We solve computer vision problems at Mail.Ru Projects: 1) Vision (b2b) 2) Cloud 3) Mail 4) ...
3. Business case
4. Business case 1) Queue optimisation a) Open the elevator (ski resort) b) Call the cashier 2) Await time estimation
7. Business requirements 1) Works in various setups 2) Real time 3) Robust on video
8. Challenges
9. Challenges Heavy occlusion
10. Challenges Pose, illumination, clothing variability.
11. Metrics and datasets
12. Intersection Over Union (IoU) Measures detection quality for single bounding box.
13. Benchmarks CrowdHuman - our main dataset Custom dataset similar to target domain
14. CrowdHuman examples
15. CrowdHuman examples
16. CrowdHuman examples
17. Why create own solution? Existing solutions: • Not adapted to crowds • Slow inference
18. Approaches
19. Approaches 1) Classical CV (HOG, Deformable Part Models, ViolaJones) 2) Motion-based detection (background subtraction) 3) CNN: a) Two stage - Faster RCNN b) Single stage - SSD, YOLO, RetinaNet.
20. Faster RCNN
21. Faster RCNN
22. Faster RCNN
23. Faster RCNN
24. Faster RCNN
25. Faster RCNN
26. Faster RCNN Person?
27. Faster RCNN + Accurate + Bigger resolution => better result - Slow - More objects => more proposals => slower detection
28. Single Shot Detector
34. SSD (Single Shot Detector) Image
35. SSD (Single Shot Detector) Extract features
36. Single Shot Detector Reducing height & width => to detect at different scales
37. Different Scales Smaller scale Bigger scale
38. Single Shot Detector Predict bounding boxes
39. Single Shot Detector Merge similar detections with NMS
40. Non Maximum Suppression
41. Problems with SSD • Backbone - VGG-16 • 512x512 => breaking aspect ratio
42. Architecture: RetinaNet 1) Backbone - ResNet 2) Feature Pyramid Network (FPN) 3) FocalLoss against class disbalance
43. Feature Pyramid Network Higher level features for smaller scales
44. FocalLoss Problem: class disbalance 99 : 1 Cross Entropy (CE): Focal Loss (FL): p t - predicted probability of g.t. class:
45. Focal Loss 1) Well classified examples => smaller contribution 2) Analogue of Online Hard Example Mining
46. Other problems
47. Problems How many people on this image?
49. Small pedestrians Bigger resolution => better result, but slower. 800x600 : 30 fps, ~73.5% AP 1200x800: 15 fps, ~78.0% AP
50. Crop augmentations Good Crop Bad Crop Random Crop Random Crop
51. Crop augmentations Good Crop Bad Crop Random Crop Random Crop IoU is roughly the same!
52. Tuned prior boxes Removed horizontal boxes
53. Tuned prior boxes Smaller boxes
54. Detection examples
55. Prediction Examples
56. Prediction Examples
57. Failure Case Extra box between people
58. Future work 1) Repulsion Loss 2) RetinaMask 3) Replace ResNet-50 with a better backbone model.
59. Tracking
60. Blinking problem
61. Tracking use cases 1) Tracking itself 2) Less False Positives on a video stream. 3) Deal with “blinking” detections.
62. SORT (Simple Online and Realtime Tracking) • Association by IoU • Kalman Filters • Fast We fine-tuned SORT
63. With tracking
64. Tracking example
65. Queues in canteen 1) Multiple cameras 2) Zoning
66. Conclusion 1) Two stage detectors are more accurate, but slower 2) Bigger resolution => better accuracy, slower 3) ResNet, FPN, Focal Loss => better result
67. Thanks! Questions?
68. Appendix
69. Metrics: AP - Average Precision (single class) • False Positive (FP) - predicted bbox without IoU>0.5 with some g.t. • False Negative (FN) - g.t. bbox without IoU>0.5 with some predicted box. 1) Compute predictions. 2) Plot Precision-Recall Curve, make
70. Intuition about Kalman Filter in SORT Box is represented with vector: • u,v - coordinates of the center • s - box scale • r - box aspect ratio • dotted u, v, s - corresponding derivatives Notes: 1. Linear prediction from frame to frame with correction from detector output. 2. Generally can model broad range of dynamic systems (fluid in a tank, the temperature of a car engine).
71. Appendix Repulsion Loss Three components: 1) Attraction to matched g.t. box. 2) Repulsion from other g.t. boxes. 3) Repulsion from other predicted boxes. Technically, IoU is
72. RetinaMask 1) RetinaNet adapted to instance segmentation 2) Mask prediction gives good improvement in detection quality (~2.3% mAP on COCO). 3) Masks are predicted in Faster-RCNN manner. Mask prediction can be discarded during inference to speed up the detector. 4) Tune masks on COCO “Person” category, detection on CrowdHuman. 5) Code and models available!
76. Resolution 1) SGD training instead of Adam. 2) Replacing SSD with RetinaNet arch. 3) FocalLoss 4) Bigger resolution (current models: 800x600 and 1200x800) 5) scale_by_aspect instead of simple resize. 6) Anchor box tuning. 7) Crop augmentations 8) Joint training with head detection. 9) Removing strides from convolutions in last
77. Things that did NOT work out 1) MTCNN for detecting small people. 2) Prediction of full bounding box instead of the visible one.