OpenTalks.AI - Олег Карташев, Компьютерное зрение на Северстали. Как извлечь больше пользы из камер видеонаблюдения.

Transcript

1. Computer vision at Severstal. How to extract additional value from IP cameras 15 Feb. 2019 Oleg Kartashev os.kartashev@severstal.com
2. Any unauthorized copying, disclosure or distribution of the materials contained in this document (or its annexes) is strictly prohibited. 2 Who we are RnD Continuous research, assessment and adoption of new technologies Internal tech innovation consulting Digital trends Including ML-demanding: CV Virtual assistants / NLP … and all others in future Prototypes and pilots CV: 5 prototypes / 2 piloted NLP: 2 prototypes / 1 piloted Idea generation Idea generation processes together with company divisions and external vendors Knowledge renewal Knowledge renewal process in existing and emerging directions 12 28
3. Any unauthorized copying, disclosure or distribution of the materials contained in this document (or its annexes) is strictly prohibited. IP cameras === Video Surveillance ? Plate recognition People counting Tracking Line crossing
4. Any unauthorized copying, disclosure or distribution of the materials contained in this document (or its annexes) is strictly prohibited. 4 Common video analytics Mostly classic CV Most of the solutions use classic CV algorithms like: • blob detection • feature detection and matching • optical flow But even for non-classic: Huge datasets – Pretrained models – Large community - forked and tuned till 99+ % accuracy –
5. Any unauthorized copying, disclosure or distribution of the materials contained in this document (or its annexes) is strictly prohibited. 5 Starting a project Analyzing the task Analyzing the conditions Warming up Get the video Visit the site, analyze the technological process Specify business requirements Ask for economic effect Which cameras “see” the process? Choose a suitable one Is the camera used by another processes? Try setting up a dedicated stream Is the picture sufficient for the goal? Tune the stream settings Which stream settings can be changed? Choose a cameras with plenty of settings The camera can be moved? The zoom can be adjusted? The light can be adjusted? Is the shutter speed in sync? Camera calibration: Distance measurements Barrel-effect correction Collecting representative dataset Dataset annotation
6. Any unauthorized copying, disclosure or distribution of the materials contained in this document (or its annexes) is strictly prohibited. 6 Project: Controlling argon pipe presence
7. Any unauthorized copying, disclosure or distribution of the materials contained in this document (or its annexes) is strictly prohibited. 7 Challenges: Broad Variety 1. Make an ensemble of 2 NN with different architecture and train them on different datasets 2. Set higher (608x608) input size 3. Analyze more object classes: pipe, cone with pipe, cone without pipe 4. Choose confidence thresholds 5. Choose frequency thresholds in 10 seconds interval 6. Still didn’t find it? Be patient.
8. Any unauthorized copying, disclosure or distribution of the materials contained in this document (or its annexes) is strictly prohibited. 8 Challenges: Visibility 1. Be patient. No alerts while the confidence is low. 2. Dive into the technological process and analyze what’s going wrong: Darkness or the main pipe is getting hotter and camera is automatically adapting the exposure Attachment point is hidden by the ladle Smoke Camera displacement Protecting glass needs cleaning The process has changed 3. Check if something is visible: DLBP + SVM. 4. None of that? You got the alert! Profit!
9. Any unauthorized copying, disclosure or distribution of the materials contained in this document (or its annexes) is strictly prohibited. 1 systemd service / stream 9 Design .py OpenCV / ffmpeg Cisco VSM API . . . RTSP h264 YOLOv2 YOLOv3 DLBP + SVM 1 FPS 0 0 image / video logs training Linux VM with 16 vCPU
10. Any unauthorized copying, disclosure or distribution of the materials contained in this document (or its annexes) is strictly prohibited. 10 Measures and Results Measure, intuitive for customer, not for scientist FP measuring across several days is time consuming TN = A full minute in period of steel casting with pipe missing FN are undesirable. Negative predictive value: = + 01 Collecting true negatives: • Experiments: pull the pipe out, check the alert • Interval between casting  check the alert • Protection pipe is missing  check… 02 Core algorithms – C++ / Cython (OpenCV DNN, custom LBP, fast histograms) Interfaces – Python, can be further optimized if needed 04 False Negatives: 0 False Positives: irrelevant (because of visibility issues) True Negatives: sufficient for controlling the process NPV = 100%, now tuning postprocessing to catch more True Negatives. 03 Processing speed is sufficient: 16 vCPUs capable of 5 streams at > 1 FPS 05
11. Any unauthorized copying, disclosure or distribution of the materials contained in this document (or its annexes) is strictly prohibited. 11 More challenges: Lighting, compression and noise Low light visibility. Blurring, especially – on sheet edges (phantoms). Sheet speed – up to 6 m/s. 01 Flickering. Effect of a darker zone moving down through the frame. 02 Sensor & compression noise 03 Defect or noise? Find a shutter speed–motion blur compromise. Analyze a frame section with lowest effect influence Use lighting invariant features Build «clean sheet» model Analyze remaining texture Train a classifier
12. Any unauthorized copying, disclosure or distribution of the materials contained in this document (or its annexes) is strictly prohibited. 12 More challenges: Single camera positioning No triangulation 01 Measure distances to scene key points Height measuring 02 Measure distance from central point to camera and to the vertical axis Shoot a video and measure a reference object in several scene locations Image quality 03 Classic CV is insufficient Train a network to detect helmets and employees Place a marker on the crane hook
13. Any unauthorized copying, disclosure or distribution of the materials contained in this document (or its annexes) is strictly prohibited. 13 More challenges: dataset size and imbalanced classes Most important objects are less frequent 02 Make experiments with existing objects: Prepare several test wagons in which the objects can be mixed Objects can be rotated or occluded Distance from camera to the surface varies 01 Artificial augmentation. From framework-standard rotation, scale, blur, etc. To data simulation and generation.
14. Questions? And yes, we’re hiring Oleg Kartashev os.kartashev@severstal.com Technical architect