MEMS-sensors in Computer Vision: we underestimate them Anastasiya Kornilova, SPbSU, CEE-SECR 2017

A key challenge in such areas of computer vision as video stabilization, 3D-reconstruction, SLAM, VR, is to increase energy efficient and performance of algorithms being implemented. In most cases, a detection of camera movement and rotation is a significant consumer of a computational power. In this report, we discuss how to decrease significantly the number of calculations using MEMS motion sensors (gyroscope, accelerometer) in case of video stabilization task. We consider main difficulties that can be encountered when implementing these algorithms on specific platforms.

This topic would be interesting for specialists in the fields related to computer vision and digital signal processing. The main goal of the report is to present our ongoing research on digital video stabilization and to analyze its application in various fields.

Transcript

1. MEMS-sensors in Computer Vision: we underestimate them Anastasiya KORNILOVA Iakov KIRILENKO SPBU, JetBrains Research Software Engineering Conference Russia October 2017, St. Petersburg
2. Video stabilization • Good frame quality • Bad video quality • Reason ー jitter 2
3. Existing solutions • Mechanical devices • OIS • Digital stabilization 3
4. Digital stabilization 4
5. Disadvantages • Based on frame scanning • Slow processing • Poor lightning • Big moving objects 5
6. How to detect motion? Gyroscope Accelerometer 6
7. Using MEMS-sensors • Instantly • Power consumption ー 2-5 mW • Widely used ◦ Smartphones ◦ Embedded systems 7
8. Camera rotation model 8
9. Camera rotation model 9
10. Camera rotation model 10
11. Depends on rotation Rotation matrix Gyro info 11
12. • Angular velocity • Integrate ◦ Timestamps • 1D-rotation Gyroscope model 12
13. Example http://bit.ly/kornilova_secr2017 13
14. More complex integration • “Digital Video Stabilization and Rolling Shutter Correction using Gyroscopes“ (2012, Standford) • Quaternions • 3D-rotation 14
15. 15 Example http://bit.ly/kornilova_secr2017
16. Rolling shutter defect 16
17. Rolling shutter model Row of point 17
18. Generalize Rotation matrix Rolling shutter parameter 18
19. Oops... Auto calibration Rotation matrix Rolling shutter parameter • Focal length • Camera and sensor displacement • Synchronization sensor and camera • Bias of gyroscope 19
20. Example. 20 usec!!! 20 http://bit.ly/kornilova_secr2017
21. • Real-time available • Simple model, but many problems • Codecs • 3D-reconstruction Conclusion 21
22. Q&A Anastasiya Kornilova +7-921-575-21-62 kornilova.anastasiia@gmail.com Video stabilization using MEMS-sensors http://bit.ly/kornilova_secr2017
23. Appendix
25. Statistics • Whatsapp: 250 million messages with video per day • Periscope: 350 000 hours streamings per day • Snapchat: 6 billion videos per day 25
26. Mobile video editors iMovie PowerDirector CameraPlus Pro Camera Genius 4$ 3.5$ 3.5$ 3$ 30 млн 5 млн 10 млн 3 млн 26