Upgrade to Pro — share decks privately, control downloads, hide ads and more …

MEMS-sensors in Computer Vision: we underestimate them Anastasiya Kornilova, SPbSU, CEE-SECR 2017

CEE-SECR
October 20, 2017

MEMS-sensors in Computer Vision: we underestimate them Anastasiya Kornilova, SPbSU, CEE-SECR 2017

A key challenge in such areas of computer vision as video stabilization, 3D-reconstruction, SLAM, VR, is to increase energy efficient and performance of algorithms being implemented. In most cases, a detection of camera movement and rotation is a significant consumer of a computational power. In this report, we discuss how to decrease significantly the number of calculations using MEMS motion sensors (gyroscope, accelerometer) in case of video stabilization task. We consider main difficulties that can be encountered when implementing these algorithms on specific platforms.

This topic would be interesting for specialists in the fields related to computer vision and digital signal processing. The main goal of the report is to present our ongoing research on digital video stabilization and to analyze its application in various fields.

CEE-SECR

October 20, 2017
Tweet

More Decks by CEE-SECR

Other Decks in Technology

Transcript

  1. MEMS-sensors in Computer Vision: we underestimate them Anastasiya KORNILOVA Iakov

    KIRILENKO SPBU, JetBrains Research Software Engineering Conference Russia October 2017, St. Petersburg
  2. Video stabilization • Good frame quality • Bad video quality

    • Reason ー jitter 2
  3. Existing solutions • Mechanical devices • OIS • Digital stabilization

    3
  4. Digital stabilization 4

  5. Disadvantages • Based on frame scanning • Slow processing •

    Poor lightning • Big moving objects 5
  6. How to detect motion? Gyroscope Accelerometer 6

  7. Using MEMS-sensors • Instantly • Power consumption ー 2-5 mW

    • Widely used ◦ Smartphones ◦ Embedded systems 7
  8. Camera rotation model 8

  9. Camera rotation model 9

  10. Camera rotation model 10

  11. Depends on rotation Rotation matrix Gyro info 11

  12. • Angular velocity • Integrate ◦ Timestamps • 1D-rotation Gyroscope

    model 12
  13. Example http://bit.ly/kornilova_secr2017 13

  14. More complex integration • “Digital Video Stabilization and Rolling Shutter

    Correction using Gyroscopes“ (2012, Standford) • Quaternions • 3D-rotation 14
  15. 15 Example http://bit.ly/kornilova_secr2017

  16. Rolling shutter defect 16

  17. Rolling shutter model Row of point 17

  18. Generalize Rotation matrix Rolling shutter parameter 18

  19. Oops... Auto calibration Rotation matrix Rolling shutter parameter • Focal

    length • Camera and sensor displacement • Synchronization sensor and camera • Bias of gyroscope 19
  20. Example. 20 usec!!! 20 http://bit.ly/kornilova_secr2017

  21. • Real-time available • Simple model, but many problems •

    Codecs • 3D-reconstruction Conclusion 21
  22. Q&A Anastasiya Kornilova +7-921-575-21-62 [email protected] Video stabilization using MEMS-sensors http://bit.ly/kornilova_secr2017

  23. Appendix

  24. None
  25. Statistics • Whatsapp: 250 million messages with video per day

    • Periscope: 350 000 hours streamings per day • Snapchat: 6 billion videos per day 25
  26. Mobile video editors iMovie PowerDirector CameraPlus Pro Camera Genius 4$

    3.5$ 3.5$ 3$ 30 млн 5 млн 10 млн 3 млн 26