Upgrade to Pro — share decks privately, control downloads, hide ads and more …

MEMS-sensors in Computer Vision: we underestimate them Anastasiya Kornilova, SPbSU, CEE-SECR 2017

CEE-SECR
October 20, 2017

MEMS-sensors in Computer Vision: we underestimate them Anastasiya Kornilova, SPbSU, CEE-SECR 2017

A key challenge in such areas of computer vision as video stabilization, 3D-reconstruction, SLAM, VR, is to increase energy efficient and performance of algorithms being implemented. In most cases, a detection of camera movement and rotation is a significant consumer of a computational power. In this report, we discuss how to decrease significantly the number of calculations using MEMS motion sensors (gyroscope, accelerometer) in case of video stabilization task. We consider main difficulties that can be encountered when implementing these algorithms on specific platforms.

This topic would be interesting for specialists in the fields related to computer vision and digital signal processing. The main goal of the report is to present our ongoing research on digital video stabilization and to analyze its application in various fields.

CEE-SECR

October 20, 2017
Tweet

More Decks by CEE-SECR

Other Decks in Technology

Transcript

  1. MEMS-sensors
    in Computer Vision:
    we underestimate them
    Anastasiya KORNILOVA
    Iakov KIRILENKO
    SPBU, JetBrains Research
    Software Engineering Conference Russia
    October 2017, St. Petersburg

    View full-size slide

  2. Video stabilization
    ● Good frame quality
    ● Bad video quality
    ● Reason ー jitter
    2

    View full-size slide

  3. Existing solutions
    ● Mechanical devices
    ● OIS
    ● Digital stabilization
    3

    View full-size slide

  4. Digital stabilization
    4

    View full-size slide

  5. Disadvantages
    ● Based on frame scanning
    ● Slow processing
    ● Poor lightning
    ● Big moving objects
    5

    View full-size slide

  6. How to detect motion?
    Gyroscope
    Accelerometer
    6

    View full-size slide

  7. Using MEMS-sensors
    ● Instantly
    ● Power consumption ー 2-5 mW
    ● Widely used
    ○ Smartphones
    ○ Embedded systems
    7

    View full-size slide

  8. Camera rotation model
    8

    View full-size slide

  9. Camera rotation model
    9

    View full-size slide

  10. Camera rotation model
    10

    View full-size slide

  11. Depends on rotation
    Rotation
    matrix
    Gyro info
    11

    View full-size slide

  12. ● Angular velocity
    ● Integrate
    ○ Timestamps
    ● 1D-rotation
    Gyroscope model
    12

    View full-size slide

  13. Example
    http://bit.ly/kornilova_secr2017
    13

    View full-size slide

  14. More complex integration
    ● “Digital Video Stabilization and Rolling
    Shutter Correction using Gyroscopes“
    (2012, Standford)
    ● Quaternions
    ● 3D-rotation
    14

    View full-size slide

  15. 15
    Example
    http://bit.ly/kornilova_secr2017

    View full-size slide

  16. Rolling shutter defect
    16

    View full-size slide

  17. Rolling shutter model
    Row of point
    17

    View full-size slide

  18. Generalize
    Rotation
    matrix
    Rolling
    shutter
    parameter
    18

    View full-size slide

  19. Oops... Auto calibration
    Rotation
    matrix
    Rolling
    shutter
    parameter
    ● Focal length
    ● Camera and sensor
    displacement
    ● Synchronization sensor
    and camera
    ● Bias of gyroscope
    19

    View full-size slide

  20. Example. 20 usec!!!
    20
    http://bit.ly/kornilova_secr2017

    View full-size slide

  21. ● Real-time available
    ● Simple model, but many problems
    ● Codecs
    ● 3D-reconstruction
    Conclusion
    21

    View full-size slide

  22. Q&A
    Anastasiya Kornilova
    +7-921-575-21-62
    [email protected]
    Video stabilization
    using MEMS-sensors
    http://bit.ly/kornilova_secr2017

    View full-size slide

  23. Statistics
    ● Whatsapp: 250 million messages with video per day
    ● Periscope: 350 000 hours streamings per day
    ● Snapchat: 6 billion videos per day
    25

    View full-size slide

  24. Mobile video editors
    iMovie PowerDirector CameraPlus Pro Camera Genius
    4$ 3.5$
    3.5$ 3$
    30 млн 5 млн
    10 млн 3 млн
    26

    View full-size slide