Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Through the Looking Glass: Diminishing Occlusions in Robot Vision Systems with Mirror Reflections

Through the Looking Glass: Diminishing Occlusions in Robot Vision Systems with Mirror Reflections

IROS 2021

9cc4ca0b6f51673c096e1588cab68832?s=128

Kentaro Yoshioka

October 26, 2021
Tweet

More Decks by Kentaro Yoshioka

Other Decks in Research

Transcript

  1. 1 of 57 Through the Looking Glass: Diminishing Occlusions in

    Robot Vision Systems with Mirror Reflections Kentaro Yoshioka1 2, Hidenori Okuni1, Tuan Thanh Ta1 and Akihide Sai1 1. Toshiba, Japan 2. Now Assistant Prof. at Keio University, Japan
  2. 2 of 57 Motivation • Occlusion stand as a large

    challenge in robot vision – Danger of overlooking or even damaging the target – Occlusions are even more challenging when multiple robots work under the same environment Depth sensor Occluded! Boxes: Detection results of Mask-RCNN
  3. 3 of 57 Motivation • Occlusion stand as a large

    challenge in robot vision – Danger of overlooking or even damaging the target – Occlusions are even more challenging when multiple robots work under the same environment Depth sensor Occluded! Boxes: Detection results of Mask-RCNN
  4. 4 of 57 Reducing occlusions in robot vision systems [Ref.]

    D. Holz, IROS 2015. [Ref.] A. Zeng, ICRA 2018. Installing sensor at End-effector  Work-efficiency ☺ Cost Installing multiple sensors ☺ Work-efficiency  Cost
  5. 5 of 57 Reducing occlusions in robot vision systems Trade-off

    between work-efficiency and sensor cost [Ref.] D. Holz, IROS 2015. [Ref.] A. Zeng, ICRA 2018. Installing sensor at End-effector  Work-efficiency ☺ Cost Installing multiple sensors ☺ Work-efficiency  Cost
  6. 6 of 57 Tilt-type reflection sensing Depth sensor Tilt unit

    c c Occluded! c c Tilt! Occlusions diminished Mirror • First robot vision system which diminishes occlusions by sensor tilting and mirror reflections – Objective: Eliminate occlusions under the robot arm – Non-line-of-sight (NLoS) sensing by mirror Direct sensing Reflection sensing
  7. 7 of 57 Tilt-type reflection sensing Depth sensor Tilt unit

    c c Occluded! c c Tilt! Occlusions diminished Mirror • First robot vision system which diminishes occlusions by sensor tilting and mirror reflections – Pros: Low-cost hardware (mirror+tilt-unit) – Pros: Tilting faster than end-effector configuration
  8. 8 of 57 Tilt-type reflection sensing Direct sensing Direct sensing

    depth data Reflection sensing depth data Occlusion detection Calculate tilt angle Reflection sensing + Tilt angle Virtual image correction Camera trans. Mirror Depth sensor Tilt unit c c
  9. 9 of 57 Tilt-type reflection sensing Direct sensing Direct sensing

    depth data Reflection sensing depth data Occlusion detection Calculate tilt angle Reflection sensing + Tilt angle Virtual image correction Camera trans. Mirror Depth sensor Tilt unit c c Tomato missing !
  10. 10 of 57 Tilt-type reflection sensing Direct sensing Direct sensing

    depth data Reflection sensing depth data Occlusion detection Calculate tilt angle Reflection sensing + Tilt angle Virtual image correction Camera trans. Mirror Detected occlusion area Depth sensor Tilt unit c c
  11. 11 of 57 Tilt-type reflection sensing Direct sensing Direct sensing

    depth data Reflection sensing depth data Occlusion detection Calculate tilt angle Reflection sensing + Tilt angle Virtual image correction Camera trans. Mirror Detected occlusion area Target Xt “Virtual” target Xtv Depth sensor Tilt unit c c 𝑯𝒎 −𝟏
  12. 12 of 57 Tilt-type reflection sensing Direct sensing Direct sensing

    depth data Reflection sensing depth data Occlusion detection Calculate tilt angle Reflection sensing + Tilt angle Virtual image correction Camera trans. Mirror Depth sensor Tilt unit Detected occlusion area Target Xt “Virtual” target Xtv c c qtilt 𝑯𝒎 −𝟏
  13. 13 of 57 Tilt-type reflection sensing Direct sensing Direct sensing

    depth data Reflection sensing depth data Occlusion detection Calculate tilt angle Reflection sensing + Tilt angle Virtual image correction Camera trans. Mirror Depth sensor Tilt unit Detected occlusion area Target Xt “Virtual” target Xtv c c qtilt Occlusion areas detected on-the-fly Adaptively remove occlusions 𝑯𝒎 −𝟏
  14. 14 of 57 Tilt-type reflection sensing Direct sensing Direct sensing

    depth data Reflection sensing depth data Occlusion detection Calculate tilt angle Reflection sensing + Virtual image correction Camera trans. Mirror Depth sensor Tilt unit Detected occlusion area Target Xt “Virtual” target Xtv c c qtilt Concatenated results Tomato recovered ! Occlusion areas detected on-the-fly Adaptively remove occlusions 𝑯𝒎 −𝟏
  15. 15 of 57 Occlusion detection • Occlusions detected based on

    height-thresholds – Simple, yet fast and effective rule-based approach. • DNN methods are more general but time-consuming – Sensing time = 𝟐 × 𝒕𝒔𝒆𝒏𝒔𝒐𝒓 + 𝒕𝒕𝒊𝒍𝒕 + 𝒕𝒅𝒆𝒕𝒆𝒄𝒕 Centroid Area[pix2] (-0.2, -0.1) 1400 (0.3, -0.1) 30 Occlusion centroid Sensor noise Segment area which exceeds height threshold Find contours and convert to centroid and area Largest area classified as occlusions Target Xt
  16. 16 of 57 Occlusion detection • Occlusions detected based on

    height-thresholds – Simple, yet fast and effective rule-based approach. • DNN methods are more general but time-consuming – Sensing time = 𝟐 × 𝒕𝒔𝒆𝒏𝒔𝒐𝒓 + 𝒕𝒕𝒊𝒍𝒕 + 𝒕𝒅𝒆𝒕𝒆𝒄𝒕 Centroid Area[pix2] (-0.2, -0.1) 1400 (0.3, -0.1) 30 Occlusion centroid Sensor noise Segment area which exceeds height threshold Find contours and convert to centroid and area Largest area classified as occlusions Target Xt
  17. 17 of 57 Calculating optimal θtilt • θtilt calculated based

    on centroid of the occlusion area – 𝑿𝒕𝒗 = 𝑯𝒎 −𝟏𝑿𝒕 – 𝛉𝒕𝒊𝒍𝒕 = 𝐚𝐫𝐜𝐭𝐚𝐧(𝒚𝒕𝒗−𝒚𝒔 𝒙𝒕𝒗−𝒙𝒔 ) Mirror Depth sensor Tilt unit Xt “Virtual” Xtv c c qtilt 𝑯𝒎 −𝟏
  18. 18 of 57 Calculating optimal θtilt • θtilt calculated based

    on centroid of the occlusion area – 𝑿𝒕𝒗 = 𝑯𝒎 −𝟏𝑿𝒕 – 𝛉𝒕𝒊𝒍𝒕 = 𝐚𝐫𝐜𝐭𝐚𝐧(𝒚𝒕𝒗−𝒚𝒔 𝒙𝒕𝒗−𝒙𝒔 ) 𝑯𝒎 −𝟏 Householder transformation Mirror Depth sensor Tilt unit Xt “Virtual” Xtv c c qtilt
  19. 19 of 57 Meas. errors for reflection sensing • Meas.

    error is proportional to the working distance(WD) – Reflection sensing’s WD is × 𝟏 𝒄𝒐𝒔𝜽𝒕𝒊𝒍𝒕 longer than direct sensing – Error of our sensor (EnsensoN35) was proportional to the square of the distance. • e.g. at 𝜽𝒕𝒊𝒍𝒕=45deg., meas. error become x2 larger
  20. 20 of 57 Experiment Setup Depth sensor (ENSENSO N35) Tilt

    unit (FLIR PUT-E46) Off-the-shelf Mirror Robot arm (DENSO COBOTTA)
  21. 21 of 57 Two sensing schemes are rapidly switched by

    tilting the zenith sensor. Occlusions are greatly reduced, which boosts detection accuracy. Direct only Direct + Reflection Speed: x2 Experiments
  22. 22 of 57 Reflection sensing results are almost equal to

    results of two sensors. The evaluated mAP was almost the same (1% difference). Results with direct + reflection Results with Two sensors Experiments
  23. 23 of 57 Our proposed system detects the occlusion in

    the scene and decides scan angles adaptively. Sensor rearrangements are not required after changes in robot operation or layout. Adaptive sensing Fixed angles Speed: x2 Experiments
  24. 24 of 57 Conclusion • Proposed Tilt-type mirror reflection sensing

    – Showed that occlusions can be adaptively eliminated with mirror reflections and sensor tilting. – Perceptions close to a two-sensor setup were obtained. • Future works – Adaptation to complex environments with multiple robots – Extension to multiple mirrors