1 of 57 Through the Looking Glass: Diminishing Occlusions in Robot Vision Systems with Mirror Reflections Kentaro Yoshioka1 2, Hidenori Okuni1, Tuan Thanh Ta1 and Akihide Sai1 1. Toshiba, Japan 2. Now Assistant Prof. at Keio University, Japan

2 of 57 Motivation • Occlusion stand as a large challenge in robot vision – Danger of overlooking or even damaging the target – Occlusions are even more challenging when multiple robots work under the same environment Depth sensor Occluded! Boxes: Detection results of Mask-RCNN

3 of 57 Motivation • Occlusion stand as a large challenge in robot vision – Danger of overlooking or even damaging the target – Occlusions are even more challenging when multiple robots work under the same environment Depth sensor Occluded! Boxes: Detection results of Mask-RCNN

4 of 57 Reducing occlusions in robot vision systems [Ref.] D. Holz, IROS 2015. [Ref.] A. Zeng, ICRA 2018. Installing sensor at End-effector  Work-efficiency ☺ Cost Installing multiple sensors ☺ Work-efficiency  Cost

5 of 57 Reducing occlusions in robot vision systems Trade-off between work-efficiency and sensor cost [Ref.] D. Holz, IROS 2015. [Ref.] A. Zeng, ICRA 2018. Installing sensor at End-effector  Work-efficiency ☺ Cost Installing multiple sensors ☺ Work-efficiency  Cost

6 of 57 Tilt-type reflection sensing Depth sensor Tilt unit c c Occluded! c c Tilt! Occlusions diminished Mirror • First robot vision system which diminishes occlusions by sensor tilting and mirror reflections – Objective: Eliminate occlusions under the robot arm – Non-line-of-sight (NLoS) sensing by mirror Direct sensing Reflection sensing

7 of 57 Tilt-type reflection sensing Depth sensor Tilt unit c c Occluded! c c Tilt! Occlusions diminished Mirror • First robot vision system which diminishes occlusions by sensor tilting and mirror reflections – Pros: Low-cost hardware (mirror+tilt-unit) – Pros: Tilting faster than end-effector configuration

8 of 57 Tilt-type reflection sensing Direct sensing Direct sensing depth data Reflection sensing depth data Occlusion detection Calculate tilt angle Reflection sensing + Tilt angle Virtual image correction Camera trans. Mirror Depth sensor Tilt unit c c

9 of 57 Tilt-type reflection sensing Direct sensing Direct sensing depth data Reflection sensing depth data Occlusion detection Calculate tilt angle Reflection sensing + Tilt angle Virtual image correction Camera trans. Mirror Depth sensor Tilt unit c c Tomato missing !

10 of 57 Tilt-type reflection sensing Direct sensing Direct sensing depth data Reflection sensing depth data Occlusion detection Calculate tilt angle Reflection sensing + Tilt angle Virtual image correction Camera trans. Mirror Detected occlusion area Depth sensor Tilt unit c c

11 of 57 Tilt-type reflection sensing Direct sensing Direct sensing depth data Reflection sensing depth data Occlusion detection Calculate tilt angle Reflection sensing + Tilt angle Virtual image correction Camera trans. Mirror Detected occlusion area Target Xt “Virtual” target Xtv Depth sensor Tilt unit c c 𝑯𝒎 −𝟏

12 of 57 Tilt-type reflection sensing Direct sensing Direct sensing depth data Reflection sensing depth data Occlusion detection Calculate tilt angle Reflection sensing + Tilt angle Virtual image correction Camera trans. Mirror Depth sensor Tilt unit Detected occlusion area Target Xt “Virtual” target Xtv c c qtilt 𝑯𝒎 −𝟏

13 of 57 Tilt-type reflection sensing Direct sensing Direct sensing depth data Reflection sensing depth data Occlusion detection Calculate tilt angle Reflection sensing + Tilt angle Virtual image correction Camera trans. Mirror Depth sensor Tilt unit Detected occlusion area Target Xt “Virtual” target Xtv c c qtilt Occlusion areas detected on-the-fly Adaptively remove occlusions 𝑯𝒎 −𝟏

14 of 57 Tilt-type reflection sensing Direct sensing Direct sensing depth data Reflection sensing depth data Occlusion detection Calculate tilt angle Reflection sensing + Virtual image correction Camera trans. Mirror Depth sensor Tilt unit Detected occlusion area Target Xt “Virtual” target Xtv c c qtilt Concatenated results Tomato recovered ! Occlusion areas detected on-the-fly Adaptively remove occlusions 𝑯𝒎 −𝟏

15 of 57 Occlusion detection • Occlusions detected based on height-thresholds – Simple, yet fast and effective rule-based approach. • DNN methods are more general but time-consuming – Sensing time = 𝟐 × 𝒕𝒔𝒆𝒏𝒔𝒐𝒓 + 𝒕𝒕𝒊𝒍𝒕 + 𝒕𝒅𝒆𝒕𝒆𝒄𝒕 Centroid Area[pix2] (-0.2, -0.1) 1400 (0.3, -0.1) 30 Occlusion centroid Sensor noise Segment area which exceeds height threshold Find contours and convert to centroid and area Largest area classified as occlusions Target Xt

16 of 57 Occlusion detection • Occlusions detected based on height-thresholds – Simple, yet fast and effective rule-based approach. • DNN methods are more general but time-consuming – Sensing time = 𝟐 × 𝒕𝒔𝒆𝒏𝒔𝒐𝒓 + 𝒕𝒕𝒊𝒍𝒕 + 𝒕𝒅𝒆𝒕𝒆𝒄𝒕 Centroid Area[pix2] (-0.2, -0.1) 1400 (0.3, -0.1) 30 Occlusion centroid Sensor noise Segment area which exceeds height threshold Find contours and convert to centroid and area Largest area classified as occlusions Target Xt

17 of 57 Calculating optimal θtilt • θtilt calculated based on centroid of the occlusion area – 𝑿𝒕𝒗 = 𝑯𝒎 −𝟏𝑿𝒕 – 𝛉𝒕𝒊𝒍𝒕 = 𝐚𝐫𝐜𝐭𝐚𝐧(𝒚𝒕𝒗−𝒚𝒔 𝒙𝒕𝒗−𝒙𝒔 ) Mirror Depth sensor Tilt unit Xt “Virtual” Xtv c c qtilt 𝑯𝒎 −𝟏

18 of 57 Calculating optimal θtilt • θtilt calculated based on centroid of the occlusion area – 𝑿𝒕𝒗 = 𝑯𝒎 −𝟏𝑿𝒕 – 𝛉𝒕𝒊𝒍𝒕 = 𝐚𝐫𝐜𝐭𝐚𝐧(𝒚𝒕𝒗−𝒚𝒔 𝒙𝒕𝒗−𝒙𝒔 ) 𝑯𝒎 −𝟏 Householder transformation Mirror Depth sensor Tilt unit Xt “Virtual” Xtv c c qtilt

19 of 57 Meas. errors for reflection sensing • Meas. error is proportional to the working distance(WD) – Reflection sensing’s WD is × 𝟏 𝒄𝒐𝒔𝜽𝒕𝒊𝒍𝒕 longer than direct sensing – Error of our sensor (EnsensoN35) was proportional to the square of the distance. • e.g. at 𝜽𝒕𝒊𝒍𝒕=45deg., meas. error become x2 larger

20 of 57 Experiment Setup Depth sensor (ENSENSO N35) Tilt unit (FLIR PUT-E46) Off-the-shelf Mirror Robot arm (DENSO COBOTTA)

21 of 57 Two sensing schemes are rapidly switched by tilting the zenith sensor. Occlusions are greatly reduced, which boosts detection accuracy. Direct only Direct + Reflection Speed: x2 Experiments

22 of 57 Reflection sensing results are almost equal to results of two sensors. The evaluated mAP was almost the same (1% difference). Results with direct + reflection Results with Two sensors Experiments

23 of 57 Our proposed system detects the occlusion in the scene and decides scan angles adaptively. Sensor rearrangements are not required after changes in robot operation or layout. Adaptive sensing Fixed angles Speed: x2 Experiments

24 of 57 Conclusion • Proposed Tilt-type mirror reflection sensing – Showed that occlusions can be adaptively eliminated with mirror reflections and sensor tilting. – Perceptions close to a two-sensor setup were obtained. • Future works – Adaptation to complex environments with multiple robots – Extension to multiple mirrors