Robot Vision Systems with Mirror Reflections Kentaro Yoshioka1 2, Hidenori Okuni1, Tuan Thanh Ta1 and Akihide Sai1 1. Toshiba, Japan 2. Now Assistant Prof. at Keio University, Japan
challenge in robot vision – Danger of overlooking or even damaging the target – Occlusions are even more challenging when multiple robots work under the same environment Depth sensor Occluded! Boxes: Detection results of Mask-RCNN
challenge in robot vision – Danger of overlooking or even damaging the target – Occlusions are even more challenging when multiple robots work under the same environment Depth sensor Occluded! Boxes: Detection results of Mask-RCNN
between work-efficiency and sensor cost [Ref.] D. Holz, IROS 2015. [Ref.] A. Zeng, ICRA 2018. Installing sensor at End-effector Work-efficiency ☺ Cost Installing multiple sensors ☺ Work-efficiency Cost
c c Occluded! c c Tilt! Occlusions diminished Mirror • First robot vision system which diminishes occlusions by sensor tilting and mirror reflections – Objective: Eliminate occlusions under the robot arm – Non-line-of-sight (NLoS) sensing by mirror Direct sensing Reflection sensing
c c Occluded! c c Tilt! Occlusions diminished Mirror • First robot vision system which diminishes occlusions by sensor tilting and mirror reflections – Pros: Low-cost hardware (mirror+tilt-unit) – Pros: Tilting faster than end-effector configuration
depth data Reflection sensing depth data Occlusion detection Calculate tilt angle Reflection sensing + Tilt angle Virtual image correction Camera trans. Mirror Depth sensor Tilt unit c c
depth data Reflection sensing depth data Occlusion detection Calculate tilt angle Reflection sensing + Tilt angle Virtual image correction Camera trans. Mirror Detected occlusion area Depth sensor Tilt unit c c
height-thresholds – Simple, yet fast and effective rule-based approach. • DNN methods are more general but time-consuming – Sensing time = 𝟐 × 𝒕𝒔𝒆𝒏𝒔𝒐𝒓 + 𝒕𝒕𝒊𝒍𝒕 + 𝒕𝒅𝒆𝒕𝒆𝒄𝒕 Centroid Area[pix2] (-0.2, -0.1) 1400 (0.3, -0.1) 30 Occlusion centroid Sensor noise Segment area which exceeds height threshold Find contours and convert to centroid and area Largest area classified as occlusions Target Xt
height-thresholds – Simple, yet fast and effective rule-based approach. • DNN methods are more general but time-consuming – Sensing time = 𝟐 × 𝒕𝒔𝒆𝒏𝒔𝒐𝒓 + 𝒕𝒕𝒊𝒍𝒕 + 𝒕𝒅𝒆𝒕𝒆𝒄𝒕 Centroid Area[pix2] (-0.2, -0.1) 1400 (0.3, -0.1) 30 Occlusion centroid Sensor noise Segment area which exceeds height threshold Find contours and convert to centroid and area Largest area classified as occlusions Target Xt
on centroid of the occlusion area – 𝑿𝒕𝒗 = 𝑯𝒎 −𝟏𝑿𝒕 – 𝛉𝒕𝒊𝒍𝒕 = 𝐚𝐫𝐜𝐭𝐚𝐧(𝒚𝒕𝒗−𝒚𝒔 𝒙𝒕𝒗−𝒙𝒔 ) 𝑯𝒎 −𝟏 Householder transformation Mirror Depth sensor Tilt unit Xt “Virtual” Xtv c c qtilt
error is proportional to the working distance(WD) – Reflection sensing’s WD is × 𝟏 𝒄𝒐𝒔𝜽𝒕𝒊𝒍𝒕 longer than direct sensing – Error of our sensor (EnsensoN35) was proportional to the square of the distance. • e.g. at 𝜽𝒕𝒊𝒍𝒕=45deg., meas. error become x2 larger
the scene and decides scan angles adaptively. Sensor rearrangements are not required after changes in robot operation or layout. Adaptive sensing Fixed angles Speed: x2 Experiments
– Showed that occlusions can be adaptively eliminated with mirror reflections and sensor tilting. – Perceptions close to a two-sensor setup were obtained. • Future works – Adaptation to complex environments with multiple robots – Extension to multiple mirrors