[IROS23]Multimodal Diffusion Segmentation Model for Object Segmentation from Manipulation Instructions

Transcript

1. Multimodal Diffusion Segmentation Model for Object Segmentation from Manipulation Instructions Keio University, Japan Yui Iioka, Yu Yoshida, Yuiga Wada, Shumpei Hatanaka and Komei Sugiura
2. Background: Aiming for enhancement in human-robot interaction • Motivation – Understand human instructions ➢ Manipulate target objects • Useful – Pixel-wise segmentation masks ➢ Identify their shapes & locations 2 8x
3. Target Task: Object Segmentation from Manipulation Instructions（OSMI） Difference from Referring Expression Segmentation – Need to Identify the target object from long instruction sentences 3 "Go to the living room and fetch the pillow closest to the radio art on the wall." Inputs Output Dataset Sentence length (avg.) G-Ref [Mao+, CVPR16] 8.4 SHIMRIE (Ours) 18.8
4. Proposed Method: Multimodal Diffusion Segmentation Model (MDSM) • Main Novelty – Refine coarse masks by the extended diffusion model 4 • Generate more appropriate masks – w/o under- or over-segmentation
5. Proposed Method: The structure of MDSM 5 2nd-stage: Mask Refinement 1st-stage: Coarse Mask Generation
6. Proposed Method: The structure of MDSM 6 2nd-stage: Mask Refinement 1st-stage: Coarse Mask Generation
7. Proposed Method: The structure of MDSM 7 2nd-stage: Mask Refinement 1st-stage: Coarse Mask Generation Visual features extracted by denoising with DDPM [Ho+, NeurIPS20]
8. Quantitative results: Outperformed the existing method 8 [%] Method mIoU oIoU P@0.5 (i) LAVT [Yang+, CVPR22] 24.27±3.15 22.25±2.85 21.27±5.66 (ii) Ours (w/o 2nd-stage) 33.03±5.51 30.25±4.91 32.76±5.28 (iii) Ours 36.15±5.95 33.18±5.12 36.63±6.92 + 11.88 • Train models on our SHIMRIE dataset (train : valid : test = 10153：856：362)
9. Qualitative results: Generate appropriate masks from complex sentences 9 "Go to the laundry room and straighten the picture closest to the light switch." Baseline
10. Qualitative results: Generate appropriate masks from complex sentences 10 "Go to the lounge and remove the small brown chair facing the counter." Baseline
11. Summary: MDSM • Background – Object manipulation from a human instruction • Our method: MDSM – 1st-stage: Introduce Multimodal Encoder – 2nd-stage: Refine the mask by the diffusion model • Result – Outperformed the existing method on the SHIMRIE dataset • generate more appropriate masks from long sentences 11
12. Appendix: Difference between OSMI and RES • Sentence – RES: ``the candle on the right’’ – OSMI: ``Go to the dining table. Then pick up the candle on the right.’’ 14 Even the latest segmentation model, SEEM [Zou+, 23], is difficult to address OSMI tasks e.g. ``Pick up the plant in front of the mirror.’’
13. Appendix: SHIMRIE dataset 15 REVERIE [Qi+, CVPR20]: Instructions, bboxes & real-world images Matterport3D [Chang+, 3DV17]: Voxel-wise segmentation & simulator images Some instructions
14. Appendix: PWAM 16 [Yang+, CVPR22]
15. Appendix: Ablation Study 17 mIoU oIoU P@0.5 P@0.6 P@0.7 P@0.8 P@0.9 ✔ ✔ ✔ ✔ 34.09±4.14 31.57±3.07 35.52±6.04 27.29±4.93 16.35±3.16 6.35±1.22 1.82±1.33 ✔ ✔ ✔ 34.40±3.79 31.59±3.03 36.63±6.14 27.79±5.28 16.30±2.98 6.41±1.19 0.66±0.62 ✔ ✔ ✔ 33.68±4.37 30.61±4.17 35.80±6.73 26.46±4.66 15.69±3.88 6.08±1.58 0.50±0.41 ✔ ✔ ✔ 33.44±4.52 30.51±3.89 35.03±6.36 26.85±6.02 15.91±4.36 5.25±0.82 1.66±1.48 ✔ ✔ ✔ 32.54±4.97 29.97±4.14 35.30±6.72 26.24±6.26 13.15±3.77 3.26±1.95 0.50±0.41
16. Appendix: Failure cases • ``Empty the tissue box in the bathroom on level one.‘’ 18 • ``Visit the bathroom and bring me the picture nearest the toilet.‘’
17. Appendix: Error Analysis 19 Errors Description #Error SC Serious comprehension errors for handling visual and language information 11 RE Reference/Exophora resolution errors for linguistic information 31 SEO Segmentation of extra objects 19 OUS Over- or under-segmentation 16 NSG No segmentation in any region of the image 11 SNI Segmentation of non-target objects in the instruction 6 AE Annotation errors in the ground-truth mask or instruction 6 Total - 100
18. Appendix: Compared to w/o diffusion step 20