ICIP 2015 - Depth map inpainting

Transcript

1. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Pierre Buyssens, Maxime Daisy, David Tschumperlé, Olivier Lézoray GREYC, Image Team, University of Caen, France Sept. 30 2015 1/24
2. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Introduction & problem statement 1 Introduction & problem statement 2 Proposed method Superpixel rendering Source planes search Infering depth values 3 Results & comparisons 4 Conclusion and actual work 2/24
3. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Introduction & problem statement left view right view 3/24
4. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Introduction & problem statement right view 4/24
5. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Introduction & problem statement synthesized view original view 5/24
6. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Introduction & problem statement synthesized view (Js) original view (Jo) 6/24
7. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Introduction & problem statement Horizontal warping ⇒ a scanline method should be sufficient. Really ? Image continuous by parts ⇒ a diffusion method should solve the problem. Really ? 7/24
8. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Introduction & problem statement Is Js 8/24
9. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Proposed method 1 Introduction & problem statement 2 Proposed method Superpixel rendering Source planes search Infering depth values 3 Results & comparisons 4 Conclusion and actual work 9/24
10. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Proposed method Main idea (suggested in [8]) : use the original view Jo to infer correct values in the synthesized view Js synthesized view (Js) original view (Jo) Overview of the proposed approach : 1 Oversegment Jo and Js into superpixels (with a matching between them), 2 Given a pixel to inpaint, retain good superpixels and model them by planes, 3 Inpaint with a linear combination of the planes. [8] Yoon et al. Interview consistent hole filling in view extrapolation for multi-view image generation, ICIP, 2014. 10/24
11. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Proposed method Superpixel rendering Idea : oversegment the original (Jo) and synthesized (Js) views into superpixels, s.t. there is a matching between superpixels of Jo and Js 11/24
12. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Proposed method Superpixel rendering Idea : oversegment the original (Jo) and synthesized (Js) views into superpixels, s.t. there is a matching between superpixels of Jo and Js 1 Oversegment the original view into superpixels with the Eikonal Region Growing Clustering algorithm [9], results in a dense over-segmentation of Jo that respect object boundaries. [9] Eikonal-based Region Growing for efficient Clustering, Image and Vision Computing, 2014. 11/24
13. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Proposed method Superpixel rendering Idea : oversegment the original (Jo) and synthesized (Js) views into superpixels, s.t. there is a matching between superpixels of Jo and Js 1 Oversegment the original view into superpixels with the Eikonal Region Growing Clustering algorithm [9], results in a dense over-segmentation of Jo that respect object boundaries. 2 Warp the resulting label map results in the oversegmentation of the synthesized view Js, with a matching between the superpixel labels. [9] Eikonal-based Region Growing for efficient Clustering, Image and Vision Computing, 2014. 11/24
14. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Proposed method Superpixel rendering Original view Synthesized view 12/24
15. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Proposed method Source planes search Key ideas : • Use the original depth map Jo to infer good depth values of the missing pixels, • Use the previous superpixels matching for more robustness. 13/24
16. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Proposed method Source planes search Key ideas : • Use the original depth map Jo to infer good depth values of the missing pixels, • Use the previous superpixels matching for more robustness. How to find the good superpixel(s) ? 13/24
17. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Proposed method Source planes search Key ideas : • Use the original depth map Jo to infer good depth values of the missing pixels, • Use the previous superpixels matching for more robustness. How to find the good superpixel(s) ? Proposed method : Given a pixel p to inpaint 1 Find the closest known pixel q1 ∈ Js (belonging to the superpixel Rs 1 ), and the one q2 on the opposite side of the hole (belonging to the superpixel Rs 2 ), Js Jo Js Jo 13/24
18. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Proposed method Source planes search Key ideas : • Use the original depth map Jo to infer good depth values of the missing pixels, • Use the previous superpixels matching for more robustness. How to find the good superpixel(s) ? Proposed method : Given a pixel p to inpaint 1 Find the closest known pixel q1 ∈ Js (belonging to the superpixel Rs 1 ), and the one q2 on the opposite side of the hole (belonging to the superpixel Rs 2 ), 2 Get the offsets ∆p 1 = p −q1 and ∆p 2 = p −q2 , Js Jo Js Jo 13/24
19. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Proposed method Source planes search Key ideas : • Use the original depth map Jo to infer good depth values of the missing pixels, • Use the previous superpixels matching for more robustness. How to find the good superpixel(s) ? Proposed method : Given a pixel p to inpaint 1 Find the closest known pixel q1 ∈ Js (belonging to the superpixel Rs 1 ), and the one q2 on the opposite side of the hole (belonging to the superpixel Rs 2 ), 2 Get the offsets ∆p 1 = p −q1 and ∆p 2 = p −q2 , 3 Report q1 and q2 on Jo by inverse warping (constraint : warped pixels belong to Ro 1 and Ro 2 respectively), Js Jo Js Jo 13/24
20. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Proposed method Source planes search Key ideas : • Use the original depth map Jo to infer good depth values of the missing pixels, • Use the previous superpixels matching for more robustness. How to find the good superpixel(s) ? Proposed method : Given a pixel p to inpaint 1 Find the closest known pixel q1 ∈ Js (belonging to the superpixel Rs 1 ), and the one q2 on the opposite side of the hole (belonging to the superpixel Rs 2 ), 2 Get the offsets ∆p 1 = p −q1 and ∆p 2 = p −q2 , 3 Report q1 and q2 on Jo by inverse warping (constraint : warped pixels belong to Ro 1 and Ro 2 respectively), 4 From these two points, get the superpixel(s) pointed by −∆p 1 and −∆p 2 respectively. Js Jo Js Jo 13/24
21. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Proposed method Infering depth values Inpainting a given pixel p : • From the set of superpixels found, retain only those with biggest depth/lowest discrepancy (background), • Model them with planes by regression, • Inpaint p with a linear combination of the planes (if many planes are retained). Js Jo Js Jo Superpixel retained : green Superpixel retained : light blue 14/24
22. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Results & comparisons 1 Introduction & problem statement 2 Proposed method Superpixel rendering Source planes search Infering depth values 3 Results & comparisons 4 Conclusion and actual work 15/24
23. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Results & comparisons 16/24
24. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Results & comparisons Original image 17/24
25. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Results & comparisons Synthesized image 17/24
26. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Results & comparisons Synthesized image in false color 17/24
27. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Results & comparisons Result with [5] [5] Ndjiki-Nya et al., Depth image-based rendering with advanced texture synthesis for 3-d video, Transactions on Multimedia, 2011. 17/24
28. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Results & comparisons Result with with [8] [8] Yoon et al., Interview consistent hole filling in view extrapolation for multi-view image generation, ICIP, 2014. 17/24
29. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Results & comparisons Our result 17/24
30. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Results & comparisons 18/24
31. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Results & comparisons 19/24
32. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Results & comparisons Hole Result with [5] Result with [8] Our result 20/24
33. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Results & comparisons Hole Result with [5] Result with [8] Our result 21/24
34. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Conclusion and actual work 1 Introduction & problem statement 2 Proposed method Superpixel rendering Source planes search Infering depth values 3 Results & comparisons 4 Conclusion and actual work 22/24
35. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Conclusion and actual work Summary : • Specialized depth map inpainting algorithm • Use of superpixels : Algorithm more robust Modelization by planes Further improvements : • Better recovery of the structures (patches) • Parallelization Adirondack Backpack Bicycle1 Cable Classroom1 Couch Flowers Jadeplant Mask Motorcycle Piano Pipes Playroom Playtable Recycle Shelves Shopvac Sticks Storage Sword1 Sword2 Umbrella Vintage Mean errors Yoon Our 0 2 4 6 8 10 0 0.5 1 1.5 2 Median errors Quantitative comparisons on the Middlebury-2014 dataset. 23/24
36. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Conclusion and actual work Depth-Aware Patch-based Image Disocclusion for Virtual View Synthesis, Siggraph Asia, 2015. 24/24
37. Superpixel-Based depth Map Inpainting for RGB-D View Synthesis Conclusion and actual work Depth-Aware Patch-based Image Disocclusion for Virtual View Synthesis, Siggraph Asia, 2015. 24/24