SIGGRAPH ASIA 2015 - 2

Transcript

1. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Pierre Buyssens, Maxime Daisy, David Tschumperlé, Olivier Lézoray GREYC, Image Team, University of Caen, France Nov. 2 2015
2. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Introduction & Background 1 Introduction & Background Context Sketch of Patch-based inpainting algorithm 2 Revisiting Patch-based Inpainting Depth-aware priority term Depth-aware search scheme Depth-aware patch copy Dealing with object aliasing 3 Results & comparisons
3. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Introduction & Background Context left view right view
4. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Introduction & Background Context right view
5. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Introduction & Background Context synthesized color image original view synthesized depth original depth
6. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Introduction & Background Context synthesized color image (Is) original view (Io) synthesized depth (inpainted) (Js)
7. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Introduction & Background Sketch of Patch-based inpainting algorithm Main iteration of the Patch-based inpainting algorithm of Criminisi et al. : 1 Priority term P(p) = C(p)×D(p)∀ pixel p of the hole boundary δΩ where D the data term : filling priority guided by the image edges, and C the confidence term : count the reliable data around p 2 Select the patch Ψt around the pixel t with maximal priority 3 Search in ¯ Ω the patch Ψˆ t that minimizes Ψˆ t = Ψp | argmin p∈Np∩(I−Ω) dSSD (Ψt ,Ψp ) (1) 4 Paste values from Ψˆ t around t in Ω : Ψt (q) = Ψˆ t (p) | q ∈ Nt ∩Ω (2) 5 Update confidences and data term values [1] Region filling and object removal by exemplar-based image inpainting, Trans. on Image Processing, 2004.
8. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Revisiting Patch-based Inpainting 1 Introduction & Background Context Sketch of Patch-based inpainting algorithm 2 Revisiting Patch-based Inpainting Depth-aware priority term Depth-aware search scheme Depth-aware patch copy Dealing with object aliasing 3 Results & comparisons
9. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Revisiting Patch-based Inpainting This work • assumes that Js is already inpainted with a dedicated method (such as [*]), • adds depth information to each key step of the Criminisi et al. algorithm, • requires only one additional parameter λ. The additional parameter λ : • is a thresholding parameter, • discriminates foreground (FG) from background (BG) objects, • depends on the depth image values. λ parameter Two adjacent pixels p and q belong to the same object (FG or BG) if |depth(p)−depth(q)| < λ [*] Superpixel-based depth map inpainting for RGB-D view synthesis, ICIP, 2015.
10. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Revisiting Patch-based Inpainting Depth-aware priority term Incorporating depth into the priority term. In the litterature : complicated and sensitive modifications • [1] : P(p) = (C(p)+D(p)+L(p))×(Znear − ¯ Zp ) • [2] : P(p) = D(p)α ×C(p)β ×E(p)γ In this work : P(p) = C(p)×D(p) • Keep the data term D(p) unchanged • Count as reliable data (i.e. confidence term C(p)) only pixels with equal depth (w.r.t. λ) C(p) = 1 |Np | ∑ q∈Np∩(Is−Ω) |Js(p)−Js(q)|<λ C(q) Hole interior boundaries data term confidence term priority term [1] Reel et al., Joint texture-depth pixel inpainting of disocclusion holes in virtual view synthesis, APSIPA, 2013. [2] Yoon et al., Inter-view consistent hole filling in view extrapolation for multi-view image generation, ICIP, 2014.
11. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Revisiting Patch-based Inpainting Depth-aware search scheme Incorporating depth into the search scheme. In the litterature : • Search Ψˆ t in the synthesized image Is, • [2] : Look for patches with similar depth. In this work : • Search Ψˆ t in the original image Io, • Look for patches with equal depth (w.r.t. λ) Ψˆ t = Ψp ∈ Io | argmin |Jo(p)−Js(t)|<λ dSSD (Ψt ,Ψp ) (3) Is Io [2] Yoon et al., Inter-view consistent hole filling in view extrapolation for multi-view image generation, ICIP, 2014.
12. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Revisiting Patch-based Inpainting Depth-aware patch copy Incorporating depth into the copy scheme. In the litterature : • No depth-aware copy scheme proposed, • Classical copy used Ψt (q) = Ψˆ t (p) | q ∈ Nt ∩Ω In this work : • Paste only pixels with equal depth (w.r.t. λ) Ψt (q) = Ψˆ t (p) q ∈ Nt ∩Ω |Jo (p)−Js (q)| < λ • Avoid copying BG pixels into FG (and vice versa). Is Ψt in Is Ψt in Js Ψˆ t (p) in Io Result
13. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Revisiting Patch-based Inpainting Dealing with object aliasing Important problem : object aliasing • The color of a FG object bleeds into the BG Excerpt of an image FG object removed (w.r.t. its depth) If not taken into account :
14. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Revisiting Patch-based Inpainting Dealing with object aliasing Simple and effective proposed solution (1/2) • Compute the so-called narrow bands NBo and NBs NBo = T (δ(Jo )−Jo ,λ) NBs = T (δ(Js )−Js ,λ) with T the thresholding operator, δ(Jo ) (resp. δ(Jo )) is the dilated depth map of Jo (resp. Js). NBs on Is NBo on Io
15. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Revisiting Patch-based Inpainting Dealing with object aliasing Simple and effective proposed solution (2/2) Given a target pixel t : • If t / ∈ NBs, search of Ψˆ t restricted to Ψp ∈ Io s.t. Ψp ∩NBo = ∅ Avoid picking a patch that contains narrow band pixels • If t ∈ NBs, search of Ψˆ t in Io without restrictions Allow the copy of narrow band pixels on the boundary of the objects Hole Result w/o the narrow bands process Result with the narrow bands process
16. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Results & comparisons 1 Introduction & Background Context Sketch of Patch-based inpainting algorithm 2 Revisiting Patch-based Inpainting Depth-aware priority term Depth-aware search scheme Depth-aware patch copy Dealing with object aliasing 3 Results & comparisons
17. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Results & comparisons
18. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Results & comparisons
19. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Results & comparisons
20. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Results & comparisons
21. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Results & comparisons
22. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Results & comparisons
23. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Results & comparisons Comparisons with • Gautier et al., Depth-based image completion for view synthesis, 3DTV-CON, 2011. • Yoon et al., Inter-view consistent hole filling in view extrapolation for multi-view image generation, ICIP, 2014.
24. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Results & comparisons Original image.
25. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Results & comparisons Synthesized image.
26. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Results & comparisons Result with Gautier et al..
27. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Results & comparisons Result with Yoon et al..
28. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Results & comparisons Our result.
29. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Results & comparisons Result with Gautier et al. Result with Yoon et al. Our result
30. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Results & comparisons Original image.
31. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Results & comparisons Synthesized image.
32. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Results & comparisons Result with Gautier et al..
33. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Results & comparisons Result with Yoon et al..
34. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Results & comparisons Our result.
35. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Results & comparisons Result with Gautier et al. Result with Yoon et al. Our result
36. Depth-aware Patch-based Image Disocclusion for Virtual View Synthesis Results & comparisons Thank you for your attention