ICIAE 2017

Transcript

1. Improvements of Exemplar-based Image Inpainting Olivier Lézoray with Maxime Daisy, Pierre Buyssens, David Tschumperlé Normandie Univ, UNICAEN, ENSICAEN, CNRS, GREYC, 14000 Caen, France ICIAE 2017 March 28, 2017
2. Overview 1 Introduction 2 Inpainting approaches 3 Improvements of Exemplar-based inpainting 4 Geometry guided patch blending for artifact reduction 5 Extension to other modalities Video inpainting Reconstruction of re-synthetized stereoscopic views Olivier Lézoray ICIAE 2017 March 28, 2017 2 / 57
3. Outline 1 Introduction 2 Inpainting approaches 3 Improvements of Exemplar-based inpainting 4 Geometry guided patch blending for artifact reduction 5 Extension to other modalities Olivier Lézoray ICIAE 2017 March 28, 2017 3 / 57
4. What is inpainting ? Olivier Lézoray ICIAE 2017 March 28, 2017 4 / 57
5. What is inpainting ? Olivier Lézoray ICIAE 2017 March 28, 2017 4 / 57
6. What is inpainting ? Olivier Lézoray ICIAE 2017 March 28, 2017 4 / 57
7. Why is it a hard problem ? Olivier Lézoray ICIAE 2017 March 28, 2017 5 / 57
8. Why is it a hard problem ? Olivier Lézoray ICIAE 2017 March 28, 2017 5 / 57
9. Why is it a hard problem ? Olivier Lézoray ICIAE 2017 March 28, 2017 5 / 57
10. How to do it manually ? Use the clone tool Olivier Lézoray ICIAE 2017 March 28, 2017 6 / 57
11. How to do it manually ? Olivier Lézoray ICIAE 2017 March 28, 2017 6 / 57
12. How to do it manually ? Olivier Lézoray ICIAE 2017 March 28, 2017 6 / 57
13. How to do it manually ? Olivier Lézoray ICIAE 2017 March 28, 2017 6 / 57
14. How to do it manually ? Olivier Lézoray ICIAE 2017 March 28, 2017 6 / 57
15. How to do it manually ? Olivier Lézoray ICIAE 2017 March 28, 2017 6 / 57
16. Outline 1 Introduction 2 Inpainting approaches 3 Improvements of Exemplar-based inpainting 4 Geometry guided patch blending for artifact reduction 5 Extension to other modalities Olivier Lézoray ICIAE 2017 March 28, 2017 7 / 57
17. Geometry-based approaches Purely geometry-driven methods [Masnou et al. ’98, Bertalmio et al. ’00, Chan et al. ’01, Tschumperlé and Deriche ’05, . . .] Idea : Use a diffusion equation guided by the isophotes directions → performs boundary geometry interpolation • Good global geometry reconstruction • Expressed with an energy functional to minimize • Not able to create complex textures • Time consuming Olivier Lézoray ICIAE 2017 March 28, 2017 8 / 57
18. Masked Image. Olivier Lézoray ICIAE 2017 March 28, 2017 9 / 57
19. Inpainted with [Tschumperlé and Deriche ’05] Olivier Lézoray ICIAE 2017 March 28, 2017 9 / 57
20. Exemplar-based approaches Pattern-based methods [Efros and Leung’99, Criminisi et al. ’04, Lemeur et al. ’11, . . .] Idea : exploit texture auto-similarity and local analysis of image geometry → copy/paste of image chunks • Large area reconstruction • Quite fast • Not enough local/global coherence e.g. block-effect artifacts ⇒ Best compromise between time and quality Olivier Lézoray ICIAE 2017 March 28, 2017 10 / 57
21. Exemplar-based approaches Criminisi et al. 2004, "Region Filling and Object Removal by Exemplar-Based Image Inpainting" C : Confidence term (reliable information for reach patch) D : Data term (local structure estimation) Olivier Lézoray ICIAE 2017 March 28, 2017 11 / 57
22. Exemplar-based approaches Criminisi et al. 2004, "Region Filling and Object Removal by Exemplar-Based Image Inpainting" C : Confidence term (reliable information for reach patch) D : Data term (local structure estimation) Olivier Lézoray ICIAE 2017 March 28, 2017 11 / 57
23. Exemplar-based approaches Criminisi et al. 2004, "Region Filling and Object Removal by Exemplar-Based Image Inpainting" C : Confidence term (reliable information for reach patch) D : Data term (local structure estimation) Olivier Lézoray ICIAE 2017 March 28, 2017 11 / 57
24. Masked Image. Olivier Lézoray ICIAE 2017 March 28, 2017 12 / 57
25. Inpainted with [Criminisi et al. ’04] Olivier Lézoray ICIAE 2017 March 28, 2017 12 / 57
26. Fast patch search PatchMatch [Barnes et al’09] • A very fast algorithm for matching similar patchs between two images • Works for full images (non-masked patchs) and is stochastic Used in several efficient image and video inpainting techniques (multi-scale approaches) : • [Wexler et al. ’07] + PatchMatch : available in PhotoShop • The solution at one scale is used as an initialization for the upper scale (may increase error propagation) Olivier Lézoray ICIAE 2017 March 28, 2017 13 / 57
27. [Wexler et al. ’07] Inpainted with [Wexler et al. ’07] Olivier Lézoray ICIAE 2017 March 28, 2017 14 / 57
28. Key-steps for improvements Two key steps can be considered for improvements : • The analysis of the image geometry • The analysis of the best patches to copy/paste Olivier Lézoray ICIAE 2017 March 28, 2017 15 / 57
29. Key-steps for improvements Two key steps can be considered for improvements : • The analysis of the image geometry • The analysis of the best patches to copy/paste • Problems with curved structures to reconstruct Olivier Lézoray ICIAE 2017 March 28, 2017 15 / 57
30. Key-steps for improvements Two key steps can be considered for improvements : • The analysis of the image geometry • The analysis of the best patches to copy/paste • Problems with curved structures to reconstruct • Visible bloc effects Olivier Lézoray ICIAE 2017 March 28, 2017 15 / 57
31. Key-steps for improvements Two key steps can be considered for improvements : • The analysis of the image geometry • The analysis of the best patches to copy/paste • Problems with curved structures to reconstruct • Visible bloc effects Olivier Lézoray ICIAE 2017 March 28, 2017 15 / 57
32. Outline 1 Introduction 2 Inpainting approaches 3 Improvements of Exemplar-based inpainting 4 Geometry guided patch blending for artifact reduction 5 Extension to other modalities Olivier Lézoray ICIAE 2017 March 28, 2017 16 / 57
33. Exemplar-based Inpainting Algorithm improvements Proposed improvements 1 priority accuracy enhancement ⇒ more global geometry consistency 2 better lookup strategy ⇒ more local geometry consistency 3 spatial patch blending ⇒ strongly reduced block-effect artifacts Olivier Lézoray ICIAE 2017 March 28, 2017 17 / 57
34. A better data term accuracy Criminisi et al. data term Dp = | − → ∇Ip ⊥ . − → np| α (1) Dp favors the reconstruction of local structures that are orthogonal to the mask "The gradient ∇Ip is computed as the maximum value of the image gradient in Ψp ∩ I." − → ∇I⊥ p = { − → ∇I⊥ q | arg max q∈((I−Ω)∩ψp) − → ∇Iq } (isophote direction) (2) Olivier Lézoray ICIAE 2017 March 28, 2017 18 / 57
35. A better data term accuracy Criminisi et al. data term Dp = | − → ∇Ip ⊥ . − → np| α (1) Dp favors the reconstruction of local structures that are orthogonal to the mask "The gradient ∇Ip is computed as the maximum value of the image gradient in Ψp ∩ I." − → ∇I⊥ p = { − → ∇I⊥ q | arg max q∈((I−Ω)∩ψp) − → ∇Iq } (isophote direction) (2) Olivier Lézoray ICIAE 2017 March 28, 2017 18 / 57
36. Structure tensors Structure tensors provide a description of the local geometry for vector images : Sp = N k=1 − → ∇Ik (p) − → ∇Ik (p)T (3) Structure tensors are symmetric and positive matrices (2 × 2) that can be decomposed with their eigen decomposition in Sp = N k=1 λk ek eT k (4) Tensor is said anisotropic when λ1 λ2 Olivier Lézoray ICIAE 2017 March 28, 2017 19 / 57
37. A better data term accuracy Data term based on the smoothed structure tensor field ˜ Dp = Gp . − → n p (5) with Gp = q∈ψp∩(I−Ω) wp (q) − → ∇Iq − → ∇IT q (6) wp : normalized 2d gaussian centered at p High data term for anisotropic tensors Olivier Lézoray ICIAE 2017 March 28, 2017 20 / 57
38. A better data term accuracy Data term based on the smoothed structure tensor field ˜ Dp = Gp . − → n p (5) with Gp = q∈ψp∩(I−Ω) wp (q) − → ∇Iq − → ∇IT q (6) wp : normalized 2d gaussian centered at p High data term for anisotropic tensors Olivier Lézoray ICIAE 2017 March 28, 2017 20 / 57
39. Example Original image Olivier Lézoray ICIAE 2017 March 28, 2017 21 / 57
40. Example Original data term from [Criminisi et al. ’03] Olivier Lézoray ICIAE 2017 March 28, 2017 21 / 57
41. Example Our data term Olivier Lézoray ICIAE 2017 March 28, 2017 21 / 57
42. Example Inpainting result with original data term from [Criminisi et al. ’03] Olivier Lézoray ICIAE 2017 March 28, 2017 21 / 57
43. Example Inpainting result with our data term Olivier Lézoray ICIAE 2017 March 28, 2017 21 / 57
44. A better lookup strategy Criminisi et al. 2004 lookup strategy • window search ? ⇒ faster ⇒ much less global • all over the image search ? ⇒ slow ⇒ not so much good results Olivier Lézoray ICIAE 2017 March 28, 2017 22 / 57
45. A better lookup strategy Window search Iteration 0 : the number of available patches is sufficient to find a good matching Olivier Lézoray ICIAE 2017 March 28, 2017 23 / 57
46. A better lookup strategy Window search Iteration n : the number of available patches is not sufficient to find a good matching Olivier Lézoray ICIAE 2017 March 28, 2017 23 / 57
47. A better lookup strategy Our lookup strategy • inspired from [Ashikhmin ’01, PatchMatch ’09] • local/global search scheme • use search sites of surrounding inpainted patches • search site sizes are inversely proportional to the number of sites ⇒ more local coherence Olivier Lézoray ICIAE 2017 March 28, 2017 24 / 57
48. A better lookup strategy • The set of already pasted patches around one pixel is determined (if no : use classical scheme) • Search windows are added around original pasted patches • most search sites overlap, and matching is faster • patch size becomes less critical Olivier Lézoray ICIAE 2017 March 28, 2017 25 / 57
49. Joshua Tree - 512 × 384 Masked Image Olivier Lézoray ICIAE 2017 March 28, 2017 26 / 57
50. Joshua Tree - 512 × 384 Windowed search (4.5s) Olivier Lézoray ICIAE 2017 March 28, 2017 26 / 57
51. Joshua Tree - 512 × 384 With smart search (3.2s) Olivier Lézoray ICIAE 2017 March 28, 2017 26 / 57
52. Fake Owl - 800 × 450 Masked Image Olivier Lézoray ICIAE 2017 March 28, 2017 27 / 57
53. Fake Owl - 800 × 450 Windowed search (1.7s) Olivier Lézoray ICIAE 2017 March 28, 2017 27 / 57
54. Fake Owl - 800 × 450 With smart search (0.8s) Olivier Lézoray ICIAE 2017 March 28, 2017 27 / 57
55. block-effect artifacts ? Olivier Lézoray ICIAE 2017 March 28, 2017 28 / 57
56. Outline 1 Introduction 2 Inpainting approaches 3 Improvements of Exemplar-based inpainting 4 Geometry guided patch blending for artifact reduction 5 Extension to other modalities Olivier Lézoray ICIAE 2017 March 28, 2017 29 / 57
57. Spatial Patch Blending Masked Image. Olivier Lézoray ICIAE 2017 March 28, 2017 30 / 57
58. Spatial Patch Blending Inpainted with [Criminisi et al. ’04] Olivier Lézoray ICIAE 2017 March 28, 2017 30 / 57
59. Spatial Patch Blending Inpainted with [Criminisi et al. ’04] + Our spatial patch blending Olivier Lézoray ICIAE 2017 March 28, 2017 30 / 57
60. Patch Blending : method Idea : Visually minimize block artifacts due to side-by-side pasted patches. • Remember the location of the pasted patches • Detect the locations of the artifacts • Spatial blending of the patches Olivier Lézoray ICIAE 2017 March 28, 2017 31 / 57
61. Artifact Detection Masked image. Olivier Lézoray ICIAE 2017 March 28, 2017 32 / 57
62. Artifact Detection Inpainted image. Olivier Lézoray ICIAE 2017 March 28, 2017 32 / 57
63. Artifact Detection Break field image. Olivier Lézoray ICIAE 2017 March 28, 2017 32 / 57
64. Artifact Detection Break points. Olivier Lézoray ICIAE 2017 March 28, 2017 32 / 57
65. Artifact Detection Blending amplitude map. Olivier Lézoray ICIAE 2017 March 28, 2017 32 / 57
66. Artifact Detection 1 Map of the artifact location probabilities 2 hypothesis for artifact locations : PA = ∇I . div(φ) • local sharp variations in the inpainted image (high gradient) • source patches come from very different locations (discontinuities in the shift map φ) 2 Strongest artifact locations LA = {p | PA(p) > τ} 3 Blending amplitude map σ(p) = q∈LA w(p,q) max r∈I q∈LA w(p,q) with w(p, q) = exp − p−q 2 PA(q)2 (7) Olivier Lézoray ICIAE 2017 March 28, 2017 33 / 57
67. Patch Blending Spatial blending of all the patches that overlap at a given pixel p Here : compute a weighted average of p1, p2 and p3 Weights depend on distance between patch centers and pixel to blend. Olivier Lézoray ICIAE 2017 March 28, 2017 34 / 57
68. Patch Blending Weighted sum of overlapping patches pixels values J(p) = ψq∈Ψp w(p, q) . ψq (p − q) ψq∈Ψp w(q, p) (8) • w(p, q) = exp − p−q 2 σ(p)2 • Ψp = {ψq | ψq ∩ ψp = ∅} (set of patches that overlap at p) → one gaussian function for each possible blending amplitude ⇒ quite slow (convolution with a spatially variant kernel) ⇒ use a quantization of σ to make it fast Olivier Lézoray ICIAE 2017 March 28, 2017 35 / 57
69. Result Masked image. Olivier Lézoray ICIAE 2017 March 28, 2017 36 / 57
70. Result Inpainted image Olivier Lézoray ICIAE 2017 March 28, 2017 37 / 57
71. Result Inpainted Image + Spatial patch blending Olivier Lézoray ICIAE 2017 March 28, 2017 38 / 57
72. block-effect artifacts ? Masked Image Olivier Lézoray ICIAE 2017 March 28, 2017 39 / 57
73. block-effect artifacts ? Inpainted Image Olivier Lézoray ICIAE 2017 March 28, 2017 39 / 57
74. block-effect artifacts ? Blended Image Olivier Lézoray ICIAE 2017 March 28, 2017 39 / 57
75. Original Image Olivier Lézoray ICIAE 2017 March 28, 2017 40 / 57
76. Inpainted with improved version of [Criminisi et al’03] Olivier Lézoray ICIAE 2017 March 28, 2017 40 / 57
77. After patch blending Olivier Lézoray ICIAE 2017 March 28, 2017 40 / 57
78. Content Aware Filling (Photoshop) Olivier Lézoray ICIAE 2017 March 28, 2017 40 / 57
79. Geometry-guided patch blending : process Idea : Is it possible to take the local geometry of the image contours into account ? Anisotropic patch blending • For flat areas (λS1 ≈ λS2 ) : large isotropic blending tensors (as structure tensors) for smoothing • For structured areas (λS1 λS2 ) : small anisotropic blending tensors to preserve sharp structures Olivier Lézoray ICIAE 2017 March 28, 2017 41 / 57
80. Geometry-guided patch blending Weighted sum of overlapping patches pixels values using blending with w(p, q) =    exp − p−q 2 σ(p)2 isotropic exp XT B(p)−1X 2σ2 B anisotropic where X = q − p Olivier Lézoray ICIAE 2017 March 28, 2017 42 / 57
81. Geometry-guided patch blending : process Example of blending tensor field Olivier Lézoray ICIAE 2017 March 28, 2017 43 / 57
82. block-effect artifacts ? Masked Image Olivier Lézoray ICIAE 2017 March 28, 2017 44 / 57
83. block-effect artifacts ? Inpainted Image Olivier Lézoray ICIAE 2017 March 28, 2017 44 / 57
84. block-effect artifacts ? Blended Image (isotropic) Olivier Lézoray ICIAE 2017 March 28, 2017 44 / 57
85. block-effect artifacts ? Blended Image (geometry-guided) Olivier Lézoray ICIAE 2017 March 28, 2017 44 / 57
86. Woman Masked image Olivier Lézoray ICIAE 2017 March 28, 2017 45 / 57
87. Woman Without blending Olivier Lézoray ICIAE 2017 March 28, 2017 45 / 57
88. Woman Isotropic patch blending Olivier Lézoray ICIAE 2017 March 28, 2017 45 / 57
89. Woman Geometry-guided patch blending Olivier Lézoray ICIAE 2017 March 28, 2017 45 / 57
90. Outline 1 Introduction 2 Inpainting approaches 3 Improvements of Exemplar-based inpainting 4 Geometry guided patch blending for artifact reduction 5 Extension to other modalities Video inpainting Reconstruction of re-synthetized stereoscopic views Olivier Lézoray ICIAE 2017 March 28, 2017 46 / 57
91. Video inpainting • priorities computed frame by frame • patches and lookup windows are now in 3D • blending both in space and time Olivier Lézoray ICIAE 2017 March 28, 2017 47 / 57
92. Masked video : (x,y) and (x,t) plans Olivier Lézoray ICIAE 2017 March 28, 2017 48 / 57
93. Inpainted video (without blending) Olivier Lézoray ICIAE 2017 March 28, 2017 48 / 57
94. Blended Image (geometry-guided) Olivier Lézoray ICIAE 2017 March 28, 2017 48 / 57
95. Inpainted video Olivier Lézoray ICIAE 2017 March 28, 2017 49 / 57
96. Outline 1 Introduction 2 Inpainting approaches 3 Improvements of Exemplar-based inpainting 4 Geometry guided patch blending for artifact reduction 5 Extension to other modalities Video inpainting Reconstruction of re-synthetized stereoscopic views Olivier Lézoray ICIAE 2017 March 28, 2017 50 / 57
97. The problem right view Olivier Lézoray ICIAE 2017 March 28, 2017 51 / 57
98. The problem synthesized color image original view synthesized depth original depth Olivier Lézoray ICIAE 2017 March 28, 2017 52 / 57
99. The problem synthesized color image (Is) original view (Io) synthesized depth (inpainted) (Js) original depth Olivier Lézoray ICIAE 2017 March 28, 2017 53 / 57
100. Inpainting with a depth map We use our improved [Criminisi et al’03] and by taking into account the depth : • Count as reliable pixels with similar depth Hole interior data term confidence priority boundaries term term Olivier Lézoray ICIAE 2017 March 28, 2017 54 / 57
101. Inpainting with a depth map We use our improved [Criminisi et al’03] and by taking into account the depth : • modify the lookup strategy to look for patches with similar depth Is Io Olivier Lézoray ICIAE 2017 March 28, 2017 54 / 57
102. Inpainting with a depth map We use our improved [Criminisi et al’03] and by taking into account the depth : • Paste only pixels with equal depth Ψt in Is Ψt in Js Ψˆ t (p) in Io Result Olivier Lézoray ICIAE 2017 March 28, 2017 54 / 57
103. Olivier Lézoray ICIAE 2017 March 28, 2017 55 / 57
104. Olivier Lézoray ICIAE 2017 March 28, 2017 55 / 57
105. Olivier Lézoray ICIAE 2017 March 28, 2017 55 / 57
106. Video illustrating the inpainting Olivier Lézoray ICIAE 2017 March 28, 2017 56 / 57
107. The end Publications available at : https://lezoray.users.greyc.fr Olivier Lézoray ICIAE 2017 March 28, 2017 57 / 57