ICIAE 2017

Transcript

1. Improvements of Exemplar-based Image Inpainting Olivier Lézoray with Maxime Daisy,

   Pierre Buyssens, David Tschumperlé Normandie Univ, UNICAEN, ENSICAEN, CNRS, GREYC, 14000 Caen, France ICIAE 2017 March 28, 2017

2. Overview 1 Introduction 2 Inpainting approaches 3 Improvements of Exemplar-based

   inpainting 4 Geometry guided patch blending for artifact reduction 5 Extension to other modalities Video inpainting Reconstruction of re-synthetized stereoscopic views Olivier Lézoray ICIAE 2017 March 28, 2017 2 / 57

3. Outline 1 Introduction 2 Inpainting approaches 3 Improvements of Exemplar-based

   inpainting 4 Geometry guided patch blending for artifact reduction 5 Extension to other modalities Olivier Lézoray ICIAE 2017 March 28, 2017 3 / 57

4. What is inpainting ? Olivier Lézoray ICIAE 2017 March 28,

   2017 4 / 57

5. What is inpainting ? Olivier Lézoray ICIAE 2017 March 28,

   2017 4 / 57

6. What is inpainting ? Olivier Lézoray ICIAE 2017 March 28,

   2017 4 / 57

7. Why is it a hard problem ? Olivier Lézoray ICIAE

   2017 March 28, 2017 5 / 57

8. Why is it a hard problem ? Olivier Lézoray ICIAE

   2017 March 28, 2017 5 / 57

9. Why is it a hard problem ? Olivier Lézoray ICIAE

   2017 March 28, 2017 5 / 57

10. How to do it manually ? Use the clone tool

    Olivier Lézoray ICIAE 2017 March 28, 2017 6 / 57

11. How to do it manually ? Olivier Lézoray ICIAE 2017

    March 28, 2017 6 / 57

12. How to do it manually ? Olivier Lézoray ICIAE 2017

    March 28, 2017 6 / 57

13. How to do it manually ? Olivier Lézoray ICIAE 2017

    March 28, 2017 6 / 57

14. How to do it manually ? Olivier Lézoray ICIAE 2017

    March 28, 2017 6 / 57

15. How to do it manually ? Olivier Lézoray ICIAE 2017

    March 28, 2017 6 / 57

16. Outline 1 Introduction 2 Inpainting approaches 3 Improvements of Exemplar-based

    inpainting 4 Geometry guided patch blending for artifact reduction 5 Extension to other modalities Olivier Lézoray ICIAE 2017 March 28, 2017 7 / 57

17. Geometry-based approaches Purely geometry-driven methods [Masnou et al. ’98, Bertalmio

    et al. ’00, Chan et al. ’01, Tschumperlé and Deriche ’05, . . .] Idea : Use a diffusion equation guided by the isophotes directions → performs boundary geometry interpolation • Good global geometry reconstruction • Expressed with an energy functional to minimize • Not able to create complex textures • Time consuming Olivier Lézoray ICIAE 2017 March 28, 2017 8 / 57

18. Masked Image. Olivier Lézoray ICIAE 2017 March 28, 2017 9

    / 57

19. Inpainted with [Tschumperlé and Deriche ’05] Olivier Lézoray ICIAE 2017

    March 28, 2017 9 / 57

20. Exemplar-based approaches Pattern-based methods [Efros and Leung’99, Criminisi et al.

    ’04, Lemeur et al. ’11, . . .] Idea : exploit texture auto-similarity and local analysis of image geometry → copy/paste of image chunks • Large area reconstruction • Quite fast • Not enough local/global coherence e.g. block-effect artifacts ⇒ Best compromise between time and quality Olivier Lézoray ICIAE 2017 March 28, 2017 10 / 57

21. Exemplar-based approaches Criminisi et al. 2004, "Region Filling and Object

    Removal by Exemplar-Based Image Inpainting" C : Confidence term (reliable information for reach patch) D : Data term (local structure estimation) Olivier Lézoray ICIAE 2017 March 28, 2017 11 / 57

22. Exemplar-based approaches Criminisi et al. 2004, "Region Filling and Object

    Removal by Exemplar-Based Image Inpainting" C : Confidence term (reliable information for reach patch) D : Data term (local structure estimation) Olivier Lézoray ICIAE 2017 March 28, 2017 11 / 57

23. Exemplar-based approaches Criminisi et al. 2004, "Region Filling and Object

    Removal by Exemplar-Based Image Inpainting" C : Confidence term (reliable information for reach patch) D : Data term (local structure estimation) Olivier Lézoray ICIAE 2017 March 28, 2017 11 / 57

24. Masked Image. Olivier Lézoray ICIAE 2017 March 28, 2017 12

    / 57

25. Inpainted with [Criminisi et al. ’04] Olivier Lézoray ICIAE 2017

    March 28, 2017 12 / 57

26. Fast patch search PatchMatch [Barnes et al’09] • A very

    fast algorithm for matching similar patchs between two images • Works for full images (non-masked patchs) and is stochastic Used in several efficient image and video inpainting techniques (multi-scale approaches) : • [Wexler et al. ’07] + PatchMatch : available in PhotoShop • The solution at one scale is used as an initialization for the upper scale (may increase error propagation) Olivier Lézoray ICIAE 2017 March 28, 2017 13 / 57

27. [Wexler et al. ’07] Inpainted with [Wexler et al. ’07]

    Olivier Lézoray ICIAE 2017 March 28, 2017 14 / 57

28. Key-steps for improvements Two key steps can be considered for

    improvements : • The analysis of the image geometry • The analysis of the best patches to copy/paste Olivier Lézoray ICIAE 2017 March 28, 2017 15 / 57

29. Key-steps for improvements Two key steps can be considered for

    improvements : • The analysis of the image geometry • The analysis of the best patches to copy/paste • Problems with curved structures to reconstruct Olivier Lézoray ICIAE 2017 March 28, 2017 15 / 57

30. Key-steps for improvements Two key steps can be considered for

    improvements : • The analysis of the image geometry • The analysis of the best patches to copy/paste • Problems with curved structures to reconstruct • Visible bloc effects Olivier Lézoray ICIAE 2017 March 28, 2017 15 / 57

31. Key-steps for improvements Two key steps can be considered for

    improvements : • The analysis of the image geometry • The analysis of the best patches to copy/paste • Problems with curved structures to reconstruct • Visible bloc effects Olivier Lézoray ICIAE 2017 March 28, 2017 15 / 57

32. Outline 1 Introduction 2 Inpainting approaches 3 Improvements of Exemplar-based

    inpainting 4 Geometry guided patch blending for artifact reduction 5 Extension to other modalities Olivier Lézoray ICIAE 2017 March 28, 2017 16 / 57

33. Exemplar-based Inpainting Algorithm improvements Proposed improvements 1 priority accuracy enhancement

    ⇒ more global geometry consistency 2 better lookup strategy ⇒ more local geometry consistency 3 spatial patch blending ⇒ strongly reduced block-effect artifacts Olivier Lézoray ICIAE 2017 March 28, 2017 17 / 57

34. A better data term accuracy Criminisi et al. data term

    Dp = | − → ∇Ip ⊥ . − → np| α (1) Dp favors the reconstruction of local structures that are orthogonal to the mask "The gradient ∇Ip is computed as the maximum value of the image gradient in Ψp ∩ I." − → ∇I⊥ p = { − → ∇I⊥ q | arg max q∈((I−Ω)∩ψp) − → ∇Iq } (isophote direction) (2) Olivier Lézoray ICIAE 2017 March 28, 2017 18 / 57

35. A better data term accuracy Criminisi et al. data term

    Dp = | − → ∇Ip ⊥ . − → np| α (1) Dp favors the reconstruction of local structures that are orthogonal to the mask "The gradient ∇Ip is computed as the maximum value of the image gradient in Ψp ∩ I." − → ∇I⊥ p = { − → ∇I⊥ q | arg max q∈((I−Ω)∩ψp) − → ∇Iq } (isophote direction) (2) Olivier Lézoray ICIAE 2017 March 28, 2017 18 / 57

36. Structure tensors Structure tensors provide a description of the local

    geometry for vector images : Sp = N k=1 − → ∇Ik (p) − → ∇Ik (p)T (3) Structure tensors are symmetric and positive matrices (2 × 2) that can be decomposed with their eigen decomposition in Sp = N k=1 λk ek eT k (4) Tensor is said anisotropic when λ1 λ2 Olivier Lézoray ICIAE 2017 March 28, 2017 19 / 57

37. A better data term accuracy Data term based on the

    smoothed structure tensor field ˜ Dp = Gp . − → n p (5) with Gp = q∈ψp∩(I−Ω) wp (q) − → ∇Iq − → ∇IT q (6) wp : normalized 2d gaussian centered at p High data term for anisotropic tensors Olivier Lézoray ICIAE 2017 March 28, 2017 20 / 57

38. A better data term accuracy Data term based on the

    smoothed structure tensor field ˜ Dp = Gp . − → n p (5) with Gp = q∈ψp∩(I−Ω) wp (q) − → ∇Iq − → ∇IT q (6) wp : normalized 2d gaussian centered at p High data term for anisotropic tensors Olivier Lézoray ICIAE 2017 March 28, 2017 20 / 57

39. Example Original image Olivier Lézoray ICIAE 2017 March 28, 2017

    21 / 57

40. Example Original data term from [Criminisi et al. ’03] Olivier

    Lézoray ICIAE 2017 March 28, 2017 21 / 57

41. Example Our data term Olivier Lézoray ICIAE 2017 March 28,

    2017 21 / 57

42. Example Inpainting result with original data term from [Criminisi et

    al. ’03] Olivier Lézoray ICIAE 2017 March 28, 2017 21 / 57

43. Example Inpainting result with our data term Olivier Lézoray ICIAE

    2017 March 28, 2017 21 / 57

44. A better lookup strategy Criminisi et al. 2004 lookup strategy

    • window search ? ⇒ faster ⇒ much less global • all over the image search ? ⇒ slow ⇒ not so much good results Olivier Lézoray ICIAE 2017 March 28, 2017 22 / 57

45. A better lookup strategy Window search Iteration 0 : the

    number of available patches is sufficient to find a good matching Olivier Lézoray ICIAE 2017 March 28, 2017 23 / 57

46. A better lookup strategy Window search Iteration n : the

    number of available patches is not sufficient to find a good matching Olivier Lézoray ICIAE 2017 March 28, 2017 23 / 57

47. A better lookup strategy Our lookup strategy • inspired from

    [Ashikhmin ’01, PatchMatch ’09] • local/global search scheme • use search sites of surrounding inpainted patches • search site sizes are inversely proportional to the number of sites ⇒ more local coherence Olivier Lézoray ICIAE 2017 March 28, 2017 24 / 57

48. A better lookup strategy • The set of already pasted

    patches around one pixel is determined (if no : use classical scheme) • Search windows are added around original pasted patches • most search sites overlap, and matching is faster • patch size becomes less critical Olivier Lézoray ICIAE 2017 March 28, 2017 25 / 57

49. Joshua Tree - 512 × 384 Masked Image Olivier Lézoray

    ICIAE 2017 March 28, 2017 26 / 57

50. Joshua Tree - 512 × 384 Windowed search (4.5s) Olivier

    Lézoray ICIAE 2017 March 28, 2017 26 / 57

51. Joshua Tree - 512 × 384 With smart search (3.2s)

    Olivier Lézoray ICIAE 2017 March 28, 2017 26 / 57

52. Fake Owl - 800 × 450 Masked Image Olivier Lézoray

    ICIAE 2017 March 28, 2017 27 / 57

53. Fake Owl - 800 × 450 Windowed search (1.7s) Olivier

    Lézoray ICIAE 2017 March 28, 2017 27 / 57

54. Fake Owl - 800 × 450 With smart search (0.8s)

    Olivier Lézoray ICIAE 2017 March 28, 2017 27 / 57

55. block-effect artifacts ? Olivier Lézoray ICIAE 2017 March 28, 2017

    28 / 57

56. Outline 1 Introduction 2 Inpainting approaches 3 Improvements of Exemplar-based

    inpainting 4 Geometry guided patch blending for artifact reduction 5 Extension to other modalities Olivier Lézoray ICIAE 2017 March 28, 2017 29 / 57

57. Spatial Patch Blending Masked Image. Olivier Lézoray ICIAE 2017 March

    28, 2017 30 / 57

58. Spatial Patch Blending Inpainted with [Criminisi et al. ’04] Olivier

    Lézoray ICIAE 2017 March 28, 2017 30 / 57

59. Spatial Patch Blending Inpainted with [Criminisi et al. ’04] +

    Our spatial patch blending Olivier Lézoray ICIAE 2017 March 28, 2017 30 / 57

60. Patch Blending : method Idea : Visually minimize block artifacts

    due to side-by-side pasted patches. • Remember the location of the pasted patches • Detect the locations of the artifacts • Spatial blending of the patches Olivier Lézoray ICIAE 2017 March 28, 2017 31 / 57

61. Artifact Detection Masked image. Olivier Lézoray ICIAE 2017 March 28,

    2017 32 / 57

62. Artifact Detection Inpainted image. Olivier Lézoray ICIAE 2017 March 28,

    2017 32 / 57

63. Artifact Detection Break field image. Olivier Lézoray ICIAE 2017 March

    28, 2017 32 / 57

64. Artifact Detection Break points. Olivier Lézoray ICIAE 2017 March 28,

    2017 32 / 57

65. Artifact Detection Blending amplitude map. Olivier Lézoray ICIAE 2017 March

    28, 2017 32 / 57

66. Artifact Detection 1 Map of the artifact location probabilities 2

    hypothesis for artifact locations : PA = ∇I . div(φ) • local sharp variations in the inpainted image (high gradient) • source patches come from very different locations (discontinuities in the shift map φ) 2 Strongest artifact locations LA = {p | PA(p) > τ} 3 Blending amplitude map σ(p) = q∈LA w(p,q) max r∈I q∈LA w(p,q) with w(p, q) = exp − p−q 2 PA(q)2 (7) Olivier Lézoray ICIAE 2017 March 28, 2017 33 / 57

67. Patch Blending Spatial blending of all the patches that overlap

    at a given pixel p Here : compute a weighted average of p1, p2 and p3 Weights depend on distance between patch centers and pixel to blend. Olivier Lézoray ICIAE 2017 March 28, 2017 34 / 57

68. Patch Blending Weighted sum of overlapping patches pixels values J(p)

    = ψq∈Ψp w(p, q) . ψq (p − q) ψq∈Ψp w(q, p) (8) • w(p, q) = exp − p−q 2 σ(p)2 • Ψp = {ψq | ψq ∩ ψp = ∅} (set of patches that overlap at p) → one gaussian function for each possible blending amplitude ⇒ quite slow (convolution with a spatially variant kernel) ⇒ use a quantization of σ to make it fast Olivier Lézoray ICIAE 2017 March 28, 2017 35 / 57

69. Result Masked image. Olivier Lézoray ICIAE 2017 March 28, 2017

    36 / 57

70. Result Inpainted image Olivier Lézoray ICIAE 2017 March 28, 2017

    37 / 57

71. Result Inpainted Image + Spatial patch blending Olivier Lézoray ICIAE

    2017 March 28, 2017 38 / 57

72. block-effect artifacts ? Masked Image Olivier Lézoray ICIAE 2017 March

    28, 2017 39 / 57

73. block-effect artifacts ? Inpainted Image Olivier Lézoray ICIAE 2017 March

    28, 2017 39 / 57

74. block-effect artifacts ? Blended Image Olivier Lézoray ICIAE 2017 March

    28, 2017 39 / 57

75. Original Image Olivier Lézoray ICIAE 2017 March 28, 2017 40

    / 57

76. Inpainted with improved version of [Criminisi et al’03] Olivier Lézoray

    ICIAE 2017 March 28, 2017 40 / 57

77. After patch blending Olivier Lézoray ICIAE 2017 March 28, 2017

    40 / 57

78. Content Aware Filling (Photoshop) Olivier Lézoray ICIAE 2017 March 28,

    2017 40 / 57

79. Geometry-guided patch blending : process Idea : Is it possible

    to take the local geometry of the image contours into account ? Anisotropic patch blending • For flat areas (λS1 ≈ λS2 ) : large isotropic blending tensors (as structure tensors) for smoothing • For structured areas (λS1 λS2 ) : small anisotropic blending tensors to preserve sharp structures Olivier Lézoray ICIAE 2017 March 28, 2017 41 / 57

80. Geometry-guided patch blending Weighted sum of overlapping patches pixels values

    using blending with w(p, q) =    exp − p−q 2 σ(p)2 isotropic exp XT B(p)−1X 2σ2 B anisotropic where X = q − p Olivier Lézoray ICIAE 2017 March 28, 2017 42 / 57

81. Geometry-guided patch blending : process Example of blending tensor field

    Olivier Lézoray ICIAE 2017 March 28, 2017 43 / 57

82. block-effect artifacts ? Masked Image Olivier Lézoray ICIAE 2017 March

    28, 2017 44 / 57

83. block-effect artifacts ? Inpainted Image Olivier Lézoray ICIAE 2017 March

    28, 2017 44 / 57

84. block-effect artifacts ? Blended Image (isotropic) Olivier Lézoray ICIAE 2017

    March 28, 2017 44 / 57

85. block-effect artifacts ? Blended Image (geometry-guided) Olivier Lézoray ICIAE 2017

    March 28, 2017 44 / 57

86. Woman Masked image Olivier Lézoray ICIAE 2017 March 28, 2017

    45 / 57

87. Woman Without blending Olivier Lézoray ICIAE 2017 March 28, 2017

    45 / 57

88. Woman Isotropic patch blending Olivier Lézoray ICIAE 2017 March 28,

    2017 45 / 57

89. Woman Geometry-guided patch blending Olivier Lézoray ICIAE 2017 March 28,

    2017 45 / 57

90. Outline 1 Introduction 2 Inpainting approaches 3 Improvements of Exemplar-based

    inpainting 4 Geometry guided patch blending for artifact reduction 5 Extension to other modalities Video inpainting Reconstruction of re-synthetized stereoscopic views Olivier Lézoray ICIAE 2017 March 28, 2017 46 / 57

91. Video inpainting • priorities computed frame by frame • patches

    and lookup windows are now in 3D • blending both in space and time Olivier Lézoray ICIAE 2017 March 28, 2017 47 / 57

92. Masked video : (x,y) and (x,t) plans Olivier Lézoray ICIAE

    2017 March 28, 2017 48 / 57

93. Inpainted video (without blending) Olivier Lézoray ICIAE 2017 March 28,

    2017 48 / 57

94. Blended Image (geometry-guided) Olivier Lézoray ICIAE 2017 March 28, 2017

    48 / 57

95. Inpainted video Olivier Lézoray ICIAE 2017 March 28, 2017 49

    / 57

96. Outline 1 Introduction 2 Inpainting approaches 3 Improvements of Exemplar-based

    inpainting 4 Geometry guided patch blending for artifact reduction 5 Extension to other modalities Video inpainting Reconstruction of re-synthetized stereoscopic views Olivier Lézoray ICIAE 2017 March 28, 2017 50 / 57

97. The problem right view Olivier Lézoray ICIAE 2017 March 28,

    2017 51 / 57

98. The problem synthesized color image original view synthesized depth original

    depth Olivier Lézoray ICIAE 2017 March 28, 2017 52 / 57

99. The problem synthesized color image (Is) original view (Io) synthesized

    depth (inpainted) (Js) original depth Olivier Lézoray ICIAE 2017 March 28, 2017 53 / 57

100. Inpainting with a depth map We use our improved [Criminisi

     et al’03] and by taking into account the depth : • Count as reliable pixels with similar depth Hole interior data term confidence priority boundaries term term Olivier Lézoray ICIAE 2017 March 28, 2017 54 / 57

101. Inpainting with a depth map We use our improved [Criminisi

     et al’03] and by taking into account the depth : • modify the lookup strategy to look for patches with similar depth Is Io Olivier Lézoray ICIAE 2017 March 28, 2017 54 / 57

102. Inpainting with a depth map We use our improved [Criminisi

     et al’03] and by taking into account the depth : • Paste only pixels with equal depth Ψt in Is Ψt in Js Ψˆ t (p) in Io Result Olivier Lézoray ICIAE 2017 March 28, 2017 54 / 57

103. Olivier Lézoray ICIAE 2017 March 28, 2017 55 / 57

104. Olivier Lézoray ICIAE 2017 March 28, 2017 55 / 57

105. Olivier Lézoray ICIAE 2017 March 28, 2017 55 / 57

106. Video illustrating the inpainting Olivier Lézoray ICIAE 2017 March 28,

     2017 56 / 57

107. The end Publications available at : https://lezoray.users.greyc.fr Olivier Lézoray ICIAE

     2017 March 28, 2017 57 / 57