ICIAE 2017

ICIAE 2017

1509d0ae6901a6cf2c2fd7cc97f02fc0?s=128

Olivier Lézoray

March 28, 2017
Tweet

Transcript

  1. Improvements of Exemplar-based Image Inpainting Olivier Lézoray with Maxime Daisy,

    Pierre Buyssens, David Tschumperlé Normandie Univ, UNICAEN, ENSICAEN, CNRS, GREYC, 14000 Caen, France ICIAE 2017 March 28, 2017
  2. Overview 1 Introduction 2 Inpainting approaches 3 Improvements of Exemplar-based

    inpainting 4 Geometry guided patch blending for artifact reduction 5 Extension to other modalities Video inpainting Reconstruction of re-synthetized stereoscopic views Olivier Lézoray ICIAE 2017 March 28, 2017 2 / 57
  3. Outline 1 Introduction 2 Inpainting approaches 3 Improvements of Exemplar-based

    inpainting 4 Geometry guided patch blending for artifact reduction 5 Extension to other modalities Olivier Lézoray ICIAE 2017 March 28, 2017 3 / 57
  4. What is inpainting ? Olivier Lézoray ICIAE 2017 March 28,

    2017 4 / 57
  5. What is inpainting ? Olivier Lézoray ICIAE 2017 March 28,

    2017 4 / 57
  6. What is inpainting ? Olivier Lézoray ICIAE 2017 March 28,

    2017 4 / 57
  7. Why is it a hard problem ? Olivier Lézoray ICIAE

    2017 March 28, 2017 5 / 57
  8. Why is it a hard problem ? Olivier Lézoray ICIAE

    2017 March 28, 2017 5 / 57
  9. Why is it a hard problem ? Olivier Lézoray ICIAE

    2017 March 28, 2017 5 / 57
  10. How to do it manually ? Use the clone tool

    Olivier Lézoray ICIAE 2017 March 28, 2017 6 / 57
  11. How to do it manually ? Olivier Lézoray ICIAE 2017

    March 28, 2017 6 / 57
  12. How to do it manually ? Olivier Lézoray ICIAE 2017

    March 28, 2017 6 / 57
  13. How to do it manually ? Olivier Lézoray ICIAE 2017

    March 28, 2017 6 / 57
  14. How to do it manually ? Olivier Lézoray ICIAE 2017

    March 28, 2017 6 / 57
  15. How to do it manually ? Olivier Lézoray ICIAE 2017

    March 28, 2017 6 / 57
  16. Outline 1 Introduction 2 Inpainting approaches 3 Improvements of Exemplar-based

    inpainting 4 Geometry guided patch blending for artifact reduction 5 Extension to other modalities Olivier Lézoray ICIAE 2017 March 28, 2017 7 / 57
  17. Geometry-based approaches Purely geometry-driven methods [Masnou et al. ’98, Bertalmio

    et al. ’00, Chan et al. ’01, Tschumperlé and Deriche ’05, . . .] Idea : Use a diffusion equation guided by the isophotes directions → performs boundary geometry interpolation • Good global geometry reconstruction • Expressed with an energy functional to minimize • Not able to create complex textures • Time consuming Olivier Lézoray ICIAE 2017 March 28, 2017 8 / 57
  18. Masked Image. Olivier Lézoray ICIAE 2017 March 28, 2017 9

    / 57
  19. Inpainted with [Tschumperlé and Deriche ’05] Olivier Lézoray ICIAE 2017

    March 28, 2017 9 / 57
  20. Exemplar-based approaches Pattern-based methods [Efros and Leung’99, Criminisi et al.

    ’04, Lemeur et al. ’11, . . .] Idea : exploit texture auto-similarity and local analysis of image geometry → copy/paste of image chunks • Large area reconstruction • Quite fast • Not enough local/global coherence e.g. block-effect artifacts ⇒ Best compromise between time and quality Olivier Lézoray ICIAE 2017 March 28, 2017 10 / 57
  21. Exemplar-based approaches Criminisi et al. 2004, "Region Filling and Object

    Removal by Exemplar-Based Image Inpainting" C : Confidence term (reliable information for reach patch) D : Data term (local structure estimation) Olivier Lézoray ICIAE 2017 March 28, 2017 11 / 57
  22. Exemplar-based approaches Criminisi et al. 2004, "Region Filling and Object

    Removal by Exemplar-Based Image Inpainting" C : Confidence term (reliable information for reach patch) D : Data term (local structure estimation) Olivier Lézoray ICIAE 2017 March 28, 2017 11 / 57
  23. Exemplar-based approaches Criminisi et al. 2004, "Region Filling and Object

    Removal by Exemplar-Based Image Inpainting" C : Confidence term (reliable information for reach patch) D : Data term (local structure estimation) Olivier Lézoray ICIAE 2017 March 28, 2017 11 / 57
  24. Masked Image. Olivier Lézoray ICIAE 2017 March 28, 2017 12

    / 57
  25. Inpainted with [Criminisi et al. ’04] Olivier Lézoray ICIAE 2017

    March 28, 2017 12 / 57
  26. Fast patch search PatchMatch [Barnes et al’09] • A very

    fast algorithm for matching similar patchs between two images • Works for full images (non-masked patchs) and is stochastic Used in several efficient image and video inpainting techniques (multi-scale approaches) : • [Wexler et al. ’07] + PatchMatch : available in PhotoShop • The solution at one scale is used as an initialization for the upper scale (may increase error propagation) Olivier Lézoray ICIAE 2017 March 28, 2017 13 / 57
  27. [Wexler et al. ’07] Inpainted with [Wexler et al. ’07]

    Olivier Lézoray ICIAE 2017 March 28, 2017 14 / 57
  28. Key-steps for improvements Two key steps can be considered for

    improvements : • The analysis of the image geometry • The analysis of the best patches to copy/paste Olivier Lézoray ICIAE 2017 March 28, 2017 15 / 57
  29. Key-steps for improvements Two key steps can be considered for

    improvements : • The analysis of the image geometry • The analysis of the best patches to copy/paste • Problems with curved structures to reconstruct Olivier Lézoray ICIAE 2017 March 28, 2017 15 / 57
  30. Key-steps for improvements Two key steps can be considered for

    improvements : • The analysis of the image geometry • The analysis of the best patches to copy/paste • Problems with curved structures to reconstruct • Visible bloc effects Olivier Lézoray ICIAE 2017 March 28, 2017 15 / 57
  31. Key-steps for improvements Two key steps can be considered for

    improvements : • The analysis of the image geometry • The analysis of the best patches to copy/paste • Problems with curved structures to reconstruct • Visible bloc effects Olivier Lézoray ICIAE 2017 March 28, 2017 15 / 57
  32. Outline 1 Introduction 2 Inpainting approaches 3 Improvements of Exemplar-based

    inpainting 4 Geometry guided patch blending for artifact reduction 5 Extension to other modalities Olivier Lézoray ICIAE 2017 March 28, 2017 16 / 57
  33. Exemplar-based Inpainting Algorithm improvements Proposed improvements 1 priority accuracy enhancement

    ⇒ more global geometry consistency 2 better lookup strategy ⇒ more local geometry consistency 3 spatial patch blending ⇒ strongly reduced block-effect artifacts Olivier Lézoray ICIAE 2017 March 28, 2017 17 / 57
  34. A better data term accuracy Criminisi et al. data term

    Dp = | − → ∇Ip ⊥ . − → np| α (1) Dp favors the reconstruction of local structures that are orthogonal to the mask "The gradient ∇Ip is computed as the maximum value of the image gradient in Ψp ∩ I." − → ∇I⊥ p = { − → ∇I⊥ q | arg max q∈((I−Ω)∩ψp) − → ∇Iq } (isophote direction) (2) Olivier Lézoray ICIAE 2017 March 28, 2017 18 / 57
  35. A better data term accuracy Criminisi et al. data term

    Dp = | − → ∇Ip ⊥ . − → np| α (1) Dp favors the reconstruction of local structures that are orthogonal to the mask "The gradient ∇Ip is computed as the maximum value of the image gradient in Ψp ∩ I." − → ∇I⊥ p = { − → ∇I⊥ q | arg max q∈((I−Ω)∩ψp) − → ∇Iq } (isophote direction) (2) Olivier Lézoray ICIAE 2017 March 28, 2017 18 / 57
  36. Structure tensors Structure tensors provide a description of the local

    geometry for vector images : Sp = N k=1 − → ∇Ik (p) − → ∇Ik (p)T (3) Structure tensors are symmetric and positive matrices (2 × 2) that can be decomposed with their eigen decomposition in Sp = N k=1 λk ek eT k (4) Tensor is said anisotropic when λ1 λ2 Olivier Lézoray ICIAE 2017 March 28, 2017 19 / 57
  37. A better data term accuracy Data term based on the

    smoothed structure tensor field ˜ Dp = Gp . − → n p (5) with Gp = q∈ψp∩(I−Ω) wp (q) − → ∇Iq − → ∇IT q (6) wp : normalized 2d gaussian centered at p High data term for anisotropic tensors Olivier Lézoray ICIAE 2017 March 28, 2017 20 / 57
  38. A better data term accuracy Data term based on the

    smoothed structure tensor field ˜ Dp = Gp . − → n p (5) with Gp = q∈ψp∩(I−Ω) wp (q) − → ∇Iq − → ∇IT q (6) wp : normalized 2d gaussian centered at p High data term for anisotropic tensors Olivier Lézoray ICIAE 2017 March 28, 2017 20 / 57
  39. Example Original image Olivier Lézoray ICIAE 2017 March 28, 2017

    21 / 57
  40. Example Original data term from [Criminisi et al. ’03] Olivier

    Lézoray ICIAE 2017 March 28, 2017 21 / 57
  41. Example Our data term Olivier Lézoray ICIAE 2017 March 28,

    2017 21 / 57
  42. Example Inpainting result with original data term from [Criminisi et

    al. ’03] Olivier Lézoray ICIAE 2017 March 28, 2017 21 / 57
  43. Example Inpainting result with our data term Olivier Lézoray ICIAE

    2017 March 28, 2017 21 / 57
  44. A better lookup strategy Criminisi et al. 2004 lookup strategy

    • window search ? ⇒ faster ⇒ much less global • all over the image search ? ⇒ slow ⇒ not so much good results Olivier Lézoray ICIAE 2017 March 28, 2017 22 / 57
  45. A better lookup strategy Window search Iteration 0 : the

    number of available patches is sufficient to find a good matching Olivier Lézoray ICIAE 2017 March 28, 2017 23 / 57
  46. A better lookup strategy Window search Iteration n : the

    number of available patches is not sufficient to find a good matching Olivier Lézoray ICIAE 2017 March 28, 2017 23 / 57
  47. A better lookup strategy Our lookup strategy • inspired from

    [Ashikhmin ’01, PatchMatch ’09] • local/global search scheme • use search sites of surrounding inpainted patches • search site sizes are inversely proportional to the number of sites ⇒ more local coherence Olivier Lézoray ICIAE 2017 March 28, 2017 24 / 57
  48. A better lookup strategy • The set of already pasted

    patches around one pixel is determined (if no : use classical scheme) • Search windows are added around original pasted patches • most search sites overlap, and matching is faster • patch size becomes less critical Olivier Lézoray ICIAE 2017 March 28, 2017 25 / 57
  49. Joshua Tree - 512 × 384 Masked Image Olivier Lézoray

    ICIAE 2017 March 28, 2017 26 / 57
  50. Joshua Tree - 512 × 384 Windowed search (4.5s) Olivier

    Lézoray ICIAE 2017 March 28, 2017 26 / 57
  51. Joshua Tree - 512 × 384 With smart search (3.2s)

    Olivier Lézoray ICIAE 2017 March 28, 2017 26 / 57
  52. Fake Owl - 800 × 450 Masked Image Olivier Lézoray

    ICIAE 2017 March 28, 2017 27 / 57
  53. Fake Owl - 800 × 450 Windowed search (1.7s) Olivier

    Lézoray ICIAE 2017 March 28, 2017 27 / 57
  54. Fake Owl - 800 × 450 With smart search (0.8s)

    Olivier Lézoray ICIAE 2017 March 28, 2017 27 / 57
  55. block-effect artifacts ? Olivier Lézoray ICIAE 2017 March 28, 2017

    28 / 57
  56. Outline 1 Introduction 2 Inpainting approaches 3 Improvements of Exemplar-based

    inpainting 4 Geometry guided patch blending for artifact reduction 5 Extension to other modalities Olivier Lézoray ICIAE 2017 March 28, 2017 29 / 57
  57. Spatial Patch Blending Masked Image. Olivier Lézoray ICIAE 2017 March

    28, 2017 30 / 57
  58. Spatial Patch Blending Inpainted with [Criminisi et al. ’04] Olivier

    Lézoray ICIAE 2017 March 28, 2017 30 / 57
  59. Spatial Patch Blending Inpainted with [Criminisi et al. ’04] +

    Our spatial patch blending Olivier Lézoray ICIAE 2017 March 28, 2017 30 / 57
  60. Patch Blending : method Idea : Visually minimize block artifacts

    due to side-by-side pasted patches. • Remember the location of the pasted patches • Detect the locations of the artifacts • Spatial blending of the patches Olivier Lézoray ICIAE 2017 March 28, 2017 31 / 57
  61. Artifact Detection Masked image. Olivier Lézoray ICIAE 2017 March 28,

    2017 32 / 57
  62. Artifact Detection Inpainted image. Olivier Lézoray ICIAE 2017 March 28,

    2017 32 / 57
  63. Artifact Detection Break field image. Olivier Lézoray ICIAE 2017 March

    28, 2017 32 / 57
  64. Artifact Detection Break points. Olivier Lézoray ICIAE 2017 March 28,

    2017 32 / 57
  65. Artifact Detection Blending amplitude map. Olivier Lézoray ICIAE 2017 March

    28, 2017 32 / 57
  66. Artifact Detection 1 Map of the artifact location probabilities 2

    hypothesis for artifact locations : PA = ∇I . div(φ) • local sharp variations in the inpainted image (high gradient) • source patches come from very different locations (discontinuities in the shift map φ) 2 Strongest artifact locations LA = {p | PA(p) > τ} 3 Blending amplitude map σ(p) = q∈LA w(p,q) max r∈I q∈LA w(p,q) with w(p, q) = exp − p−q 2 PA(q)2 (7) Olivier Lézoray ICIAE 2017 March 28, 2017 33 / 57
  67. Patch Blending Spatial blending of all the patches that overlap

    at a given pixel p Here : compute a weighted average of p1, p2 and p3 Weights depend on distance between patch centers and pixel to blend. Olivier Lézoray ICIAE 2017 March 28, 2017 34 / 57
  68. Patch Blending Weighted sum of overlapping patches pixels values J(p)

    = ψq∈Ψp w(p, q) . ψq (p − q) ψq∈Ψp w(q, p) (8) • w(p, q) = exp − p−q 2 σ(p)2 • Ψp = {ψq | ψq ∩ ψp = ∅} (set of patches that overlap at p) → one gaussian function for each possible blending amplitude ⇒ quite slow (convolution with a spatially variant kernel) ⇒ use a quantization of σ to make it fast Olivier Lézoray ICIAE 2017 March 28, 2017 35 / 57
  69. Result Masked image. Olivier Lézoray ICIAE 2017 March 28, 2017

    36 / 57
  70. Result Inpainted image Olivier Lézoray ICIAE 2017 March 28, 2017

    37 / 57
  71. Result Inpainted Image + Spatial patch blending Olivier Lézoray ICIAE

    2017 March 28, 2017 38 / 57
  72. block-effect artifacts ? Masked Image Olivier Lézoray ICIAE 2017 March

    28, 2017 39 / 57
  73. block-effect artifacts ? Inpainted Image Olivier Lézoray ICIAE 2017 March

    28, 2017 39 / 57
  74. block-effect artifacts ? Blended Image Olivier Lézoray ICIAE 2017 March

    28, 2017 39 / 57
  75. Original Image Olivier Lézoray ICIAE 2017 March 28, 2017 40

    / 57
  76. Inpainted with improved version of [Criminisi et al’03] Olivier Lézoray

    ICIAE 2017 March 28, 2017 40 / 57
  77. After patch blending Olivier Lézoray ICIAE 2017 March 28, 2017

    40 / 57
  78. Content Aware Filling (Photoshop) Olivier Lézoray ICIAE 2017 March 28,

    2017 40 / 57
  79. Geometry-guided patch blending : process Idea : Is it possible

    to take the local geometry of the image contours into account ? Anisotropic patch blending • For flat areas (λS1 ≈ λS2 ) : large isotropic blending tensors (as structure tensors) for smoothing • For structured areas (λS1 λS2 ) : small anisotropic blending tensors to preserve sharp structures Olivier Lézoray ICIAE 2017 March 28, 2017 41 / 57
  80. Geometry-guided patch blending Weighted sum of overlapping patches pixels values

    using blending with w(p, q) =    exp − p−q 2 σ(p)2 isotropic exp XT B(p)−1X 2σ2 B anisotropic where X = q − p Olivier Lézoray ICIAE 2017 March 28, 2017 42 / 57
  81. Geometry-guided patch blending : process Example of blending tensor field

    Olivier Lézoray ICIAE 2017 March 28, 2017 43 / 57
  82. block-effect artifacts ? Masked Image Olivier Lézoray ICIAE 2017 March

    28, 2017 44 / 57
  83. block-effect artifacts ? Inpainted Image Olivier Lézoray ICIAE 2017 March

    28, 2017 44 / 57
  84. block-effect artifacts ? Blended Image (isotropic) Olivier Lézoray ICIAE 2017

    March 28, 2017 44 / 57
  85. block-effect artifacts ? Blended Image (geometry-guided) Olivier Lézoray ICIAE 2017

    March 28, 2017 44 / 57
  86. Woman Masked image Olivier Lézoray ICIAE 2017 March 28, 2017

    45 / 57
  87. Woman Without blending Olivier Lézoray ICIAE 2017 March 28, 2017

    45 / 57
  88. Woman Isotropic patch blending Olivier Lézoray ICIAE 2017 March 28,

    2017 45 / 57
  89. Woman Geometry-guided patch blending Olivier Lézoray ICIAE 2017 March 28,

    2017 45 / 57
  90. Outline 1 Introduction 2 Inpainting approaches 3 Improvements of Exemplar-based

    inpainting 4 Geometry guided patch blending for artifact reduction 5 Extension to other modalities Video inpainting Reconstruction of re-synthetized stereoscopic views Olivier Lézoray ICIAE 2017 March 28, 2017 46 / 57
  91. Video inpainting • priorities computed frame by frame • patches

    and lookup windows are now in 3D • blending both in space and time Olivier Lézoray ICIAE 2017 March 28, 2017 47 / 57
  92. Masked video : (x,y) and (x,t) plans Olivier Lézoray ICIAE

    2017 March 28, 2017 48 / 57
  93. Inpainted video (without blending) Olivier Lézoray ICIAE 2017 March 28,

    2017 48 / 57
  94. Blended Image (geometry-guided) Olivier Lézoray ICIAE 2017 March 28, 2017

    48 / 57
  95. Inpainted video Olivier Lézoray ICIAE 2017 March 28, 2017 49

    / 57
  96. Outline 1 Introduction 2 Inpainting approaches 3 Improvements of Exemplar-based

    inpainting 4 Geometry guided patch blending for artifact reduction 5 Extension to other modalities Video inpainting Reconstruction of re-synthetized stereoscopic views Olivier Lézoray ICIAE 2017 March 28, 2017 50 / 57
  97. The problem right view Olivier Lézoray ICIAE 2017 March 28,

    2017 51 / 57
  98. The problem synthesized color image original view synthesized depth original

    depth Olivier Lézoray ICIAE 2017 March 28, 2017 52 / 57
  99. The problem synthesized color image (Is) original view (Io) synthesized

    depth (inpainted) (Js) original depth Olivier Lézoray ICIAE 2017 March 28, 2017 53 / 57
  100. Inpainting with a depth map We use our improved [Criminisi

    et al’03] and by taking into account the depth : • Count as reliable pixels with similar depth Hole interior data term confidence priority boundaries term term Olivier Lézoray ICIAE 2017 March 28, 2017 54 / 57
  101. Inpainting with a depth map We use our improved [Criminisi

    et al’03] and by taking into account the depth : • modify the lookup strategy to look for patches with similar depth Is Io Olivier Lézoray ICIAE 2017 March 28, 2017 54 / 57
  102. Inpainting with a depth map We use our improved [Criminisi

    et al’03] and by taking into account the depth : • Paste only pixels with equal depth Ψt in Is Ψt in Js Ψˆ t (p) in Io Result Olivier Lézoray ICIAE 2017 March 28, 2017 54 / 57
  103. Olivier Lézoray ICIAE 2017 March 28, 2017 55 / 57

  104. Olivier Lézoray ICIAE 2017 March 28, 2017 55 / 57

  105. Olivier Lézoray ICIAE 2017 March 28, 2017 55 / 57

  106. Video illustrating the inpainting Olivier Lézoray ICIAE 2017 March 28,

    2017 56 / 57
  107. The end Publications available at : https://lezoray.users.greyc.fr Olivier Lézoray ICIAE

    2017 March 28, 2017 57 / 57