Improvements of Exemplar-based Image Inpainting
Olivier Lézoray
with Maxime Daisy, Pierre Buyssens, David Tschumperlé
Normandie Univ, UNICAEN, ENSICAEN, CNRS, GREYC, 14000 Caen, France
ICIAE 2017
March 28, 2017
Slide 2
Slide 2 text
Overview
1 Introduction
2 Inpainting approaches
3 Improvements of Exemplar-based inpainting
4 Geometry guided patch blending for artifact reduction
5 Extension to other modalities
Video inpainting
Reconstruction of re-synthetized stereoscopic views
Olivier Lézoray ICIAE 2017 March 28, 2017 2 / 57
Slide 3
Slide 3 text
Outline
1 Introduction
2 Inpainting approaches
3 Improvements of Exemplar-based inpainting
4 Geometry guided patch blending for artifact reduction
5 Extension to other modalities
Olivier Lézoray ICIAE 2017 March 28, 2017 3 / 57
Slide 4
Slide 4 text
What is inpainting ?
Olivier Lézoray ICIAE 2017 March 28, 2017 4 / 57
Slide 5
Slide 5 text
What is inpainting ?
Olivier Lézoray ICIAE 2017 March 28, 2017 4 / 57
Slide 6
Slide 6 text
What is inpainting ?
Olivier Lézoray ICIAE 2017 March 28, 2017 4 / 57
Slide 7
Slide 7 text
Why is it a hard problem ?
Olivier Lézoray ICIAE 2017 March 28, 2017 5 / 57
Slide 8
Slide 8 text
Why is it a hard problem ?
Olivier Lézoray ICIAE 2017 March 28, 2017 5 / 57
Slide 9
Slide 9 text
Why is it a hard problem ?
Olivier Lézoray ICIAE 2017 March 28, 2017 5 / 57
Slide 10
Slide 10 text
How to do it manually ?
Use the clone tool
Olivier Lézoray ICIAE 2017 March 28, 2017 6 / 57
Slide 11
Slide 11 text
How to do it manually ?
Olivier Lézoray ICIAE 2017 March 28, 2017 6 / 57
Slide 12
Slide 12 text
How to do it manually ?
Olivier Lézoray ICIAE 2017 March 28, 2017 6 / 57
Slide 13
Slide 13 text
How to do it manually ?
Olivier Lézoray ICIAE 2017 March 28, 2017 6 / 57
Slide 14
Slide 14 text
How to do it manually ?
Olivier Lézoray ICIAE 2017 March 28, 2017 6 / 57
Slide 15
Slide 15 text
How to do it manually ?
Olivier Lézoray ICIAE 2017 March 28, 2017 6 / 57
Slide 16
Slide 16 text
Outline
1 Introduction
2 Inpainting approaches
3 Improvements of Exemplar-based inpainting
4 Geometry guided patch blending for artifact reduction
5 Extension to other modalities
Olivier Lézoray ICIAE 2017 March 28, 2017 7 / 57
Slide 17
Slide 17 text
Geometry-based approaches
Purely geometry-driven methods
[Masnou et al. ’98, Bertalmio et al. ’00, Chan et al. ’01, Tschumperlé
and Deriche ’05, . . .]
Idea : Use a diffusion equation guided by the isophotes directions
→ performs boundary geometry interpolation
• Good global geometry reconstruction
• Expressed with an energy functional to minimize
• Not able to create complex textures
• Time consuming
Olivier Lézoray ICIAE 2017 March 28, 2017 8 / 57
Inpainted with [Tschumperlé and Deriche ’05]
Olivier Lézoray ICIAE 2017 March 28, 2017 9 / 57
Slide 20
Slide 20 text
Exemplar-based approaches
Pattern-based methods
[Efros and Leung’99, Criminisi et al. ’04, Lemeur et al. ’11, . . .]
Idea : exploit texture auto-similarity and local analysis of image
geometry
→ copy/paste of image chunks
• Large area reconstruction
• Quite fast
• Not enough local/global coherence e.g. block-effect artifacts
⇒ Best compromise between time and quality
Olivier Lézoray ICIAE 2017 March 28, 2017 10 / 57
Slide 21
Slide 21 text
Exemplar-based approaches
Criminisi et al. 2004, "Region Filling and Object Removal by Exemplar-Based Image Inpainting"
C : Confidence term (reliable information for reach patch)
D : Data term (local structure estimation)
Olivier Lézoray ICIAE 2017 March 28, 2017 11 / 57
Slide 22
Slide 22 text
Exemplar-based approaches
Criminisi et al. 2004, "Region Filling and Object Removal by Exemplar-Based Image Inpainting"
C : Confidence term (reliable information for reach patch)
D : Data term (local structure estimation)
Olivier Lézoray ICIAE 2017 March 28, 2017 11 / 57
Slide 23
Slide 23 text
Exemplar-based approaches
Criminisi et al. 2004, "Region Filling and Object Removal by Exemplar-Based Image Inpainting"
C : Confidence term (reliable information for reach patch)
D : Data term (local structure estimation)
Olivier Lézoray ICIAE 2017 March 28, 2017 11 / 57
Inpainted with [Criminisi et al. ’04]
Olivier Lézoray ICIAE 2017 March 28, 2017 12 / 57
Slide 26
Slide 26 text
Fast patch search
PatchMatch [Barnes et al’09]
• A very fast algorithm for matching similar patchs between two
images
• Works for full images (non-masked patchs) and is stochastic
Used in several efficient image and video inpainting techniques
(multi-scale approaches) :
• [Wexler et al. ’07] + PatchMatch : available in PhotoShop
• The solution at one scale is used as an initialization for the
upper scale (may increase error propagation)
Olivier Lézoray ICIAE 2017 March 28, 2017 13 / 57
Slide 27
Slide 27 text
[Wexler et al. ’07]
Inpainted with [Wexler et al. ’07]
Olivier Lézoray ICIAE 2017 March 28, 2017 14 / 57
Slide 28
Slide 28 text
Key-steps for improvements
Two key steps can be considered for improvements :
• The analysis of the image geometry
• The analysis of the best patches to copy/paste
Olivier Lézoray ICIAE 2017 March 28, 2017 15 / 57
Slide 29
Slide 29 text
Key-steps for improvements
Two key steps can be considered for improvements :
• The analysis of the image geometry
• The analysis of the best patches to copy/paste
• Problems with curved structures to reconstruct
Olivier Lézoray ICIAE 2017 March 28, 2017 15 / 57
Slide 30
Slide 30 text
Key-steps for improvements
Two key steps can be considered for improvements :
• The analysis of the image geometry
• The analysis of the best patches to copy/paste
• Problems with curved structures to reconstruct
• Visible bloc effects
Olivier Lézoray ICIAE 2017 March 28, 2017 15 / 57
Slide 31
Slide 31 text
Key-steps for improvements
Two key steps can be considered for improvements :
• The analysis of the image geometry
• The analysis of the best patches to copy/paste
• Problems with curved structures to reconstruct
• Visible bloc effects
Olivier Lézoray ICIAE 2017 March 28, 2017 15 / 57
Slide 32
Slide 32 text
Outline
1 Introduction
2 Inpainting approaches
3 Improvements of Exemplar-based inpainting
4 Geometry guided patch blending for artifact reduction
5 Extension to other modalities
Olivier Lézoray ICIAE 2017 March 28, 2017 16 / 57
Slide 33
Slide 33 text
Exemplar-based Inpainting Algorithm improvements
Proposed improvements
1 priority accuracy enhancement
⇒ more global geometry consistency
2 better lookup strategy
⇒ more local geometry consistency
3 spatial patch blending
⇒ strongly reduced block-effect artifacts
Olivier Lézoray ICIAE 2017 March 28, 2017 17 / 57
Slide 34
Slide 34 text
A better data term accuracy
Criminisi et al. data term
Dp
=
|
−
→
∇Ip
⊥ .
−
→
np|
α
(1)
Dp favors the reconstruction of local structures that are orthogonal to the mask
"The gradient ∇Ip is computed as the maximum value of the image gradient in Ψp ∩ I."
−
→
∇I⊥
p
= {
−
→
∇I⊥
q
| arg max
q∈((I−Ω)∩ψp)
−
→
∇Iq } (isophote direction) (2)
Olivier Lézoray ICIAE 2017 March 28, 2017 18 / 57
Slide 35
Slide 35 text
A better data term accuracy
Criminisi et al. data term
Dp
=
|
−
→
∇Ip
⊥ .
−
→
np|
α
(1)
Dp favors the reconstruction of local structures that are orthogonal to the mask
"The gradient ∇Ip is computed as the maximum value of the image gradient in Ψp ∩ I."
−
→
∇I⊥
p
= {
−
→
∇I⊥
q
| arg max
q∈((I−Ω)∩ψp)
−
→
∇Iq } (isophote direction) (2)
Olivier Lézoray ICIAE 2017 March 28, 2017 18 / 57
Slide 36
Slide 36 text
Structure tensors
Structure tensors provide a description of the local geometry for
vector images :
Sp
=
N
k=1
−
→
∇Ik
(p)
−
→
∇Ik
(p)T (3)
Structure tensors are symmetric and positive matrices (2 × 2) that
can be decomposed with their eigen decomposition in
Sp
=
N
k=1
λk ek eT
k
(4)
Tensor is said anisotropic when λ1 λ2
Olivier Lézoray ICIAE 2017 March 28, 2017 19 / 57
Slide 37
Slide 37 text
A better data term accuracy
Data term based on the smoothed structure tensor field
˜
Dp
= Gp .
−
→
n p (5)
with
Gp
=
q∈ψp∩(I−Ω)
wp
(q)
−
→
∇Iq
−
→
∇IT
q
(6)
wp : normalized 2d gaussian centered at p
High data term for anisotropic tensors
Olivier Lézoray ICIAE 2017 March 28, 2017 20 / 57
Slide 38
Slide 38 text
A better data term accuracy
Data term based on the smoothed structure tensor field
˜
Dp
= Gp .
−
→
n p (5)
with
Gp
=
q∈ψp∩(I−Ω)
wp
(q)
−
→
∇Iq
−
→
∇IT
q
(6)
wp : normalized 2d gaussian centered at p
High data term for anisotropic tensors
Olivier Lézoray ICIAE 2017 March 28, 2017 20 / 57
Slide 39
Slide 39 text
Example
Original image
Olivier Lézoray ICIAE 2017 March 28, 2017 21 / 57
Slide 40
Slide 40 text
Example
Original data term from [Criminisi et al. ’03]
Olivier Lézoray ICIAE 2017 March 28, 2017 21 / 57
Slide 41
Slide 41 text
Example
Our data term
Olivier Lézoray ICIAE 2017 March 28, 2017 21 / 57
Slide 42
Slide 42 text
Example
Inpainting result with original data term from [Criminisi et al. ’03]
Olivier Lézoray ICIAE 2017 March 28, 2017 21 / 57
Slide 43
Slide 43 text
Example
Inpainting result with our data term
Olivier Lézoray ICIAE 2017 March 28, 2017 21 / 57
Slide 44
Slide 44 text
A better lookup strategy
Criminisi et al. 2004 lookup strategy
• window search ?
⇒ faster
⇒ much less global
• all over the image search ?
⇒ slow
⇒ not so much good results
Olivier Lézoray ICIAE 2017 March 28, 2017 22 / 57
Slide 45
Slide 45 text
A better lookup strategy
Window search
Iteration 0 : the number of available patches is sufficient to find a
good matching
Olivier Lézoray ICIAE 2017 March 28, 2017 23 / 57
Slide 46
Slide 46 text
A better lookup strategy
Window search
Iteration n : the number of available patches is not sufficient to find a
good matching
Olivier Lézoray ICIAE 2017 March 28, 2017 23 / 57
Slide 47
Slide 47 text
A better lookup strategy
Our lookup strategy
• inspired from [Ashikhmin ’01, PatchMatch ’09]
• local/global search scheme
• use search sites of surrounding inpainted patches
• search site sizes are inversely proportional to the number of
sites
⇒ more local coherence
Olivier Lézoray ICIAE 2017 March 28, 2017 24 / 57
Slide 48
Slide 48 text
A better lookup strategy
• The set of already pasted patches around one pixel is
determined (if no : use classical scheme)
• Search windows are added around original pasted patches
• most search sites overlap, and matching is faster
• patch size becomes less critical
Olivier Lézoray ICIAE 2017 March 28, 2017 25 / 57
Slide 49
Slide 49 text
Joshua Tree - 512 × 384
Masked Image
Olivier Lézoray ICIAE 2017 March 28, 2017 26 / 57
Slide 50
Slide 50 text
Joshua Tree - 512 × 384
Windowed search (4.5s)
Olivier Lézoray ICIAE 2017 March 28, 2017 26 / 57
Slide 51
Slide 51 text
Joshua Tree - 512 × 384
With smart search (3.2s)
Olivier Lézoray ICIAE 2017 March 28, 2017 26 / 57
Spatial Patch Blending
Inpainted with [Criminisi et al. ’04]
Olivier Lézoray ICIAE 2017 March 28, 2017 30 / 57
Slide 59
Slide 59 text
Spatial Patch Blending
Inpainted with [Criminisi et al. ’04] + Our spatial patch blending
Olivier Lézoray ICIAE 2017 March 28, 2017 30 / 57
Slide 60
Slide 60 text
Patch Blending : method
Idea : Visually minimize block artifacts due to side-by-side pasted
patches.
• Remember the location of the pasted patches
• Detect the locations of the artifacts
• Spatial blending of the patches
Olivier Lézoray ICIAE 2017 March 28, 2017 31 / 57
Artifact Detection
1 Map of the artifact location probabilities
2 hypothesis for artifact locations : PA = ∇I . div(φ)
• local sharp variations in the inpainted image (high gradient)
• source patches come from very different locations
(discontinuities in the shift map φ)
2 Strongest artifact locations
LA = {p | PA(p) > τ}
3 Blending amplitude map
σ(p) = q∈LA
w(p,q)
max
r∈I q∈LA
w(p,q)
with w(p, q) = exp − p−q 2
PA(q)2
(7)
Olivier Lézoray ICIAE 2017 March 28, 2017 33 / 57
Slide 67
Slide 67 text
Patch Blending
Spatial blending of all the patches that overlap at a given pixel p
Here : compute a weighted average of p1, p2 and p3
Weights depend on distance between patch centers and pixel to
blend.
Olivier Lézoray ICIAE 2017 March 28, 2017 34 / 57
Slide 68
Slide 68 text
Patch Blending
Weighted sum of overlapping patches pixels values
J(p) = ψq∈Ψp
w(p, q) . ψq
(p − q)
ψq∈Ψp
w(q, p)
(8)
• w(p, q) = exp − p−q 2
σ(p)2
• Ψp
= {ψq | ψq ∩ ψp
= ∅} (set of patches that overlap at p)
→ one gaussian function for each possible blending amplitude
⇒ quite slow (convolution with a spatially variant kernel)
⇒ use a quantization of σ to make it fast
Olivier Lézoray ICIAE 2017 March 28, 2017 35 / 57
Slide 69
Slide 69 text
Result
Masked image.
Olivier Lézoray ICIAE 2017 March 28, 2017 36 / 57
Slide 70
Slide 70 text
Result
Inpainted image
Olivier Lézoray ICIAE 2017 March 28, 2017 37 / 57
Slide 71
Slide 71 text
Result
Inpainted Image + Spatial patch blending
Olivier Lézoray ICIAE 2017 March 28, 2017 38 / 57
Geometry-guided patch blending : process
Idea : Is it possible to take the local geometry of the image contours
into account ? Anisotropic patch blending
• For flat areas (λS1
≈ λS2
) : large isotropic blending tensors (as
structure tensors) for smoothing
• For structured areas (λS1
λS2
) : small anisotropic blending
tensors to preserve sharp structures
Olivier Lézoray ICIAE 2017 March 28, 2017 41 / 57
Slide 80
Slide 80 text
Geometry-guided patch blending
Weighted sum of overlapping patches pixels values using blending
with
w(p, q) =
exp − p−q 2
σ(p)2
isotropic
exp XT B(p)−1X
2σ2
B
anisotropic
where X = q − p
Olivier Lézoray ICIAE 2017 March 28, 2017 42 / 57
Slide 81
Slide 81 text
Geometry-guided patch blending : process
Example of blending tensor field
Olivier Lézoray ICIAE 2017 March 28, 2017 43 / 57
Outline
1 Introduction
2 Inpainting approaches
3 Improvements of Exemplar-based inpainting
4 Geometry guided patch blending for artifact reduction
5 Extension to other modalities
Video inpainting
Reconstruction of re-synthetized stereoscopic views
Olivier Lézoray ICIAE 2017 March 28, 2017 46 / 57
Slide 91
Slide 91 text
Video inpainting
• priorities computed frame by frame
• patches and lookup windows are now in 3D
• blending both in space and time
Olivier Lézoray ICIAE 2017 March 28, 2017 47 / 57
Slide 92
Slide 92 text
Masked video : (x,y) and (x,t) plans
Olivier Lézoray ICIAE 2017 March 28, 2017 48 / 57
Slide 93
Slide 93 text
Inpainted video (without blending)
Olivier Lézoray ICIAE 2017 March 28, 2017 48 / 57
Inpainted video
Olivier Lézoray ICIAE 2017 March 28, 2017 49 / 57
Slide 96
Slide 96 text
Outline
1 Introduction
2 Inpainting approaches
3 Improvements of Exemplar-based inpainting
4 Geometry guided patch blending for artifact reduction
5 Extension to other modalities
Video inpainting
Reconstruction of re-synthetized stereoscopic views
Olivier Lézoray ICIAE 2017 March 28, 2017 50 / 57
Slide 97
Slide 97 text
The problem
right view
Olivier Lézoray ICIAE 2017 March 28, 2017 51 / 57
Slide 98
Slide 98 text
The problem
synthesized color image original view
synthesized depth original depth
Olivier Lézoray ICIAE 2017 March 28, 2017 52 / 57
Slide 99
Slide 99 text
The problem
synthesized color image (Is) original view (Io)
synthesized depth (inpainted) (Js) original depth
Olivier Lézoray ICIAE 2017 March 28, 2017 53 / 57
Slide 100
Slide 100 text
Inpainting with a depth map
We use our improved [Criminisi et al’03] and by taking into account
the depth :
• Count as reliable pixels with similar depth
Hole interior data term confidence priority
boundaries term term
Olivier Lézoray ICIAE 2017 March 28, 2017 54 / 57
Slide 101
Slide 101 text
Inpainting with a depth map
We use our improved [Criminisi et al’03] and by taking into account
the depth :
• modify the lookup strategy to look for patches with similar depth
Is Io
Olivier Lézoray ICIAE 2017 March 28, 2017 54 / 57
Slide 102
Slide 102 text
Inpainting with a depth map
We use our improved [Criminisi et al’03] and by taking into account
the depth :
• Paste only pixels with equal depth
Ψt in Is
Ψt in Js
Ψˆ
t
(p) in Io Result
Olivier Lézoray ICIAE 2017 March 28, 2017 54 / 57
Slide 103
Slide 103 text
Olivier Lézoray ICIAE 2017 March 28, 2017 55 / 57
Slide 104
Slide 104 text
Olivier Lézoray ICIAE 2017 March 28, 2017 55 / 57
Slide 105
Slide 105 text
Olivier Lézoray ICIAE 2017 March 28, 2017 55 / 57
Slide 106
Slide 106 text
Video illustrating the inpainting
Olivier Lézoray ICIAE 2017 March 28, 2017 56 / 57
Slide 107
Slide 107 text
The end
Publications available at :
https://lezoray.users.greyc.fr
Olivier Lézoray ICIAE 2017 March 28, 2017 57 / 57