Share
Embed
Transcript
107 slides
-
Improvements of Exemplar-based Image Inpainting Olivier Lézoray with Maxime Daisy,
Pierre Buyssens, David Tschumperlé Normandie Univ, UNICAEN, ENSICAEN, CNRS, GREYC, 14000 Caen, France ICIAE 2017 March 28, 2017 -
Overview 1 Introduction 2 Inpainting approaches 3 Improvements of Exemplar-based
inpainting 4 Geometry guided patch blending for artifact reduction 5 Extension to other modalities Video inpainting Reconstruction of re-synthetized stereoscopic views Olivier Lézoray ICIAE 2017 March 28, 2017 2 / 57 -
Outline 1 Introduction 2 Inpainting approaches 3 Improvements of Exemplar-based
inpainting 4 Geometry guided patch blending for artifact reduction 5 Extension to other modalities Olivier Lézoray ICIAE 2017 March 28, 2017 3 / 57 -
Why is it a hard problem ? Olivier Lézoray ICIAE
2017 March 28, 2017 5 / 57 -
Why is it a hard problem ? Olivier Lézoray ICIAE
2017 March 28, 2017 5 / 57 -
Why is it a hard problem ? Olivier Lézoray ICIAE
2017 March 28, 2017 5 / 57 -
How to do it manually ? Use the clone tool
Olivier Lézoray ICIAE 2017 March 28, 2017 6 / 57 -
How to do it manually ? Olivier Lézoray ICIAE 2017
March 28, 2017 6 / 57 -
How to do it manually ? Olivier Lézoray ICIAE 2017
March 28, 2017 6 / 57 -
How to do it manually ? Olivier Lézoray ICIAE 2017
March 28, 2017 6 / 57 -
How to do it manually ? Olivier Lézoray ICIAE 2017
March 28, 2017 6 / 57 -
How to do it manually ? Olivier Lézoray ICIAE 2017
March 28, 2017 6 / 57 -
Outline 1 Introduction 2 Inpainting approaches 3 Improvements of Exemplar-based
inpainting 4 Geometry guided patch blending for artifact reduction 5 Extension to other modalities Olivier Lézoray ICIAE 2017 March 28, 2017 7 / 57 -
Geometry-based approaches Purely geometry-driven methods [Masnou et al. ’98, Bertalmio
et al. ’00, Chan et al. ’01, Tschumperlé and Deriche ’05, . . .] Idea : Use a diffusion equation guided by the isophotes directions → performs boundary geometry interpolation • Good global geometry reconstruction • Expressed with an energy functional to minimize • Not able to create complex textures • Time consuming Olivier Lézoray ICIAE 2017 March 28, 2017 8 / 57 -
Inpainted with [Tschumperlé and Deriche ’05] Olivier Lézoray ICIAE 2017
March 28, 2017 9 / 57 -
Exemplar-based approaches Pattern-based methods [Efros and Leung’99, Criminisi et al.
’04, Lemeur et al. ’11, . . .] Idea : exploit texture auto-similarity and local analysis of image geometry → copy/paste of image chunks • Large area reconstruction • Quite fast • Not enough local/global coherence e.g. block-effect artifacts ⇒ Best compromise between time and quality Olivier Lézoray ICIAE 2017 March 28, 2017 10 / 57 -
Exemplar-based approaches Criminisi et al. 2004, "Region Filling and Object
Removal by Exemplar-Based Image Inpainting" C : Confidence term (reliable information for reach patch) D : Data term (local structure estimation) Olivier Lézoray ICIAE 2017 March 28, 2017 11 / 57 -
Exemplar-based approaches Criminisi et al. 2004, "Region Filling and Object
Removal by Exemplar-Based Image Inpainting" C : Confidence term (reliable information for reach patch) D : Data term (local structure estimation) Olivier Lézoray ICIAE 2017 March 28, 2017 11 / 57 -
Exemplar-based approaches Criminisi et al. 2004, "Region Filling and Object
Removal by Exemplar-Based Image Inpainting" C : Confidence term (reliable information for reach patch) D : Data term (local structure estimation) Olivier Lézoray ICIAE 2017 March 28, 2017 11 / 57 -
Inpainted with [Criminisi et al. ’04] Olivier Lézoray ICIAE 2017
March 28, 2017 12 / 57 -
Fast patch search PatchMatch [Barnes et al’09] • A very
fast algorithm for matching similar patchs between two images • Works for full images (non-masked patchs) and is stochastic Used in several efficient image and video inpainting techniques (multi-scale approaches) : • [Wexler et al. ’07] + PatchMatch : available in PhotoShop • The solution at one scale is used as an initialization for the upper scale (may increase error propagation) Olivier Lézoray ICIAE 2017 March 28, 2017 13 / 57 -
[Wexler et al. ’07] Inpainted with [Wexler et al. ’07]
Olivier Lézoray ICIAE 2017 March 28, 2017 14 / 57 -
Key-steps for improvements Two key steps can be considered for
improvements : • The analysis of the image geometry • The analysis of the best patches to copy/paste Olivier Lézoray ICIAE 2017 March 28, 2017 15 / 57 -
Key-steps for improvements Two key steps can be considered for
improvements : • The analysis of the image geometry • The analysis of the best patches to copy/paste • Problems with curved structures to reconstruct Olivier Lézoray ICIAE 2017 March 28, 2017 15 / 57 -
Key-steps for improvements Two key steps can be considered for
improvements : • The analysis of the image geometry • The analysis of the best patches to copy/paste • Problems with curved structures to reconstruct • Visible bloc effects Olivier Lézoray ICIAE 2017 March 28, 2017 15 / 57 -
Key-steps for improvements Two key steps can be considered for
improvements : • The analysis of the image geometry • The analysis of the best patches to copy/paste • Problems with curved structures to reconstruct • Visible bloc effects Olivier Lézoray ICIAE 2017 March 28, 2017 15 / 57 -
Outline 1 Introduction 2 Inpainting approaches 3 Improvements of Exemplar-based
inpainting 4 Geometry guided patch blending for artifact reduction 5 Extension to other modalities Olivier Lézoray ICIAE 2017 March 28, 2017 16 / 57 -
Exemplar-based Inpainting Algorithm improvements Proposed improvements 1 priority accuracy enhancement
⇒ more global geometry consistency 2 better lookup strategy ⇒ more local geometry consistency 3 spatial patch blending ⇒ strongly reduced block-effect artifacts Olivier Lézoray ICIAE 2017 March 28, 2017 17 / 57 -
A better data term accuracy Criminisi et al. data term
Dp = | − → ∇Ip ⊥ . − → np| α (1) Dp favors the reconstruction of local structures that are orthogonal to the mask "The gradient ∇Ip is computed as the maximum value of the image gradient in Ψp ∩ I." − → ∇I⊥ p = { − → ∇I⊥ q | arg max q∈((I−Ω)∩ψp) − → ∇Iq } (isophote direction) (2) Olivier Lézoray ICIAE 2017 March 28, 2017 18 / 57 -
A better data term accuracy Criminisi et al. data term
Dp = | − → ∇Ip ⊥ . − → np| α (1) Dp favors the reconstruction of local structures that are orthogonal to the mask "The gradient ∇Ip is computed as the maximum value of the image gradient in Ψp ∩ I." − → ∇I⊥ p = { − → ∇I⊥ q | arg max q∈((I−Ω)∩ψp) − → ∇Iq } (isophote direction) (2) Olivier Lézoray ICIAE 2017 March 28, 2017 18 / 57 -
Structure tensors Structure tensors provide a description of the local
geometry for vector images : Sp = N k=1 − → ∇Ik (p) − → ∇Ik (p)T (3) Structure tensors are symmetric and positive matrices (2 × 2) that can be decomposed with their eigen decomposition in Sp = N k=1 λk ek eT k (4) Tensor is said anisotropic when λ1 λ2 Olivier Lézoray ICIAE 2017 March 28, 2017 19 / 57 -
A better data term accuracy Data term based on the
smoothed structure tensor field ˜ Dp = Gp . − → n p (5) with Gp = q∈ψp∩(I−Ω) wp (q) − → ∇Iq − → ∇IT q (6) wp : normalized 2d gaussian centered at p High data term for anisotropic tensors Olivier Lézoray ICIAE 2017 March 28, 2017 20 / 57 -
A better data term accuracy Data term based on the
smoothed structure tensor field ˜ Dp = Gp . − → n p (5) with Gp = q∈ψp∩(I−Ω) wp (q) − → ∇Iq − → ∇IT q (6) wp : normalized 2d gaussian centered at p High data term for anisotropic tensors Olivier Lézoray ICIAE 2017 March 28, 2017 20 / 57 -
Example Original data term from [Criminisi et al. ’03] Olivier
Lézoray ICIAE 2017 March 28, 2017 21 / 57 -
Example Inpainting result with original data term from [Criminisi et
al. ’03] Olivier Lézoray ICIAE 2017 March 28, 2017 21 / 57 -
Example Inpainting result with our data term Olivier Lézoray ICIAE
2017 March 28, 2017 21 / 57 -
A better lookup strategy Criminisi et al. 2004 lookup strategy
• window search ? ⇒ faster ⇒ much less global • all over the image search ? ⇒ slow ⇒ not so much good results Olivier Lézoray ICIAE 2017 March 28, 2017 22 / 57 -
A better lookup strategy Window search Iteration 0 : the
number of available patches is sufficient to find a good matching Olivier Lézoray ICIAE 2017 March 28, 2017 23 / 57 -
A better lookup strategy Window search Iteration n : the
number of available patches is not sufficient to find a good matching Olivier Lézoray ICIAE 2017 March 28, 2017 23 / 57 -
A better lookup strategy Our lookup strategy • inspired from
[Ashikhmin ’01, PatchMatch ’09] • local/global search scheme • use search sites of surrounding inpainted patches • search site sizes are inversely proportional to the number of sites ⇒ more local coherence Olivier Lézoray ICIAE 2017 March 28, 2017 24 / 57 -
A better lookup strategy • The set of already pasted
patches around one pixel is determined (if no : use classical scheme) • Search windows are added around original pasted patches • most search sites overlap, and matching is faster • patch size becomes less critical Olivier Lézoray ICIAE 2017 March 28, 2017 25 / 57 -
Joshua Tree - 512 × 384 Masked Image Olivier Lézoray
ICIAE 2017 March 28, 2017 26 / 57 -
Joshua Tree - 512 × 384 Windowed search (4.5s) Olivier
Lézoray ICIAE 2017 March 28, 2017 26 / 57 -
Joshua Tree - 512 × 384 With smart search (3.2s)
Olivier Lézoray ICIAE 2017 March 28, 2017 26 / 57 -
Fake Owl - 800 × 450 Masked Image Olivier Lézoray
ICIAE 2017 March 28, 2017 27 / 57 -
Fake Owl - 800 × 450 Windowed search (1.7s) Olivier
Lézoray ICIAE 2017 March 28, 2017 27 / 57 -
Fake Owl - 800 × 450 With smart search (0.8s)
Olivier Lézoray ICIAE 2017 March 28, 2017 27 / 57 -
Outline 1 Introduction 2 Inpainting approaches 3 Improvements of Exemplar-based
inpainting 4 Geometry guided patch blending for artifact reduction 5 Extension to other modalities Olivier Lézoray ICIAE 2017 March 28, 2017 29 / 57 -
Spatial Patch Blending Inpainted with [Criminisi et al. ’04] Olivier
Lézoray ICIAE 2017 March 28, 2017 30 / 57 -
Spatial Patch Blending Inpainted with [Criminisi et al. ’04] +
Our spatial patch blending Olivier Lézoray ICIAE 2017 March 28, 2017 30 / 57 -
Patch Blending : method Idea : Visually minimize block artifacts
due to side-by-side pasted patches. • Remember the location of the pasted patches • Detect the locations of the artifacts • Spatial blending of the patches Olivier Lézoray ICIAE 2017 March 28, 2017 31 / 57 -
Artifact Detection 1 Map of the artifact location probabilities 2
hypothesis for artifact locations : PA = ∇I . div(φ) • local sharp variations in the inpainted image (high gradient) • source patches come from very different locations (discontinuities in the shift map φ) 2 Strongest artifact locations LA = {p | PA(p) > τ} 3 Blending amplitude map σ(p) = q∈LA w(p,q) max r∈I q∈LA w(p,q) with w(p, q) = exp − p−q 2 PA(q)2 (7) Olivier Lézoray ICIAE 2017 March 28, 2017 33 / 57 -
Patch Blending Spatial blending of all the patches that overlap
at a given pixel p Here : compute a weighted average of p1, p2 and p3 Weights depend on distance between patch centers and pixel to blend. Olivier Lézoray ICIAE 2017 March 28, 2017 34 / 57 -
Patch Blending Weighted sum of overlapping patches pixels values J(p)
= ψq∈Ψp w(p, q) . ψq (p − q) ψq∈Ψp w(q, p) (8) • w(p, q) = exp − p−q 2 σ(p)2 • Ψp = {ψq | ψq ∩ ψp = ∅} (set of patches that overlap at p) → one gaussian function for each possible blending amplitude ⇒ quite slow (convolution with a spatially variant kernel) ⇒ use a quantization of σ to make it fast Olivier Lézoray ICIAE 2017 March 28, 2017 35 / 57 -
Result Inpainted Image + Spatial patch blending Olivier Lézoray ICIAE
2017 March 28, 2017 38 / 57 -
Inpainted with improved version of [Criminisi et al’03] Olivier Lézoray
ICIAE 2017 March 28, 2017 40 / 57 -
Geometry-guided patch blending : process Idea : Is it possible
to take the local geometry of the image contours into account ? Anisotropic patch blending • For flat areas (λS1 ≈ λS2 ) : large isotropic blending tensors (as structure tensors) for smoothing • For structured areas (λS1 λS2 ) : small anisotropic blending tensors to preserve sharp structures Olivier Lézoray ICIAE 2017 March 28, 2017 41 / 57 -
Geometry-guided patch blending Weighted sum of overlapping patches pixels values
using blending with w(p, q) = exp − p−q 2 σ(p)2 isotropic exp XT B(p)−1X 2σ2 B anisotropic where X = q − p Olivier Lézoray ICIAE 2017 March 28, 2017 42 / 57 -
Geometry-guided patch blending : process Example of blending tensor field
Olivier Lézoray ICIAE 2017 March 28, 2017 43 / 57 -
block-effect artifacts ? Blended Image (isotropic) Olivier Lézoray ICIAE 2017
March 28, 2017 44 / 57 -
block-effect artifacts ? Blended Image (geometry-guided) Olivier Lézoray ICIAE 2017
March 28, 2017 44 / 57 -
Outline 1 Introduction 2 Inpainting approaches 3 Improvements of Exemplar-based
inpainting 4 Geometry guided patch blending for artifact reduction 5 Extension to other modalities Video inpainting Reconstruction of re-synthetized stereoscopic views Olivier Lézoray ICIAE 2017 March 28, 2017 46 / 57 -
Video inpainting • priorities computed frame by frame • patches
and lookup windows are now in 3D • blending both in space and time Olivier Lézoray ICIAE 2017 March 28, 2017 47 / 57 -
Masked video : (x,y) and (x,t) plans Olivier Lézoray ICIAE
2017 March 28, 2017 48 / 57 -
Outline 1 Introduction 2 Inpainting approaches 3 Improvements of Exemplar-based
inpainting 4 Geometry guided patch blending for artifact reduction 5 Extension to other modalities Video inpainting Reconstruction of re-synthetized stereoscopic views Olivier Lézoray ICIAE 2017 March 28, 2017 50 / 57 -
The problem synthesized color image original view synthesized depth original
depth Olivier Lézoray ICIAE 2017 March 28, 2017 52 / 57 -
The problem synthesized color image (Is) original view (Io) synthesized
depth (inpainted) (Js) original depth Olivier Lézoray ICIAE 2017 March 28, 2017 53 / 57 -
Inpainting with a depth map We use our improved [Criminisi
et al’03] and by taking into account the depth : • Count as reliable pixels with similar depth Hole interior data term confidence priority boundaries term term Olivier Lézoray ICIAE 2017 March 28, 2017 54 / 57 -
Inpainting with a depth map We use our improved [Criminisi
et al’03] and by taking into account the depth : • modify the lookup strategy to look for patches with similar depth Is Io Olivier Lézoray ICIAE 2017 March 28, 2017 54 / 57 -
Inpainting with a depth map We use our improved [Criminisi
et al’03] and by taking into account the depth : • Paste only pixels with equal depth Ψt in Is Ψt in Js Ψˆ t (p) in Io Result Olivier Lézoray ICIAE 2017 March 28, 2017 54 / 57 -
The end Publications available at : https://lezoray.users.greyc.fr Olivier Lézoray ICIAE
2017 March 28, 2017 57 / 57