et al. ’00, Chan et al. ’01, Tschumperlé and Deriche ’05, . . .] Idea : Use a diffusion equation guided by the isophotes directions → performs boundary geometry interpolation • Good global geometry reconstruction • Expressed with an energy functional to minimize • Not able to create complex textures • Time consuming Olivier Lézoray ICIAE 2017 March 28, 2017 8 / 57
’04, Lemeur et al. ’11, . . .] Idea : exploit texture auto-similarity and local analysis of image geometry → copy/paste of image chunks • Large area reconstruction • Quite fast • Not enough local/global coherence e.g. block-effect artifacts ⇒ Best compromise between time and quality Olivier Lézoray ICIAE 2017 March 28, 2017 10 / 57
Removal by Exemplar-Based Image Inpainting" C : Conﬁdence term (reliable information for reach patch) D : Data term (local structure estimation) Olivier Lézoray ICIAE 2017 March 28, 2017 11 / 57
Removal by Exemplar-Based Image Inpainting" C : Conﬁdence term (reliable information for reach patch) D : Data term (local structure estimation) Olivier Lézoray ICIAE 2017 March 28, 2017 11 / 57
Removal by Exemplar-Based Image Inpainting" C : Conﬁdence term (reliable information for reach patch) D : Data term (local structure estimation) Olivier Lézoray ICIAE 2017 March 28, 2017 11 / 57
fast algorithm for matching similar patchs between two images • Works for full images (non-masked patchs) and is stochastic Used in several efﬁcient image and video inpainting techniques (multi-scale approaches) : • [Wexler et al. ’07] + PatchMatch : available in PhotoShop • The solution at one scale is used as an initialization for the upper scale (may increase error propagation) Olivier Lézoray ICIAE 2017 March 28, 2017 13 / 57
improvements : • The analysis of the image geometry • The analysis of the best patches to copy/paste • Problems with curved structures to reconstruct Olivier Lézoray ICIAE 2017 March 28, 2017 15 / 57
improvements : • The analysis of the image geometry • The analysis of the best patches to copy/paste • Problems with curved structures to reconstruct • Visible bloc effects Olivier Lézoray ICIAE 2017 March 28, 2017 15 / 57
improvements : • The analysis of the image geometry • The analysis of the best patches to copy/paste • Problems with curved structures to reconstruct • Visible bloc effects Olivier Lézoray ICIAE 2017 March 28, 2017 15 / 57
Dp = | − → ∇Ip ⊥ . − → np| α (1) Dp favors the reconstruction of local structures that are orthogonal to the mask "The gradient ∇Ip is computed as the maximum value of the image gradient in Ψp ∩ I." − → ∇I⊥ p = { − → ∇I⊥ q | arg max q∈((I−Ω)∩ψp) − → ∇Iq } (isophote direction) (2) Olivier Lézoray ICIAE 2017 March 28, 2017 18 / 57
Dp = | − → ∇Ip ⊥ . − → np| α (1) Dp favors the reconstruction of local structures that are orthogonal to the mask "The gradient ∇Ip is computed as the maximum value of the image gradient in Ψp ∩ I." − → ∇I⊥ p = { − → ∇I⊥ q | arg max q∈((I−Ω)∩ψp) − → ∇Iq } (isophote direction) (2) Olivier Lézoray ICIAE 2017 March 28, 2017 18 / 57
geometry for vector images : Sp = N k=1 − → ∇Ik (p) − → ∇Ik (p)T (3) Structure tensors are symmetric and positive matrices (2 × 2) that can be decomposed with their eigen decomposition in Sp = N k=1 λk ek eT k (4) Tensor is said anisotropic when λ1 λ2 Olivier Lézoray ICIAE 2017 March 28, 2017 19 / 57
smoothed structure tensor ﬁeld ˜ Dp = Gp . − → n p (5) with Gp = q∈ψp∩(I−Ω) wp (q) − → ∇Iq − → ∇IT q (6) wp : normalized 2d gaussian centered at p High data term for anisotropic tensors Olivier Lézoray ICIAE 2017 March 28, 2017 20 / 57
smoothed structure tensor ﬁeld ˜ Dp = Gp . − → n p (5) with Gp = q∈ψp∩(I−Ω) wp (q) − → ∇Iq − → ∇IT q (6) wp : normalized 2d gaussian centered at p High data term for anisotropic tensors Olivier Lézoray ICIAE 2017 March 28, 2017 20 / 57
• window search ? ⇒ faster ⇒ much less global • all over the image search ? ⇒ slow ⇒ not so much good results Olivier Lézoray ICIAE 2017 March 28, 2017 22 / 57
[Ashikhmin ’01, PatchMatch ’09] • local/global search scheme • use search sites of surrounding inpainted patches • search site sizes are inversely proportional to the number of sites ⇒ more local coherence Olivier Lézoray ICIAE 2017 March 28, 2017 24 / 57
patches around one pixel is determined (if no : use classical scheme) • Search windows are added around original pasted patches • most search sites overlap, and matching is faster • patch size becomes less critical Olivier Lézoray ICIAE 2017 March 28, 2017 25 / 57
due to side-by-side pasted patches. • Remember the location of the pasted patches • Detect the locations of the artifacts • Spatial blending of the patches Olivier Lézoray ICIAE 2017 March 28, 2017 31 / 57
at a given pixel p Here : compute a weighted average of p1, p2 and p3 Weights depend on distance between patch centers and pixel to blend. Olivier Lézoray ICIAE 2017 March 28, 2017 34 / 57
= ψq∈Ψp w(p, q) . ψq (p − q) ψq∈Ψp w(q, p) (8) • w(p, q) = exp − p−q 2 σ(p)2 • Ψp = {ψq | ψq ∩ ψp = ∅} (set of patches that overlap at p) → one gaussian function for each possible blending amplitude ⇒ quite slow (convolution with a spatially variant kernel) ⇒ use a quantization of σ to make it fast Olivier Lézoray ICIAE 2017 March 28, 2017 35 / 57
to take the local geometry of the image contours into account ? Anisotropic patch blending • For ﬂat areas (λS1 ≈ λS2 ) : large isotropic blending tensors (as structure tensors) for smoothing • For structured areas (λS1 λS2 ) : small anisotropic blending tensors to preserve sharp structures Olivier Lézoray ICIAE 2017 March 28, 2017 41 / 57
et al’03] and by taking into account the depth : • Count as reliable pixels with similar depth Hole interior data term conﬁdence priority boundaries term term Olivier Lézoray ICIAE 2017 March 28, 2017 54 / 57
et al’03] and by taking into account the depth : • modify the lookup strategy to look for patches with similar depth Is Io Olivier Lézoray ICIAE 2017 March 28, 2017 54 / 57
et al’03] and by taking into account the depth : • Paste only pixels with equal depth Ψt in Is Ψt in Js Ψˆ t (p) in Io Result Olivier Lézoray ICIAE 2017 March 28, 2017 54 / 57