Share
Embed
Transcript
77 slides
-
Blending Methods and Other Improvements for Exemplar-based Image Inpainting Techniques
Maxime Daisy, Pierre Buyssens, David Tschumperlé and Olivier Lézoray GREYC - CNRS UMR 6072, Image team 9th of April 2015 -
Context D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen
9th of April 2015 2 / 45 -
Context D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen
9th of April 2015 2 / 45 -
Context D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen
9th of April 2015 2 / 45 -
Context D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen
9th of April 2015 2 / 45 -
Context D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen
9th of April 2015 2 / 45 -
Context Geometry-based methods [Masnou et al. ’98, Bertalmio et al.
’00, Chan et al. ’01, Tschumperlé and Deriche ’03, . . .] → boundary geometry interpolation • Good global geometry reconstruction • Not able to create complex textures • Time consuming D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 3 / 45 -
Masked image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie
Rouen 9th of April 2015 4 / 45 -
Inpainted with Diffusion PDE’s [Tschumperlé and Deriche ’03] D. Tschumperlé
(GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 4 / 45 -
Context Pattern-based methods [Efros and Leung’99, Criminisi et al. ’04,
Lemeur et al. ’11, . . .] → copy/paste of image chunks • Large area reconstruction • Quite fast • Not enough local/global coherence e.g. block-effect artifacts ⇒ Best compromise between time and quality D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 5 / 45 -
Context Criminisi et al. 2004, "Region Filling and Object Removal
by Exemplar-Based Image Inpainting" D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 6 / 45 -
Masked image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie
Rouen 9th of April 2015 7 / 45 -
Inpainted with [Criminisi et al. ’04] D. Tschumperlé (GREYC, Caen
- FRANCE) Journée Imagerie Rouen 9th of April 2015 7 / 45 -
Exemplar-based inpainting : Proposed improvements D. Tschumperlé (GREYC, Caen -
FRANCE) Journée Imagerie Rouen 9th of April 2015 8 / 45 -
Exemplar-based Inpainting Algorithm Proposed improvements 1 priority accuracy enhancement ⇒
more global geometry consistency 2 better lookup statregy ⇒ more local geometry consistency 3 spatial patch blending ⇒ strongly reduced block-effect artifacts D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 9 / 45 -
A better data term accuracy Criminisi et al. data term
Dp = | − → ∇Ip ⊥ . − → np| α (1) "The gradient ∇Ip is computed as the maximum value of the image gradient in Ψp ∩ I." − → ∇I⊥ p = { − → ∇I⊥ q | arg max q∈((I−Ω)∩ψp) − → ∇Iq } (2) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 10 / 45 -
A better data term accuracy Proposed data term ˜ Dp
= Gp . − → n p (3) with Gp = q∈ψp∩(I−Ω) wp (q) − → ∇Iq − → ∇IT q (4) wp : normalized 2d gaussian centered at p D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 11 / 45 -
A better lookup strategy Criminisi et al. 2014 lookup strategy
• window search ? ⇒ faster ⇒ much less global • all over the image search ? ⇒ slow ⇒ not so much good results D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 12 / 45 -
A better lookup strategy Window search Iteration 0 D. Tschumperlé
(GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 13 / 45 -
A better lookup strategy Window search Iteration n D. Tschumperlé
(GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 13 / 45 -
A better lookup strategy Our lookup strategy • inspired from
[Ashikhmin ’01, PatchMatch ’09] • local/global search scheme • use search sites of surrounding inpainted patches ⇒ more local coherence D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 14 / 45 -
A better lookup strategy D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen 9th of April 2015 15 / 45 -
Joshua Tree - 512 × 384 Masked image D. Tschumperlé
(GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 16 / 45 -
Joshua Tree - 512 × 384 Inpainted with windowed search
(4.5s) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 16 / 45 -
Joshua Tree - 512 × 384 Inpainted with smart search
(3.2s) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 16 / 45 -
Fake Owl - 800 × 450 Masked image D. Tschumperlé
(GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 17 / 45 -
Fake Owl - 800 × 450 Inpainted with windowed search
(1.7s) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 17 / 45 -
Fake Owl - 800 × 450 Inpainted with smart search
(0.8s) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 17 / 45 -
block-effect artifacts ? D. Tschumperlé (GREYC, Caen - FRANCE) Journée
Imagerie Rouen 9th of April 2015 18 / 45 -
Spatial Patch Blending D. Tschumperlé (GREYC, Caen - FRANCE) Journée
Imagerie Rouen 9th of April 2015 19 / 45 -
Spatial Patch Blending Masked image D. Tschumperlé (GREYC, Caen -
FRANCE) Journée Imagerie Rouen 9th of April 2015 20 / 45 -
Spatial Patch Blending Inpainted with [Criminisi et al. ’04] D.
Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 20 / 45 -
Spatial Patch Blending Inpainted with diffusion PDE’s [Tschumperlé and Deriche
’03] D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 20 / 45 -
Spatial Patch Blending Inpainted with [Criminisi et al. ’04] +
Our spatial patch blending D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 20 / 45 -
Artifact Detection Masked image D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen 9th of April 2015 21 / 45 -
Artifact Detection Inpainted with [Criminisi et al. ’04] D. Tschumperlé
(GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 21 / 45 -
Artifact Detection 1 Map of the artifact location probabilities 2
hypothesis for artifact locations • local sharp variations in the inpainted image • source patches come from very different locations PA = ∇I . div(φ) φ : inpainting correspondence map 2 Strongest artifact locations LA = {p | PA(p) > τ} 3 Blending amplitude map σ(p) = ρ . q∈LA w(p,q) max r∈I q∈LA w(p,q) with w(p, q) = exp − p−q 2 PA(q)2 (5) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 22 / 45 -
Artifact Detection Inpainted image. D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen 9th of April 2015 23 / 45 -
Artifact Detection Break field image. D. Tschumperlé (GREYC, Caen -
FRANCE) Journée Imagerie Rouen 9th of April 2015 23 / 45 -
Artifact Detection Break points. D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen 9th of April 2015 23 / 45 -
Artifact Detection Blending amplitude map. D. Tschumperlé (GREYC, Caen -
FRANCE) Journée Imagerie Rouen 9th of April 2015 23 / 45 -
Patch Blending D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie
Rouen 9th of April 2015 24 / 45 -
Patch Blending Weighted sum of overlapping patches pixels values J(p)
= ψq∈Ψp w(p, q) . ψq (p − q) ψq∈Ψp w(q, p) (6) • w(p, q) = exp − p−q 2 σ(p)2 • Ψp = {ψq | ψq ∩ ψp = ∅} → one gaussian function for each possible blending amplitude ⇒ quite slow D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 25 / 45 -
A faster patch blending algorithm D. Tschumperlé (GREYC, Caen -
FRANCE) Journée Imagerie Rouen 9th of April 2015 26 / 45 -
Result Masked image D. Tschumperlé (GREYC, Caen - FRANCE) Journée
Imagerie Rouen 9th of April 2015 27 / 45 -
Result Inpainted image D. Tschumperlé (GREYC, Caen - FRANCE) Journée
Imagerie Rouen 9th of April 2015 28 / 45 -
Result Inpainted Image + Spatial patch blending D. Tschumperlé (GREYC,
Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 29 / 45 -
Block-effect artifacts ? Masked image D. Tschumperlé (GREYC, Caen -
FRANCE) Journée Imagerie Rouen 9th of April 2015 30 / 45 -
Block-effect artifacts ? Inpainted Image D. Tschumperlé (GREYC, Caen -
FRANCE) Journée Imagerie Rouen 9th of April 2015 30 / 45 -
Block-effect artifacts ? Blended Image (isotropic) D. Tschumperlé (GREYC, Caen
- FRANCE) Journée Imagerie Rouen 9th of April 2015 30 / 45 -
Geometry-guided patch blending : process Anisotropic image regularization D. Tschumperlé
(GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 31 / 45 -
Geometry-guided patch blending : process Anisotropic patch blending D. Tschumperlé
(GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 32 / 45 -
Geometry-guided patch blending 1 Structure eigen values normalization ˆ λS(p)i
= λS(p) max p∈I λS(p)i 2 Blending tensors eigen values computation λBi = 1 (1 + ˆ λS1 + ˆ λS2 )γi 3 Blending tensor building B = λσB1e⊥ S1 .T e⊥ S1 + λσB2e⊥ S2 .T e⊥ S2 (7) • eSi : structure tensor eigen vectors • λσBi = σB λBi • σB : maximum blending bandwidth D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 33 / 45 -
Geometry-guided patch blending Weighted sum of overlapping patches pixels values
using Eq. (6) with w(p, q) = exp − p−q 2 σ(p)2 isotropic exp XT B(p)−1X 2σ2 B anisotropic where X = q − p D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 34 / 45 -
block-effect artifacts ? Masked image D. Tschumperlé (GREYC, Caen -
FRANCE) Journée Imagerie Rouen 9th of April 2015 35 / 45 -
block-effect artifacts ? Inpainted image D. Tschumperlé (GREYC, Caen -
FRANCE) Journée Imagerie Rouen 9th of April 2015 35 / 45 -
block-effect artifacts ? Blended image (isotropic) D. Tschumperlé (GREYC, Caen
- FRANCE) Journée Imagerie Rouen 9th of April 2015 35 / 45 -
block-effect artifacts ? Blended image (geometry-guided) D. Tschumperlé (GREYC, Caen
- FRANCE) Journée Imagerie Rouen 9th of April 2015 35 / 45 -
Woman Masked image D. Tschumperlé (GREYC, Caen - FRANCE) Journée
Imagerie Rouen 9th of April 2015 36 / 45 -
Woman Without blending D. Tschumperlé (GREYC, Caen - FRANCE) Journée
Imagerie Rouen 9th of April 2015 36 / 45 -
Woman Isotropic patch blending D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen 9th of April 2015 36 / 45 -
Woman Geometry-guided patch blending D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen 9th of April 2015 36 / 45 -
Copter Masked image D. Tschumperlé (GREYC, Caen - FRANCE) Journée
Imagerie Rouen 9th of April 2015 37 / 45 -
Copter Copter + mask D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen 9th of April 2015 37 / 45 -
Copter Inpainted + anisotropic patch blending D. Tschumperlé (GREYC, Caen
- FRANCE) Journée Imagerie Rouen 9th of April 2015 37 / 45 -
Opera Masked image D. Tschumperlé (GREYC, Caen - FRANCE) Journée
Imagerie Rouen 9th of April 2015 38 / 45 -
Opera Inpainted + anisotropic patch blending D. Tschumperlé (GREYC, Caen
- FRANCE) Journée Imagerie Rouen 9th of April 2015 38 / 45 -
Opera Inpainted without blending [Lemeur et al’11] D. Tschumperlé (GREYC,
Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 38 / 45 -
Video Inpainting D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie
Rouen 9th of April 2015 39 / 45 -
Adaptation to videos • priorities computed frame by frame •
cobblestone patches (e.g. of size 5 × 5 × 3) and lookup windows Shown to be working in the state of the art • Wexler et al. TPAMI 2007, "Space-time completion of Videos" • Newson et al. SIAM J. Imaging sciences 2014, "Video inpainting of complex scenes" D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 40 / 45 -
Space-time artifacts Space-time patch blending D. Tschumperlé (GREYC, Caen -
FRANCE) Journée Imagerie Rouen 9th of April 2015 41 / 45 -
Adaptation to videos Space-time patch blending D. Tschumperlé (GREYC, Caen
- FRANCE) Journée Imagerie Rouen 9th of April 2015 42 / 45 -
Masked video : (x,y) and (x,t) plans D. Tschumperlé (GREYC,
Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 43 / 45 -
Inpainted video (without blending) D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen 9th of April 2015 43 / 45 -
Blended Image (geometry-guided) D. Tschumperlé (GREYC, Caen - FRANCE) Journée
Imagerie Rouen 9th of April 2015 43 / 45 -
Conclusion Contributions 1 Improvements of a reference inpainting method 2
Method to reduce block-effect artifacts 3 Adaptation of (1) and (2) to video data Recent advances • Depth-aware patch blending for stereoscopic inpainted results More results at https://daisy.users.greyc.fr/@research D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 44 / 45 -
References and publications References • Wexler et al., Space-Time Video
Completion, IEEE TPAMI 2007 • Newson et al., Video Inpainting of Complex Scenes, SIAM J. IMAGING SCIENCES 2014 Publications • Daisy M., Tschumperlé D. and Lézoray O., Spatial Patch Blending for Artefact Reduction in Pattern-Based Inpainting Techniques, CAIP’13 • Daisy M., Tschumperlé D. and Lézoray O., A Fast Spatial Patch Blending Algorithm for Artefact Reduction in Pattern-based Image Inpainting, SIGGRAPH-ASIA’13 • Daisy M., Buyssens P., Tschumperlé D. and Lézoray O., A smarter exemplar-based inpainting algorithm using local and global heuristics for more geometric coherence, ICIP’14 D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 45 / 45