Upgrade to Pro — share decks privately, control downloads, hide ads and more …

M2NUM

 M2NUM

Olivier Lézoray

May 18, 2015
Tweet

More Decks by Olivier Lézoray

Other Decks in Research

Transcript

  1. Blending Methods and Other Improvements for Exemplar-based Image Inpainting Techniques

    Maxime Daisy, Pierre Buyssens, David Tschumperlé and Olivier Lézoray GREYC - CNRS UMR 6072, Image team 9th of April 2015
  2. Context Geometry-based methods [Masnou et al. ’98, Bertalmio et al.

    ’00, Chan et al. ’01, Tschumperlé and Deriche ’03, . . .] → boundary geometry interpolation • Good global geometry reconstruction • Not able to create complex textures • Time consuming D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 3 / 45
  3. Inpainted with Diffusion PDE’s [Tschumperlé and Deriche ’03] D. Tschumperlé

    (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 4 / 45
  4. Context Pattern-based methods [Efros and Leung’99, Criminisi et al. ’04,

    Lemeur et al. ’11, . . .] → copy/paste of image chunks • Large area reconstruction • Quite fast • Not enough local/global coherence e.g. block-effect artifacts ⇒ Best compromise between time and quality D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 5 / 45
  5. Context Criminisi et al. 2004, "Region Filling and Object Removal

    by Exemplar-Based Image Inpainting" D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 6 / 45
  6. Inpainted with [Criminisi et al. ’04] D. Tschumperlé (GREYC, Caen

    - FRANCE) Journée Imagerie Rouen 9th of April 2015 7 / 45
  7. Exemplar-based inpainting : Proposed improvements D. Tschumperlé (GREYC, Caen -

    FRANCE) Journée Imagerie Rouen 9th of April 2015 8 / 45
  8. Exemplar-based Inpainting Algorithm Proposed improvements 1 priority accuracy enhancement ⇒

    more global geometry consistency 2 better lookup statregy ⇒ more local geometry consistency 3 spatial patch blending ⇒ strongly reduced block-effect artifacts D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 9 / 45
  9. A better data term accuracy Criminisi et al. data term

    Dp = | − → ∇Ip ⊥ . − → np| α (1) "The gradient ∇Ip is computed as the maximum value of the image gradient in Ψp ∩ I." − → ∇I⊥ p = { − → ∇I⊥ q | arg max q∈((I−Ω)∩ψp) − → ∇Iq } (2) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 10 / 45
  10. A better data term accuracy Proposed data term ˜ Dp

    = Gp . − → n p (3) with Gp = q∈ψp∩(I−Ω) wp (q) − → ∇Iq − → ∇IT q (4) wp : normalized 2d gaussian centered at p D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 11 / 45
  11. A better lookup strategy Criminisi et al. 2014 lookup strategy

    • window search ? ⇒ faster ⇒ much less global • all over the image search ? ⇒ slow ⇒ not so much good results D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 12 / 45
  12. A better lookup strategy Window search Iteration 0 D. Tschumperlé

    (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 13 / 45
  13. A better lookup strategy Window search Iteration n D. Tschumperlé

    (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 13 / 45
  14. A better lookup strategy Our lookup strategy • inspired from

    [Ashikhmin ’01, PatchMatch ’09] • local/global search scheme • use search sites of surrounding inpainted patches ⇒ more local coherence D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 14 / 45
  15. A better lookup strategy D. Tschumperlé (GREYC, Caen - FRANCE)

    Journée Imagerie Rouen 9th of April 2015 15 / 45
  16. Joshua Tree - 512 × 384 Masked image D. Tschumperlé

    (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 16 / 45
  17. Joshua Tree - 512 × 384 Inpainted with windowed search

    (4.5s) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 16 / 45
  18. Joshua Tree - 512 × 384 Inpainted with smart search

    (3.2s) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 16 / 45
  19. Fake Owl - 800 × 450 Masked image D. Tschumperlé

    (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 17 / 45
  20. Fake Owl - 800 × 450 Inpainted with windowed search

    (1.7s) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 17 / 45
  21. Fake Owl - 800 × 450 Inpainted with smart search

    (0.8s) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 17 / 45
  22. Spatial Patch Blending Masked image D. Tschumperlé (GREYC, Caen -

    FRANCE) Journée Imagerie Rouen 9th of April 2015 20 / 45
  23. Spatial Patch Blending Inpainted with [Criminisi et al. ’04] D.

    Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 20 / 45
  24. Spatial Patch Blending Inpainted with diffusion PDE’s [Tschumperlé and Deriche

    ’03] D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 20 / 45
  25. Spatial Patch Blending Inpainted with [Criminisi et al. ’04] +

    Our spatial patch blending D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 20 / 45
  26. Artifact Detection Masked image D. Tschumperlé (GREYC, Caen - FRANCE)

    Journée Imagerie Rouen 9th of April 2015 21 / 45
  27. Artifact Detection Inpainted with [Criminisi et al. ’04] D. Tschumperlé

    (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 21 / 45
  28. Artifact Detection 1 Map of the artifact location probabilities 2

    hypothesis for artifact locations • local sharp variations in the inpainted image • source patches come from very different locations PA = ∇I . div(φ) φ : inpainting correspondence map 2 Strongest artifact locations LA = {p | PA(p) > τ} 3 Blending amplitude map σ(p) = ρ . q∈LA w(p,q) max r∈I q∈LA w(p,q) with w(p, q) = exp − p−q 2 PA(q)2 (5) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 22 / 45
  29. Artifact Detection Inpainted image. D. Tschumperlé (GREYC, Caen - FRANCE)

    Journée Imagerie Rouen 9th of April 2015 23 / 45
  30. Artifact Detection Break field image. D. Tschumperlé (GREYC, Caen -

    FRANCE) Journée Imagerie Rouen 9th of April 2015 23 / 45
  31. Artifact Detection Break points. D. Tschumperlé (GREYC, Caen - FRANCE)

    Journée Imagerie Rouen 9th of April 2015 23 / 45
  32. Artifact Detection Blending amplitude map. D. Tschumperlé (GREYC, Caen -

    FRANCE) Journée Imagerie Rouen 9th of April 2015 23 / 45
  33. Patch Blending Weighted sum of overlapping patches pixels values J(p)

    = ψq∈Ψp w(p, q) . ψq (p − q) ψq∈Ψp w(q, p) (6) • w(p, q) = exp − p−q 2 σ(p)2 • Ψp = {ψq | ψq ∩ ψp = ∅} → one gaussian function for each possible blending amplitude ⇒ quite slow D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 25 / 45
  34. A faster patch blending algorithm D. Tschumperlé (GREYC, Caen -

    FRANCE) Journée Imagerie Rouen 9th of April 2015 26 / 45
  35. Result Inpainted Image + Spatial patch blending D. Tschumperlé (GREYC,

    Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 29 / 45
  36. Block-effect artifacts ? Masked image D. Tschumperlé (GREYC, Caen -

    FRANCE) Journée Imagerie Rouen 9th of April 2015 30 / 45
  37. Block-effect artifacts ? Inpainted Image D. Tschumperlé (GREYC, Caen -

    FRANCE) Journée Imagerie Rouen 9th of April 2015 30 / 45
  38. Block-effect artifacts ? Blended Image (isotropic) D. Tschumperlé (GREYC, Caen

    - FRANCE) Journée Imagerie Rouen 9th of April 2015 30 / 45
  39. Geometry-guided patch blending : process Anisotropic image regularization D. Tschumperlé

    (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 31 / 45
  40. Geometry-guided patch blending : process Anisotropic patch blending D. Tschumperlé

    (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 32 / 45
  41. Geometry-guided patch blending 1 Structure eigen values normalization ˆ λS(p)i

    = λS(p) max p∈I λS(p)i 2 Blending tensors eigen values computation λBi = 1 (1 + ˆ λS1 + ˆ λS2 )γi 3 Blending tensor building B = λσB1e⊥ S1 .T e⊥ S1 + λσB2e⊥ S2 .T e⊥ S2 (7) • eSi : structure tensor eigen vectors • λσBi = σB λBi • σB : maximum blending bandwidth D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 33 / 45
  42. Geometry-guided patch blending Weighted sum of overlapping patches pixels values

    using Eq. (6) with w(p, q) =    exp − p−q 2 σ(p)2 isotropic exp XT B(p)−1X 2σ2 B anisotropic where X = q − p D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 34 / 45
  43. block-effect artifacts ? Masked image D. Tschumperlé (GREYC, Caen -

    FRANCE) Journée Imagerie Rouen 9th of April 2015 35 / 45
  44. block-effect artifacts ? Inpainted image D. Tschumperlé (GREYC, Caen -

    FRANCE) Journée Imagerie Rouen 9th of April 2015 35 / 45
  45. block-effect artifacts ? Blended image (isotropic) D. Tschumperlé (GREYC, Caen

    - FRANCE) Journée Imagerie Rouen 9th of April 2015 35 / 45
  46. block-effect artifacts ? Blended image (geometry-guided) D. Tschumperlé (GREYC, Caen

    - FRANCE) Journée Imagerie Rouen 9th of April 2015 35 / 45
  47. Woman Isotropic patch blending D. Tschumperlé (GREYC, Caen - FRANCE)

    Journée Imagerie Rouen 9th of April 2015 36 / 45
  48. Woman Geometry-guided patch blending D. Tschumperlé (GREYC, Caen - FRANCE)

    Journée Imagerie Rouen 9th of April 2015 36 / 45
  49. Copter Copter + mask D. Tschumperlé (GREYC, Caen - FRANCE)

    Journée Imagerie Rouen 9th of April 2015 37 / 45
  50. Copter Inpainted + anisotropic patch blending D. Tschumperlé (GREYC, Caen

    - FRANCE) Journée Imagerie Rouen 9th of April 2015 37 / 45
  51. Opera Inpainted + anisotropic patch blending D. Tschumperlé (GREYC, Caen

    - FRANCE) Journée Imagerie Rouen 9th of April 2015 38 / 45
  52. Opera Inpainted without blending [Lemeur et al’11] D. Tschumperlé (GREYC,

    Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 38 / 45
  53. Adaptation to videos • priorities computed frame by frame •

    cobblestone patches (e.g. of size 5 × 5 × 3) and lookup windows Shown to be working in the state of the art • Wexler et al. TPAMI 2007, "Space-time completion of Videos" • Newson et al. SIAM J. Imaging sciences 2014, "Video inpainting of complex scenes" D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 40 / 45
  54. Space-time artifacts Space-time patch blending D. Tschumperlé (GREYC, Caen -

    FRANCE) Journée Imagerie Rouen 9th of April 2015 41 / 45
  55. Adaptation to videos Space-time patch blending D. Tschumperlé (GREYC, Caen

    - FRANCE) Journée Imagerie Rouen 9th of April 2015 42 / 45
  56. Masked video : (x,y) and (x,t) plans D. Tschumperlé (GREYC,

    Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 43 / 45
  57. Inpainted video (without blending) D. Tschumperlé (GREYC, Caen - FRANCE)

    Journée Imagerie Rouen 9th of April 2015 43 / 45
  58. Conclusion Contributions 1 Improvements of a reference inpainting method 2

    Method to reduce block-effect artifacts 3 Adaptation of (1) and (2) to video data Recent advances • Depth-aware patch blending for stereoscopic inpainted results More results at https://daisy.users.greyc.fr/@research D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 44 / 45
  59. References and publications References • Wexler et al., Space-Time Video

    Completion, IEEE TPAMI 2007 • Newson et al., Video Inpainting of Complex Scenes, SIAM J. IMAGING SCIENCES 2014 Publications • Daisy M., Tschumperlé D. and Lézoray O., Spatial Patch Blending for Artefact Reduction in Pattern-Based Inpainting Techniques, CAIP’13 • Daisy M., Tschumperlé D. and Lézoray O., A Fast Spatial Patch Blending Algorithm for Artefact Reduction in Pattern-based Image Inpainting, SIGGRAPH-ASIA’13 • Daisy M., Buyssens P., Tschumperlé D. and Lézoray O., A smarter exemplar-based inpainting algorithm using local and global heuristics for more geometric coherence, ICIP’14 D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 45 / 45