M2NUM

Transcript

1. Blending Methods and Other Improvements for Exemplar-based Image Inpainting Techniques Maxime Daisy, Pierre Buyssens, David Tschumperlé and Olivier Lézoray GREYC - CNRS UMR 6072, Image team 9th of April 2015
2. Context D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 2 / 45
3. Context D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 2 / 45
4. Context D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 2 / 45
5. Context D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 2 / 45
6. Context D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 2 / 45
7. Context Geometry-based methods [Masnou et al. ’98, Bertalmio et al. ’00, Chan et al. ’01, Tschumperlé and Deriche ’03, . . .] → boundary geometry interpolation • Good global geometry reconstruction • Not able to create complex textures • Time consuming D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 3 / 45
8. Masked image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 4 / 45
9. Inpainted with Diffusion PDE’s [Tschumperlé and Deriche ’03] D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 4 / 45
10. Context Pattern-based methods [Efros and Leung’99, Criminisi et al. ’04, Lemeur et al. ’11, . . .] → copy/paste of image chunks • Large area reconstruction • Quite fast • Not enough local/global coherence e.g. block-effect artifacts ⇒ Best compromise between time and quality D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 5 / 45
11. Context Criminisi et al. 2004, "Region Filling and Object Removal by Exemplar-Based Image Inpainting" D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 6 / 45
12. Masked image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 7 / 45
13. Inpainted with [Criminisi et al. ’04] D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 7 / 45
14. Exemplar-based inpainting : Proposed improvements D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 8 / 45
15. Exemplar-based Inpainting Algorithm Proposed improvements 1 priority accuracy enhancement ⇒ more global geometry consistency 2 better lookup statregy ⇒ more local geometry consistency 3 spatial patch blending ⇒ strongly reduced block-effect artifacts D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 9 / 45
16. A better data term accuracy Criminisi et al. data term Dp = | − → ∇Ip ⊥ . − → np| α (1) "The gradient ∇Ip is computed as the maximum value of the image gradient in Ψp ∩ I." − → ∇I⊥ p = { − → ∇I⊥ q | arg max q∈((I−Ω)∩ψp) − → ∇Iq } (2) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 10 / 45
17. A better data term accuracy Proposed data term ˜ Dp = Gp . − → n p (3) with Gp = q∈ψp∩(I−Ω) wp (q) − → ∇Iq − → ∇IT q (4) wp : normalized 2d gaussian centered at p D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 11 / 45
18. A better lookup strategy Criminisi et al. 2014 lookup strategy • window search ? ⇒ faster ⇒ much less global • all over the image search ? ⇒ slow ⇒ not so much good results D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 12 / 45
19. A better lookup strategy Window search Iteration 0 D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 13 / 45
20. A better lookup strategy Window search Iteration n D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 13 / 45
21. A better lookup strategy Our lookup strategy • inspired from [Ashikhmin ’01, PatchMatch ’09] • local/global search scheme • use search sites of surrounding inpainted patches ⇒ more local coherence D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 14 / 45
22. A better lookup strategy D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 15 / 45
23. Joshua Tree - 512 × 384 Masked image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 16 / 45
24. Joshua Tree - 512 × 384 Inpainted with windowed search (4.5s) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 16 / 45
25. Joshua Tree - 512 × 384 Inpainted with smart search (3.2s) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 16 / 45
26. Fake Owl - 800 × 450 Masked image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 17 / 45
27. Fake Owl - 800 × 450 Inpainted with windowed search (1.7s) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 17 / 45
28. Fake Owl - 800 × 450 Inpainted with smart search (0.8s) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 17 / 45
29. block-effect artifacts ? D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 18 / 45
30. Spatial Patch Blending D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 19 / 45
31. Spatial Patch Blending Masked image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 20 / 45
32. Spatial Patch Blending Inpainted with [Criminisi et al. ’04] D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 20 / 45
33. Spatial Patch Blending Inpainted with diffusion PDE’s [Tschumperlé and Deriche ’03] D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 20 / 45
34. Spatial Patch Blending Inpainted with [Criminisi et al. ’04] + Our spatial patch blending D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 20 / 45
35. Artifact Detection Masked image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 21 / 45
36. Artifact Detection Inpainted with [Criminisi et al. ’04] D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 21 / 45
37. Artifact Detection 1 Map of the artifact location probabilities 2 hypothesis for artifact locations • local sharp variations in the inpainted image • source patches come from very different locations PA = ∇I . div(φ) φ : inpainting correspondence map 2 Strongest artifact locations LA = {p | PA(p) > τ} 3 Blending amplitude map σ(p) = ρ . q∈LA w(p,q) max r∈I q∈LA w(p,q) with w(p, q) = exp − p−q 2 PA(q)2 (5) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 22 / 45
38. Artifact Detection Inpainted image. D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 23 / 45
39. Artifact Detection Break field image. D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 23 / 45
40. Artifact Detection Break points. D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 23 / 45
41. Artifact Detection Blending amplitude map. D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 23 / 45
42. Patch Blending D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 24 / 45
43. Patch Blending Weighted sum of overlapping patches pixels values J(p) = ψq∈Ψp w(p, q) . ψq (p − q) ψq∈Ψp w(q, p) (6) • w(p, q) = exp − p−q 2 σ(p)2 • Ψp = {ψq | ψq ∩ ψp = ∅} → one gaussian function for each possible blending amplitude ⇒ quite slow D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 25 / 45
44. A faster patch blending algorithm D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 26 / 45
45. Result Masked image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 27 / 45
46. Result Inpainted image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 28 / 45
47. Result Inpainted Image + Spatial patch blending D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 29 / 45
48. Block-effect artifacts ? Masked image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 30 / 45
49. Block-effect artifacts ? Inpainted Image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 30 / 45
50. Block-effect artifacts ? Blended Image (isotropic) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 30 / 45
51. Geometry-guided patch blending : process Anisotropic image regularization D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 31 / 45
52. Geometry-guided patch blending : process Anisotropic patch blending D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 32 / 45
53. Geometry-guided patch blending 1 Structure eigen values normalization ˆ λS(p)i = λS(p) max p∈I λS(p)i 2 Blending tensors eigen values computation λBi = 1 (1 + ˆ λS1 + ˆ λS2 )γi 3 Blending tensor building B = λσB1e⊥ S1 .T e⊥ S1 + λσB2e⊥ S2 .T e⊥ S2 (7) • eSi : structure tensor eigen vectors • λσBi = σB λBi • σB : maximum blending bandwidth D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 33 / 45
54. Geometry-guided patch blending Weighted sum of overlapping patches pixels values using Eq. (6) with w(p, q) =    exp − p−q 2 σ(p)2 isotropic exp XT B(p)−1X 2σ2 B anisotropic where X = q − p D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 34 / 45
55. block-effect artifacts ? Masked image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 35 / 45
56. block-effect artifacts ? Inpainted image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 35 / 45
57. block-effect artifacts ? Blended image (isotropic) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 35 / 45
58. block-effect artifacts ? Blended image (geometry-guided) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 35 / 45
59. Woman Masked image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 36 / 45
60. Woman Without blending D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 36 / 45
61. Woman Isotropic patch blending D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 36 / 45
62. Woman Geometry-guided patch blending D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 36 / 45
63. Copter Masked image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 37 / 45
64. Copter Copter + mask D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 37 / 45
65. Copter Inpainted + anisotropic patch blending D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 37 / 45
66. Opera Masked image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 38 / 45
67. Opera Inpainted + anisotropic patch blending D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 38 / 45
68. Opera Inpainted without blending [Lemeur et al’11] D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 38 / 45
69. Video Inpainting D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 39 / 45
70. Adaptation to videos • priorities computed frame by frame • cobblestone patches (e.g. of size 5 × 5 × 3) and lookup windows Shown to be working in the state of the art • Wexler et al. TPAMI 2007, "Space-time completion of Videos" • Newson et al. SIAM J. Imaging sciences 2014, "Video inpainting of complex scenes" D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 40 / 45
71. Space-time artifacts Space-time patch blending D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 41 / 45
72. Adaptation to videos Space-time patch blending D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 42 / 45
73. Masked video : (x,y) and (x,t) plans D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 43 / 45
74. Inpainted video (without blending) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 43 / 45
75. Blended Image (geometry-guided) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 43 / 45
76. Conclusion Contributions 1 Improvements of a reference inpainting method 2 Method to reduce block-effect artifacts 3 Adaptation of (1) and (2) to video data Recent advances • Depth-aware patch blending for stereoscopic inpainted results More results at https://daisy.users.greyc.fr/@research D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 44 / 45
77. References and publications References • Wexler et al., Space-Time Video Completion, IEEE TPAMI 2007 • Newson et al., Video Inpainting of Complex Scenes, SIAM J. IMAGING SCIENCES 2014 Publications • Daisy M., Tschumperlé D. and Lézoray O., Spatial Patch Blending for Artefact Reduction in Pattern-Based Inpainting Techniques, CAIP’13 • Daisy M., Tschumperlé D. and Lézoray O., A Fast Spatial Patch Blending Algorithm for Artefact Reduction in Pattern-based Image Inpainting, SIGGRAPH-ASIA’13 • Daisy M., Buyssens P., Tschumperlé D. and Lézoray O., A smarter exemplar-based inpainting algorithm using local and global heuristics for more geometric coherence, ICIP’14 D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9th of April 2015 45 / 45