Siggraph Asia 2013

1. A Fast Spatial Patch Blending Algorithm for Artefact Reduction in Pattern-based Image Inpainting Maxime Daisy, David Tschumperl´ e, Olivier L´ ezoray GREYC - UMR 6072 CNRS, ENSICAEN, University of Caen Image team SIGGRAPH Asia November 21, 2013
2. Outline 1 Inpainting : Context and State of the Art 2 Contribution : Fast Multiscale Patch Blending for Inpainting 3 Results and Comparisons with the State of the Art M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 2 / 42
3. Outlines 1 Inpainting : Context and State of the Art 2 Contribution : Fast Multiscale Patch Blending for Inpainting 3 Results and Comparisons with the State of the Art M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 3 / 42
4. Image Inpainting : Overview Original M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 4 / 42
5. Image Inpainting : Overview Masked image M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 5 / 42
6. Image Inpainting : Overview Inpainted image M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 6 / 42
7. Image Inpainting : a little history • Masnou and Morel : Level lines based disocclusion, ICIP 1998 • Efros and Leung : Texture synthesis by non-parametric sampling, ICCV 1999 • Bertalm´ ıo et al. : Image Inpainting, SIGGRAPH 2000 • Criminisi et al. : Object Removal by Exemplar-based Inpainting, CVPR 2003 • Wexler et al. : Space-time video completion, IEEE PAMI 2007 • Sun et al. : Image Inpainting by Patch Propagation Using Patch Sparsity IEEE 2010 • Darabi et al. : Image Melding : Combining Inconsistent Images using Patch-based Synthesis, SIGGRAPH 2012 • . . . M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 7 / 42
8. Criminisi et al. 2003 : overview M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 8 / 42
9. Good texture reconstruction M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 9 / 42
10. Good texture reconstruction M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 10 / 42
11. Local geometric inconsistencies M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 11 / 42
12. Since Criminisi et al. 2003 inpainting algorithm Content-aware Filling, Adobe 2010 • Wexler et al. 2007 : Space-time completion of videos • Barnes et al. 2009 : PatchMatch ⇒ blurry textures Le Meur et al. 2011 • local geometry aware : structure tensor analysis • K-nearest neighbours averaging ⇒ blurry textures, geometric inconsistencies M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 12 / 42
13. Since Criminisi et al. 2003 inpainting algorithm (a) Content-aware filling (b) Le Meur et al. 2011 M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 13 / 42
14. The question How to keep better textures and geometry consistency ? M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 14 / 42
15. Outlines 1 Inpainting : Context and State of the Art 2 Contribution : Fast Multiscale Patch Blending for Inpainting 3 Results and Comparisons with the State of the Art M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 15 / 42
16. Spatial patch blending - Extreme case Masked image M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 16 / 42
17. Spatial patch blending - Extreme case Inpainted image Criminisi et al. 2003 M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 17 / 42
18. Spatial patch blending - Extreme case Diffusion PDE inpainting result Tschumperl´ e 2006 M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 18 / 42
19. Spatial patch blending - Extreme case Content-aware filling algorithm Adobe 2010 M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 19 / 42
20. Spatial patch blending - Extreme case Effect of our spatial patch blending Contribution M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 20 / 42
21. σ(p) : an adaptative blending amplitude Break field R(p) Blending amplitude field σ(p) M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 21 / 42
22. Spatial Patch Blending : principle M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 22 / 42
23. Mathematical expression • Usage of reconstruction points and patch locations, saved during any patch-based inpainting process J(p) = ∑ ψq∈Ψp w(q,p) ψq(p −q) ε + ∑ ψq∈Ψp w(q,p) • Ψp : set of patches used for the reconstruction around p • ψq(p −q) : pixel value at ((N/2,N/2)T +(p −q)) in ψq • σ(p) : blending amplitude at p • w(q,p) gaussian weight of variance σ(p)2 M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 23 / 42
24. Fast algorithm - Blend scales M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 24 / 42
25. Outlines 1 Inpainting : Context and State of the Art 2 Contribution : Fast Multiscale Patch Blending for Inpainting 3 Results and Comparisons with the State of the Art M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 25 / 42
26. Time Comparison M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 26 / 42
27. Result - Nature Original Image M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 27 / 42
28. Result - Nature Content-aware Filling M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 28 / 42
29. Result - Nature Criminisi + our spatial patch blending M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 29 / 42
30. Result - Fake Owl Original Image M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 30 / 42
31. Result - Fake Owl Criminisi et al. 2003 M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 31 / 42
32. Result - Fake Owl Content-aware Filling M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 32 / 42
33. Result - Fake Owl Criminisi + our spatial patch blending M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 33 / 42
34. Result - Savanna Original Image M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 34 / 42
35. Result - Savanna Criminisi et al. 2003 M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 35 / 42
36. Result - Savanna Content-aware Filling M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 36 / 42
37. Result - Savanna Criminisi + our spatial patch blending M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 37 / 42
38. GIMP Software Integration : G’MIC plugin M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 38 / 42
39. GIMP Software Integration : G’MIC plugin M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 39 / 42
40. GIMP Software Integration : G’MIC plugin M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 40 / 42
41. GIMP Software Integration : G’MIC plugin M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 41 / 42
42. GIMP Software Integration : G’MIC plugin • maxime.daisy@ensicaen.fr • http ://gmic.sourceforge.net/gimp.shtml • https ://daisy.users.greyc.fr/daisy-etal-siggraph-asia-2013 M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 November 21, 2013 42 / 42