SIGGRAPH ASIA 2015 - 1

Transcript

1. Exemplar-based Video Completion with Geometry-guided Space-time Patch Blending Maxime Daisy 1,Pierre Buyssens, David Tschumperlé and Olivier Lézoray GREYC - CNRS UMR 6072 Image team Université de Caen Normandie, FRANCE SIGGRAPH Asia 2015 2nd November 2015 1. This research was supported by French national grant Action 3DS
2. Context Greedy patch-based inpainting & Artefact Reduction Technique M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 2 / 32
3. Greedy pattern-based inpainting M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 3 / 32
4. Greedy pattern-based inpainting Fake Owl - 800 ⇥ 450 Masked Image M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 4 / 32
5. Greedy pattern-based inpainting Fake Owl - 800 ⇥ 450 Inpainted Image M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 4 / 32
6. Greedy pattern-based inpainting block-effect artifacts M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 5 / 32
7. Spatial Patch Blending M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 6 / 32
8. Spatial Patch Blending M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 6 / 32
9. Spatial Patch Blending M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 6 / 32
10. Artifact Detection 1 Map of the artifact location probabilities 2 hypothesis for artifact locations • local sharp variations in the inpainted image • source patches come from very different locations PA = k r I k . div ( ) : inpainting correspondence map published in CAIP 2013 : "Spatial Patch Blending for Artefact Reduction in Pattern-Based Inpainting Techniques" M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 7 / 32
11. Artifact Detection 1 Map of the artifact location probabilities 2 hypothesis for artifact locations • local sharp variations in the inpainted image • source patches come from very different locations PA = k r I k . div ( ) : inpainting correspondence map 2 Strongest artifact locations LA = { p | PA( p ) > ⌧} published in CAIP 2013 : "Spatial Patch Blending for Artefact Reduction in Pattern-Based Inpainting Techniques" M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 7 / 32
12. Artifact Detection 1 Map of the artifact location probabilities 2 hypothesis for artifact locations • local sharp variations in the inpainted image • source patches come from very different locations PA = k r I k . div ( ) : inpainting correspondence map 2 Strongest artifact locations LA = { p | PA( p ) > ⌧} 3 Blending amplitude map ( p ) = ⇢ . P q 2 LA w ( p , q ) max r 2I P q 2 LA w ( p , q ) with w ( p , q ) = exp ⇣ k p q k2 PA( q )2 ⌘ (1) published in CAIP 2013 : "Spatial Patch Blending for Artefact Reduction in Pattern-Based Inpainting Techniques" M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 7 / 32
13. Patch Blending Weighted sum of overlapping patches pixels values J ( p ) = P q 2 p w ( p , q ) . q ( p q ) P q 2 p w ( q , p ) (2) • w ( p , q ) = exp ⇣ k p q k2 ( p )2 ⌘ • p = { q | q \ p 6= ?} ! one gaussian function for each possible blending amplitude ) quite slow M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 8 / 32
14. A faster patch blending algorithm published in SIGGRAPH-ASIA 2013 : "A Fast Spatial Patch Blending Algorithm for Artefact Reduction in Pattern-based Image Inpainting" M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 9 / 32
15. block-effect artifacts ? Inpainted Image M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 10 / 32
16. block-effect artifacts ? Blended Image (isotropic) M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 11 / 32
17. block-effect artifacts ? Blended Image (geometry-guided) M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 12 / 32
18. Spatial Patch Blending CAIP 2013 Spatial Patch Blending + + Multiscale scheme SIGGRAPH ASIA 2013 Fast Spatial Patch Blending + + Anisotropic model CAIP 2015 Fast Geometry-guided Patch Blending + + Video inpainting SIGGRAPH Asia 2015 Space-time Geometry-guided Patch Blending M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 13 / 32
19. Contribution Video inpainting + Space-time geometry guided patch blending M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 14 / 32
20. Video Inpainting M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 15 / 32
21. Video Inpainting M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 16 / 32
22. Geometry-guided patch blending : process Anisotropic image regularization M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 17 / 32
23. Geometry-guided patch blending : process Geometry-guided patch blending M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 18 / 32
24. Geometry-guided patch blending 1 Structure eigen values normalization ˆ S( p ) i = S( p ) max p 2I S( p ) i M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 19 / 32
25. Geometry-guided patch blending 1 Structure eigen values normalization ˆ S( p ) i = S( p ) max p 2I S( p ) i 2 Blending tensors eigen values computation B i = 1 ( 1 + ˆ S 1 + ˆ S 2 + ˆ S 3 ) i M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 19 / 32
26. Geometry-guided patch blending 1 Structure eigen values normalization ˆ S( p ) i = S( p ) max p 2I S( p ) i 2 Blending tensors eigen values computation B i = 1 ( 1 + ˆ S 1 + ˆ S 2 + ˆ S 3 ) i 3 Blending tensor building B = s B 1 e? S 1 .T e? S 1 + s B 2 e? S 2 .T e? S 2 | {z } spatial term + t Be? S t .T e? S t | {z } temporal term (3) • eS i : structure tensor eigen vectors • s1 , 2 = s B 1 , 2 t = t B 3 • s , t : spatial (resp. temporal) blending bandwidth M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 19 / 32
27. Geometry-guided patch blending i parameters in 2D M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 20 / 32
28. Geometry-guided patch blending i parameters in 2D M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 20 / 32
29. Geometry-guided patch blending i parameters in 2D M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 20 / 32
30. Geometry-guided patch blending i parameters in 2D M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 20 / 32
31. Geometry-guided patch blending i parameters in 2D M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 20 / 32
32. Geometry-guided patch blending xy plan xt plan M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 21 / 32
33. Geometry-guided patch blending Weighted sum of overlapping patches pixels values using Eq. (2) with w ( p , q ) = 8 > < > : exp ⇣ k p q k2 ( p )2 ⌘ isotropic expXT B( p ) 1 X anisotropic where X = q p M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 22 / 32
34. Results M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 23 / 32
35. block-effect artifacts ? Inpainted Image M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 24 / 32
36. block-effect artifacts ? Blended Image (isotropic) M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 25 / 32
37. block-effect artifacts ? Blended Image (geometry-guided) M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 26 / 32
38. Orange / Bricks Without blending M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 27 / 32
39. Orange / Bricks Isotropic patch blending M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 27 / 32
40. Orange / Bricks Geometry-guided patch blending M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 27 / 32
41. Woman Without blending M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 28 / 32
42. Woman Isotropic patch blending M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 28 / 32
43. Woman Geometry-guided patch blending M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 28 / 32
44. Video result https://daisy.users.greyc.fr/articles/SA15/ M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 29 / 32
45. Conclusion Contributions • Video inpainting algorithm • Method to reduce block-effect while keeping sharpness Recent advances Depth-aware patch blending for stereoscopic inpainted results M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 30 / 32
46. Masked image.
47. Depth map.
48. Inpainted image (without blending).
49. Inpainted image + isotropic depth-aware blending.
50. Inpainted image + anisotropic blending.
51. Inpainted image + anisotropic depth-aware blending.
52. Thank you