Upgrade to Pro — share decks privately, control downloads, hide ads and more …

SIGGRAPH ASIA 2015 - 1

SIGGRAPH ASIA 2015 - 1

Olivier Lézoray

November 02, 2015
Tweet

More Decks by Olivier Lézoray

Other Decks in Research

Transcript

  1. Exemplar-based Video Completion with Geometry-guided Space-time Patch Blending Maxime Daisy

    1,Pierre Buyssens, David Tschumperlé and Olivier Lézoray GREYC - CNRS UMR 6072 Image team Université de Caen Normandie, FRANCE SIGGRAPH Asia 2015 2nd November 2015 1. This research was supported by French national grant Action 3DS
  2. Context Greedy patch-based inpainting & Artefact Reduction Technique M. Daisy

    (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 2 / 32
  3. Greedy pattern-based inpainting Fake Owl - 800 ⇥ 450 Masked

    Image M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 4 / 32
  4. Greedy pattern-based inpainting Fake Owl - 800 ⇥ 450 Inpainted

    Image M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 4 / 32
  5. Greedy pattern-based inpainting block-effect artifacts M. Daisy (GREYC, Caen -

    FRANCE) SIGGRAPH Asia 2015 2nd November 2015 5 / 32
  6. Artifact Detection 1 Map of the artifact location probabilities 2

    hypothesis for artifact locations • local sharp variations in the inpainted image • source patches come from very different locations PA = k r I k . div ( ) : inpainting correspondence map published in CAIP 2013 : "Spatial Patch Blending for Artefact Reduction in Pattern-Based Inpainting Techniques" M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 7 / 32
  7. Artifact Detection 1 Map of the artifact location probabilities 2

    hypothesis for artifact locations • local sharp variations in the inpainted image • source patches come from very different locations PA = k r I k . div ( ) : inpainting correspondence map 2 Strongest artifact locations LA = { p | PA( p ) > ⌧} published in CAIP 2013 : "Spatial Patch Blending for Artefact Reduction in Pattern-Based Inpainting Techniques" M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 7 / 32
  8. Artifact Detection 1 Map of the artifact location probabilities 2

    hypothesis for artifact locations • local sharp variations in the inpainted image • source patches come from very different locations PA = k r I k . div ( ) : inpainting correspondence map 2 Strongest artifact locations LA = { p | PA( p ) > ⌧} 3 Blending amplitude map ( p ) = ⇢ . P q 2 LA w ( p , q ) max r 2I P q 2 LA w ( p , q ) with w ( p , q ) = exp ⇣ k p q k2 PA( q )2 ⌘ (1) published in CAIP 2013 : "Spatial Patch Blending for Artefact Reduction in Pattern-Based Inpainting Techniques" M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 7 / 32
  9. Patch Blending Weighted sum of overlapping patches pixels values J

    ( p ) = P q 2 p w ( p , q ) . q ( p q ) P q 2 p w ( q , p ) (2) • w ( p , q ) = exp ⇣ k p q k2 ( p )2 ⌘ • p = { q | q \ p 6= ?} ! one gaussian function for each possible blending amplitude ) quite slow M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 8 / 32
  10. A faster patch blending algorithm published in SIGGRAPH-ASIA 2013 :

    "A Fast Spatial Patch Blending Algorithm for Artefact Reduction in Pattern-based Image Inpainting" M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 9 / 32
  11. block-effect artifacts ? Inpainted Image M. Daisy (GREYC, Caen -

    FRANCE) SIGGRAPH Asia 2015 2nd November 2015 10 / 32
  12. block-effect artifacts ? Blended Image (isotropic) M. Daisy (GREYC, Caen

    - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 11 / 32
  13. block-effect artifacts ? Blended Image (geometry-guided) M. Daisy (GREYC, Caen

    - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 12 / 32
  14. Spatial Patch Blending CAIP 2013 Spatial Patch Blending + +

    Multiscale scheme SIGGRAPH ASIA 2013 Fast Spatial Patch Blending + + Anisotropic model CAIP 2015 Fast Geometry-guided Patch Blending + + Video inpainting SIGGRAPH Asia 2015 Space-time Geometry-guided Patch Blending M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 13 / 32
  15. Contribution Video inpainting + Space-time geometry guided patch blending M.

    Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 14 / 32
  16. Geometry-guided patch blending : process Anisotropic image regularization M. Daisy

    (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 17 / 32
  17. Geometry-guided patch blending : process Geometry-guided patch blending M. Daisy

    (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 18 / 32
  18. Geometry-guided patch blending 1 Structure eigen values normalization ˆ S(

    p ) i = S( p ) max p 2I S( p ) i M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 19 / 32
  19. Geometry-guided patch blending 1 Structure eigen values normalization ˆ S(

    p ) i = S( p ) max p 2I S( p ) i 2 Blending tensors eigen values computation B i = 1 ( 1 + ˆ S 1 + ˆ S 2 + ˆ S 3 ) i M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 19 / 32
  20. Geometry-guided patch blending 1 Structure eigen values normalization ˆ S(

    p ) i = S( p ) max p 2I S( p ) i 2 Blending tensors eigen values computation B i = 1 ( 1 + ˆ S 1 + ˆ S 2 + ˆ S 3 ) i 3 Blending tensor building B = s B 1 e? S 1 .T e? S 1 + s B 2 e? S 2 .T e? S 2 | {z } spatial term + t Be? S t .T e? S t | {z } temporal term (3) • eS i : structure tensor eigen vectors • s1 , 2 = s B 1 , 2 t = t B 3 • s , t : spatial (resp. temporal) blending bandwidth M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 19 / 32
  21. Geometry-guided patch blending i parameters in 2D M. Daisy (GREYC,

    Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 20 / 32
  22. Geometry-guided patch blending i parameters in 2D M. Daisy (GREYC,

    Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 20 / 32
  23. Geometry-guided patch blending i parameters in 2D M. Daisy (GREYC,

    Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 20 / 32
  24. Geometry-guided patch blending i parameters in 2D M. Daisy (GREYC,

    Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 20 / 32
  25. Geometry-guided patch blending i parameters in 2D M. Daisy (GREYC,

    Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 20 / 32
  26. Geometry-guided patch blending xy plan xt plan M. Daisy (GREYC,

    Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 21 / 32
  27. Geometry-guided patch blending Weighted sum of overlapping patches pixels values

    using Eq. (2) with w ( p , q ) = 8 > < > : exp ⇣ k p q k2 ( p )2 ⌘ isotropic expXT B( p ) 1 X anisotropic where X = q p M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 22 / 32
  28. block-effect artifacts ? Inpainted Image M. Daisy (GREYC, Caen -

    FRANCE) SIGGRAPH Asia 2015 2nd November 2015 24 / 32
  29. block-effect artifacts ? Blended Image (isotropic) M. Daisy (GREYC, Caen

    - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 25 / 32
  30. block-effect artifacts ? Blended Image (geometry-guided) M. Daisy (GREYC, Caen

    - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 26 / 32
  31. Orange / Bricks Without blending M. Daisy (GREYC, Caen -

    FRANCE) SIGGRAPH Asia 2015 2nd November 2015 27 / 32
  32. Orange / Bricks Isotropic patch blending M. Daisy (GREYC, Caen

    - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 27 / 32
  33. Orange / Bricks Geometry-guided patch blending M. Daisy (GREYC, Caen

    - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 27 / 32
  34. Woman Isotropic patch blending M. Daisy (GREYC, Caen - FRANCE)

    SIGGRAPH Asia 2015 2nd November 2015 28 / 32
  35. Woman Geometry-guided patch blending M. Daisy (GREYC, Caen - FRANCE)

    SIGGRAPH Asia 2015 2nd November 2015 28 / 32
  36. Conclusion Contributions • Video inpainting algorithm • Method to reduce

    block-effect while keeping sharpness Recent advances Depth-aware patch blending for stereoscopic inpainted results M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 30 / 32