Share
Embed
Transcript
52 slides
-
Exemplar-based Video Completion with Geometry-guided Space-time Patch Blending Maxime Daisy
1,Pierre Buyssens, David Tschumperlé and Olivier Lézoray GREYC - CNRS UMR 6072 Image team Université de Caen Normandie, FRANCE SIGGRAPH Asia 2015 2nd November 2015 1. This research was supported by French national grant Action 3DS -
Context Greedy patch-based inpainting & Artefact Reduction Technique M. Daisy
(GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 2 / 32 -
Greedy pattern-based inpainting M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH
Asia 2015 2nd November 2015 3 / 32 -
Greedy pattern-based inpainting Fake Owl - 800 ⇥ 450 Masked
Image M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 4 / 32 -
Greedy pattern-based inpainting Fake Owl - 800 ⇥ 450 Inpainted
Image M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 4 / 32 -
Greedy pattern-based inpainting block-effect artifacts M. Daisy (GREYC, Caen -
FRANCE) SIGGRAPH Asia 2015 2nd November 2015 5 / 32 -
Spatial Patch Blending M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH
Asia 2015 2nd November 2015 6 / 32 -
Spatial Patch Blending M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH
Asia 2015 2nd November 2015 6 / 32 -
Spatial Patch Blending M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH
Asia 2015 2nd November 2015 6 / 32 -
Artifact Detection 1 Map of the artifact location probabilities 2
hypothesis for artifact locations • local sharp variations in the inpainted image • source patches come from very different locations PA = k r I k . div ( ) : inpainting correspondence map published in CAIP 2013 : "Spatial Patch Blending for Artefact Reduction in Pattern-Based Inpainting Techniques" M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 7 / 32 -
Artifact Detection 1 Map of the artifact location probabilities 2
hypothesis for artifact locations • local sharp variations in the inpainted image • source patches come from very different locations PA = k r I k . div ( ) : inpainting correspondence map 2 Strongest artifact locations LA = { p | PA( p ) > ⌧} published in CAIP 2013 : "Spatial Patch Blending for Artefact Reduction in Pattern-Based Inpainting Techniques" M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 7 / 32 -
Artifact Detection 1 Map of the artifact location probabilities 2
hypothesis for artifact locations • local sharp variations in the inpainted image • source patches come from very different locations PA = k r I k . div ( ) : inpainting correspondence map 2 Strongest artifact locations LA = { p | PA( p ) > ⌧} 3 Blending amplitude map ( p ) = ⇢ . P q 2 LA w ( p , q ) max r 2I P q 2 LA w ( p , q ) with w ( p , q ) = exp ⇣ k p q k2 PA( q )2 ⌘ (1) published in CAIP 2013 : "Spatial Patch Blending for Artefact Reduction in Pattern-Based Inpainting Techniques" M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 7 / 32 -
Patch Blending Weighted sum of overlapping patches pixels values J
( p ) = P q 2 p w ( p , q ) . q ( p q ) P q 2 p w ( q , p ) (2) • w ( p , q ) = exp ⇣ k p q k2 ( p )2 ⌘ • p = { q | q \ p 6= ?} ! one gaussian function for each possible blending amplitude ) quite slow M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 8 / 32 -
A faster patch blending algorithm published in SIGGRAPH-ASIA 2013 :
"A Fast Spatial Patch Blending Algorithm for Artefact Reduction in Pattern-based Image Inpainting" M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 9 / 32 -
block-effect artifacts ? Inpainted Image M. Daisy (GREYC, Caen -
FRANCE) SIGGRAPH Asia 2015 2nd November 2015 10 / 32 -
block-effect artifacts ? Blended Image (isotropic) M. Daisy (GREYC, Caen
- FRANCE) SIGGRAPH Asia 2015 2nd November 2015 11 / 32 -
block-effect artifacts ? Blended Image (geometry-guided) M. Daisy (GREYC, Caen
- FRANCE) SIGGRAPH Asia 2015 2nd November 2015 12 / 32 -
Spatial Patch Blending CAIP 2013 Spatial Patch Blending + +
Multiscale scheme SIGGRAPH ASIA 2013 Fast Spatial Patch Blending + + Anisotropic model CAIP 2015 Fast Geometry-guided Patch Blending + + Video inpainting SIGGRAPH Asia 2015 Space-time Geometry-guided Patch Blending M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 13 / 32 -
Contribution Video inpainting + Space-time geometry guided patch blending M.
Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 14 / 32 -
Video Inpainting M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia
2015 2nd November 2015 15 / 32 -
Video Inpainting M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia
2015 2nd November 2015 16 / 32 -
Geometry-guided patch blending : process Anisotropic image regularization M. Daisy
(GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 17 / 32 -
Geometry-guided patch blending : process Geometry-guided patch blending M. Daisy
(GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 18 / 32 -
Geometry-guided patch blending 1 Structure eigen values normalization ˆ S(
p ) i = S( p ) max p 2I S( p ) i M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 19 / 32 -
Geometry-guided patch blending 1 Structure eigen values normalization ˆ S(
p ) i = S( p ) max p 2I S( p ) i 2 Blending tensors eigen values computation B i = 1 ( 1 + ˆ S 1 + ˆ S 2 + ˆ S 3 ) i M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 19 / 32 -
Geometry-guided patch blending 1 Structure eigen values normalization ˆ S(
p ) i = S( p ) max p 2I S( p ) i 2 Blending tensors eigen values computation B i = 1 ( 1 + ˆ S 1 + ˆ S 2 + ˆ S 3 ) i 3 Blending tensor building B = s B 1 e? S 1 .T e? S 1 + s B 2 e? S 2 .T e? S 2 | {z } spatial term + t Be? S t .T e? S t | {z } temporal term (3) • eS i : structure tensor eigen vectors • s1 , 2 = s B 1 , 2 t = t B 3 • s , t : spatial (resp. temporal) blending bandwidth M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 19 / 32 -
Geometry-guided patch blending i parameters in 2D M. Daisy (GREYC,
Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 20 / 32 -
Geometry-guided patch blending i parameters in 2D M. Daisy (GREYC,
Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 20 / 32 -
Geometry-guided patch blending i parameters in 2D M. Daisy (GREYC,
Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 20 / 32 -
Geometry-guided patch blending i parameters in 2D M. Daisy (GREYC,
Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 20 / 32 -
Geometry-guided patch blending i parameters in 2D M. Daisy (GREYC,
Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 20 / 32 -
Geometry-guided patch blending xy plan xt plan M. Daisy (GREYC,
Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 21 / 32 -
Geometry-guided patch blending Weighted sum of overlapping patches pixels values
using Eq. (2) with w ( p , q ) = 8 > < > : exp ⇣ k p q k2 ( p )2 ⌘ isotropic expXT B( p ) 1 X anisotropic where X = q p M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 22 / 32 -
Results M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015
2nd November 2015 23 / 32 -
block-effect artifacts ? Inpainted Image M. Daisy (GREYC, Caen -
FRANCE) SIGGRAPH Asia 2015 2nd November 2015 24 / 32 -
block-effect artifacts ? Blended Image (isotropic) M. Daisy (GREYC, Caen
- FRANCE) SIGGRAPH Asia 2015 2nd November 2015 25 / 32 -
block-effect artifacts ? Blended Image (geometry-guided) M. Daisy (GREYC, Caen
- FRANCE) SIGGRAPH Asia 2015 2nd November 2015 26 / 32 -
Orange / Bricks Without blending M. Daisy (GREYC, Caen -
FRANCE) SIGGRAPH Asia 2015 2nd November 2015 27 / 32 -
Orange / Bricks Isotropic patch blending M. Daisy (GREYC, Caen
- FRANCE) SIGGRAPH Asia 2015 2nd November 2015 27 / 32 -
Orange / Bricks Geometry-guided patch blending M. Daisy (GREYC, Caen
- FRANCE) SIGGRAPH Asia 2015 2nd November 2015 27 / 32 -
Woman Without blending M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH
Asia 2015 2nd November 2015 28 / 32 -
Woman Isotropic patch blending M. Daisy (GREYC, Caen - FRANCE)
SIGGRAPH Asia 2015 2nd November 2015 28 / 32 -
Woman Geometry-guided patch blending M. Daisy (GREYC, Caen - FRANCE)
SIGGRAPH Asia 2015 2nd November 2015 28 / 32 -
Video result https://daisy.users.greyc.fr/articles/SA15/ M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH
Asia 2015 2nd November 2015 29 / 32 -
Conclusion Contributions • Video inpainting algorithm • Method to reduce
block-effect while keeping sharpness Recent advances Depth-aware patch blending for stereoscopic inpainted results M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2015 2nd November 2015 30 / 32