ICIP 2011

1. M. Ghoniem, A. Elmoataz, O. Lézoray Université de Caen Basse-Normandie, GREYC, CNRS UMR 6072 Olivier.lezoray@unicaen.fr Discrete Infinity Harmonic Functions: Towards a Unified Interpolation Framework on Graphs
2. Outline •  Introduction/Motivations •  Partial difference operators on graphs •  Non Local Infinity Laplacian on Graphs •  Infinity harmonic interpolation on Graphs •  Experimentations •  Conclusion
3. Introduction/Motivations •  Large data sets are now collected in the form of: •  Graphs or networks •  Data in High dimensional Spaces
4. •  Analysis of these types of data is a major challenge for image and machine learning communities •  Need to transfer tools for signal and image processing to Graphs and networks •  Examples of such effort includes: Diffusion wavelets ( Coifman-Magionni2006 ) Spectral graph wavelet ( Hammond-vandergheynst 2009) Partial difference Equations on Graphs (Elmoataz-Bougleux-Lézoray 2008, Ta-Elmoataz-Lezoray 2011) •  Our goal is to provide methods that mimic on graphs well-known PDE variational formulations under a functional analysis point of view. •  To do this we use Partial difference Equations (PdE) over graphs. •  PdEs mimic PDEs in domains having a graph Exploit this framework for interpolation in image and data processing Introduction/Motivations
5. Introduction/Motivations •  Many problems on image and Data Processing can be formulated as interpolation problems •  Inpainting •  Colorization •  Semi-supervised image and High dimensional data clustering
6. inpainting as representative example of interpolation •  Inpainting = reconstruction of a missing/damaged/undesirable Domain in a 2D/3D image or a sequence •  Usual assumption: the inpainting domain is known •  Many methods were developed for interpolation Geometry : local PDEs and variational methods Texture: exemplar based methods Both geometry and texture: non local PDEs and variational methods
7. The associated formal Euler-Lagrange equation : p-Laplacian equation 0 D D min subject to . p f f dx f f ∂ ∫ ∇ = f 0 :! " R 2 # R p → ∞ 2 0 ( ) 0 p p f f on D f f f on D − ⎡ ⎤ ∇ ⎢ ⎥ Δ = ∇ ⋅ = ⎢ ⎥ ∇ ⎣ ⎦ = ∂ 0 \ | given D f Ω missing on D Ω p=2 harmonic inpainting Laplace Equation p=1 total variation inpainting 1-laplacian equation p-harmonic inpainting Infinity Laplacian equation Function f is said to be p-harmonic
8. If f minimizes D min p f f dx ∫ ∇ 2 , i ij i i j f f f f x x x ∂ ∂ = = ∂ ∂ ∂ p → ∞ , ( ) 0 ij i j i j f f f f on D ∞ ∑ Δ = = Infinity harmonic Inpainting Discretization : finite difference method, Oberman discretization Then as f satifies The Infinity laplacian equation has several applications Image processing, computer vision, optimal transportation Related to tug-of-War game ( Aronson, Grandall, Casselles, Peres, etc) 0 \ | given D f Ω missing on D Ω
9. Some limitations of local infinity harmonic inpainting: •  Discontinuities are not well recovered •  Texture and repetitive structures are not recovered •  Difficulties to discretize the infinity Laplacian equation on graphs with arbitrary topology Infinity harmonic Inpainting
10. Our motivations and objectives We would like to extend the PDE infinity Laplacian equation (infinity harmonic functions) on weighted graphs with the following objectives: •  Unify local and nonlocal interpolation problems on images •  Interpolation on graphs with an arbitrary topology How ? With the framework of Partial difference Equations (PdEs) on Graphs
11. ( , , ) G V E w = { } 1 2 , ,.., , n n i V u u u u R = ∈ E = uv !V "V / u #v and u,v $V { } ( ) deg ( ) V uv v N u u w ∈ ∑ = : , 0 uv vu uv w E R w w w + → = >= Partial difference operators on graphs
12. f :V ! R, or f :V ! R m ( ) , ( ) ( ) H V u V f g f u g u ∈ ∑ = ( ( ))T V u V f f u R ∈ = ∈ ( ) u V G f f u ∈ ∑ ∫ = ( ) H V Images and data are functions on graphs ( ) , ( ) ( ) H E uv E F G F uv G uv ∈ ∑ = ( ) H E Hilbert space on vertices Hilbert space on edges Partial difference operators on graphs The analog of the integral of a function is considered as
13. * : ( ) ( ) w d f H E H V → ( ) ( ( ) ( )) def w uv d f uv w f v f u = − : ( ) ( ) w d f H v H E → Difference operator : Adjoint and Divergence operators on graphs : Each function F of H(E) has the following property * ( )( ) ( )( ) ( ( ) ( ) w w uv v V d H u div H u w H vu H uv ∈ ∑ = − = − ( )( ) 0 w u V div F u ∈ ∑ = df ,G H (E ) = def f ,d *G H (V ) Elmoataz-Lezoray-Bougleux, Friedman,Bensoussan, etc Partial difference operators on graphs
14. ( ) max ( ) ( ) ( ) w uv v V f u w f v f u ∞ ∈ ∇ = − ( ) max(0, ( )) max(0, ( ) ( )) v v uv f u f u w f v f u + ∂ = ∂ = − !w ± f (u) = "v ± f (u):v ! u ( ) Discrete gradients and their norms ( ) min(0, ( )) min(0, ( ) ( )) v v uv f u f u w f v f u − ∂ = − ∂ = − − !w f (u) = "v f (u):v ! u ( ) !w f (u) p = w(u,v) p/2 f (v)" f (u) p v!u # $ % & ' ( ) 1/ p Upwind derivatives -  Formulations independent of topology of the graph -  Unifies Local and non local -  Provides adaptation Partial difference operators on graphs Partial derivatives ( ) ( ) ( ( ) ( )) v w uv f u d f uv w f v f u ∂ = = − !w ± f (u) ! = max v!u w uv M ± 0, f (v)! f (u) ( ) ( ) !± w f (u) p = w(u,v) p/2 M ± 0, f (v)" f (u) ( ) p v!u # $ % & ' ( ) 1/ p M + = max and M ! = min
15. Anisotropic p-Laplacian 2 * , ( ) def p a w p w w w f d d f d f − Δ = !w, p a f (u) = w(u,v)1/2 f (u)" f (v) v!u # p"2 ( f (u)" f (v)), , 1 ( ) ( ) 2 w p p a w u V p J f f u p ∈ ∑ = ∇ , : ( ) ( ) a w p H V H V Δ → , , ( ) ( ), , ( ) ( ) a w p a w p J f f u u V f H V f u ∂ = Δ ∀ ∈ ∀ ∈ ∂ Property : Difference operators on graphs
16. is known on some subset of the graph 0 0 : f V V R ⊂ → : f V R → Inpainting = Interpolation 0 \ D V V = 0 \ | given V D f D on missing V We consider a general domain a Graph ( , , ) G V E w = 0 f p w u p min ( ) f D f u ∈ ∑ ∇ 0 0 f=f on V Infinity harmonic interpolation on graphs As on the Euclidean domain , we study minimizers of the discrete functional as p goes to infinity
17. p w u p min ( ) f D f u ∈ ∑ ∇ !w, p a f (u) = "w ' # f (u) p#1 p#1 # "w ' + f (u) p#1 p#1 with w '(u,v) =w(u,v) p p#1 p → ∞ From the p-Laplacian and the oriented gradient norms definitions, we can show that: We study minimizers of the discrete functional As !w, p a f (u) = w(u,v)1/2 f (u)" f (v) v!u ! p"2 ( f (u)" f (v)) = 0 The minimization leads !w," f (u) = #w $ f (u) " $ #w + f (u) " Infinity harmonic interpolation on graphs
18. •  Infinity Laplacian on Graphs = local and non local infinity Laplacian , ( ) ( ) ( ) w w w f u f u f u − + ∞ ∞ ∞ Δ = ∇ − ∇ Infinity harmonic interpolation on graphs !w " f (u) # = max v!u w uv max( f (u)" f (v),0) ( ) !w + f (u) " = max( f (v)) v!u # f (u) =!( f )(u)# f (u) !w " f (u) # = f (u)" min( f (v)) v!u = f (u)"!( f )(u) !w + f (u) " = max v!u w uv max( f (v)# f (u),0) ( ) •  Unweighted graph w(u,v)=1 Morphological Gradients on graphs , ( ) 2 ( ) ( )( ) ( )( ) w f u f u f u f u δ ε ∞ Δ = − − Morphological Laplacian on graphs
19. •  Non local Infinity Laplacian on Euclidean domains , ( , , ) n R G E w Ω ⊂ = Ω Infinity harmonic interpolation on graphs { } ( , ) , ( , ) 0 E x y w x y = ∈Ω×Ω > : w R+ Ω×Ω → [ ] 0,1 1 ( , ) ( , ) 0 w x y if y x x y w x y if y x α α ∈ = ≠ − = = , , , ( ) ( ) ( ) ( ) ( ) max min w y y x y y x f y f x f y f x f x x y x y α α ∞ ∈Ω ≠ ∈Ω ≠ ⎛ ⎞ ⎛ ⎞ − − ⎜ ⎟ ⎜ ⎟ −Δ = + ⎜ ⎟ ⎜ ⎟ − − ⎝ ⎠ ⎝ ⎠ , ( ) ( ) ( ) w w w f x f x f x − + ∞ ∞ ∞ Δ = ∇ − ∇ ( ) ( ) , ( ) max ( , )max( ( ) ( ),0) min ( , )min( ( ) ( ),0) w f x w x y f y f x w x y f y f x ∞ Ω Ω −Δ = − + − •  Non Local Hölder infinity Laplacian (Chambolle-Lindgren-Monneau 2011)
20. is known on some subset of the graph 0 0 : f V V R ⊂ → : f V R → Inpainting = solution of the following dirichlet problem 0 \ D V V = !" w,# f (u) = 0 on D =V \V0 f (u) = f 0(u) on $D We consider a general domain a Graph ( , , ) G V E w = 0 f Find f Infinity harmonic interpolation on graphs This Dirichlet problem has an unique solution 0 \ | given V D f D on missing V
21. Infinity harmonic interpolation on graphs [ ] 1 ( ) ( )( ) ( )( ) ( )( ) 2 f u NLA f u NLD f u NLE f u = = + The equation can be rewritten as (a non local infinity-harmonious function) : ( )( ) ( ) ( ) ( )( ) ( ) ( ) w w NLD f u f u f u NLE f u f u f u + ∞ − ∞ = + ∇ = − ∇ Non local morphological Dilation Non local morphological Erosion !f (u) !t = "w + f (u) # !f (u) !t = $ "w $ f (u) # Non Local PdEs morphology on Graphs Ta-Elmoataz-Lézoray (2008,2011) !"w,# f (u) = 0 We can show that the sequence Admits a unique solution and converges to the f=NLA(f) f n+1(u) = NLA( f n )(u) Simple digital algorithm= repeated average of non local dilatation and erosion f (u) = 1 2 2 f (u)+ !w + f (u) " # !w # f (u) " $ % & ' ( ) or
22. f :! " Z 2 # R, becomes f :V # R p i = (x i , y i ) $ !%u $V 2 2 2 ( ) ( ) ( , ) xp( ) f f F u F v w u v e σ − − = Non local inpainting
23. Non local inpainting
25. Non local inpainting
26. Non local upscaling
27. Non local upscaling
28. Semi-supervised segmentation
29. •  We  proposed   –  A  connec,on  between  the  anisotropic  p-­‐Laplacian   and  discrete  norms  of  morphological  gradients  on   graphs   –  Derived  an  explicit  expression  of  the  infinity  Laplacian   on  graphs   –   An  algorithm  for  solving  the  Dirichlet  problem  with   infinity-­‐harmonic  func,ons  on  graphs   –  Unifies  local  and  nonlocal  interpola,on  on  images   •  We  used  this  formalism  to  perform  local  and   nolocal  interpola,on  on  graphs   Conclusion
30. Thanks,   Any  ques,ons  ?