ICASSP 2012

Transcript

1. Nonlocal PdEs on graphs for active contours models with applications to image segmentation and data clustering O. L´ ezoray1, A. Elmoataz1, V.-T. Ta2 1Universit´ e de Caen Basse Normandie, France 2Universit´ e de Bordeaux, France olivier.lezoray@unicaen.fr http://www.info.unicaen.fr/~lezoray/
2. 1 Introduction 2 Graphs and difference operators 3 Variational framework of active contours on graphs 4 Nonlocal active contours models as global minimizers 5 Applications O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 2 / 30
3. 1 Introduction 2 Graphs and difference operators 3 Variational framework of active contours on graphs 4 Nonlocal active contours models as global minimizers 5 Applications O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 3 / 30
4. From image and data to graphs Graphs occur as a the most natural and flexible way of representing arbitrary data by modeling the neighborhood properties between these data (whenever they are organized or not). O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 4 / 30
5. From image and data to graphs Graphs occur as a the most natural and flexible way of representing arbitrary data by modeling the neighborhood properties between these data (whenever they are organized or not). Images or Superpixel graphs O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 4 / 30
6. From image and data to graphs Graphs occur as a the most natural and flexible way of representing arbitrary data by modeling the neighborhood properties between these data (whenever they are organized or not). 3D Meshes O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 4 / 30
7. From image and data to graphs Graphs occur as a the most natural and flexible way of representing arbitrary data by modeling the neighborhood properties between these data (whenever they are organized or not). Social Networks O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 4 / 30
8. From image and data to graphs Graphs occur as a the most natural and flexible way of representing arbitrary data by modeling the neighborhood properties between these data (whenever they are organized or not). Web Graphs O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 4 / 30
9. Processing signals on Graphs Our line of research There are a lot of recent works that aim at extending signal processing tools for the processing of functions on graphs Typical examples are graph wavelets and PDEs on graphs Our goal is to provide methods that mimic on graphs well-known PDE variational formulations To do this we use Partial difference Equations (PdE) that mimic PDEs in domains having a graph structure. In this paper We propose an extension of variational active contours for the processing of functions on graphs. We introduce nonlocal perimeters and co-area on graphs We consider nonlocal global minimizers for segmentation and clustering on graphs O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 5 / 30
10. 1 Introduction 2 Graphs and difference operators 3 Variational framework of active contours on graphs 4 Nonlocal active contours models as global minimizers 5 Applications O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 6 / 30
11. Weighted graphs Basics A weighted graph G = (V, E, w) consists in a finite set V = {v1, . . . , vN } of N vertices O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 7 / 30
12. Weighted graphs Basics A weighted graph G = (V, E, w) consists in a finite set V = {v1, . . . , vN } of N vertices and a finite set E = {e1, . . . , eN } ⊂ V × V of N weighted edges. We assume G to be simple, undirected (always (vi ← vj ) and (vj → vi )), with no self-loops and no multiple edges. O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 7 / 30
13. Weighted graphs Basics A weighted graph G = (V, E, w) consists in a finite set V = {v1, . . . , vN } of N vertices and a finite set E = {e1, . . . , eN } ⊂ V × V of N weighted edges. We assume G to be simple, undirected (always (vi ← vj ) and (vj → vi )), with no self-loops and no multiple edges. eij = (vi , vj ) is the edge of E that connects vertices vi and vj of V. Its weight, denoted by wij = w(vi , vj ), represents the similarity between its vertices. Similarities are usually computed by using a positive symmetric function w : V × V → R+ satisfying w(vi , vj ) = 0 if (vi , vj ) / ∈ E. w w w w w w w w w w w O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 7 / 30
14. Weighted graphs Basics A weighted graph G = (V, E, w) consists in a finite set V = {v1, . . . , vN } of N vertices and a finite set E = {e1, . . . , eN } ⊂ V × V of N weighted edges. We assume G to be simple, undirected (always (vi ← vj ) and (vj → vi )), with no self-loops and no multiple edges. eij = (vi , vj ) is the edge of E that connects vertices vi and vj of V. Its weight, denoted by wij = w(vi , vj ), represents the similarity between its vertices. Similarities are usually computed by using a positive symmetric function w : V × V → R+ satisfying w(vi , vj ) = 0 if (vi , vj ) / ∈ E. The notation vi ∼ vj is used to denote two adjacent vertices. O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 7 / 30
15. Graph boundaries Let A be a set of connected vertices with A⊂V. We denote by ∂+A and ∂−A, the external and internal boundary sets of A, respectively. A−=V \ A is the complement of A. For vi ∈V, ∂+A = {vi ∈A− : ∃vj ∈A with (vi , vj )∈E} ∂−A = {vi ∈A : ∃vj ∈A− with (vi , vj )∈E} ∂A = {(vi , vj ) ∈ E : vi ∈ ∂+A and vj ∈ ∂−A} - - - - - - - - + + + + + + - + + + + + + + + - Blue vertices correspond to A. ’−’ sign for the internal boundary ∂−A and ’+’ sign for the external boundary ∂+A. The degree of a vertex relatively to a set of vertices A is dA (vi ) = vj ∼vi w(vi , vj ), ∀vi ∈ A. The volume of a set of vertices A is vol(A) = vi ∈A dA (vi ). O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 8 / 30
16. Spaces of functions on Graphs Let H(V) be the Hilbert space of real-valued functions defined on the vertices of a graph. A function f : V → R of H(V) assigns a real value xi = f (vi ) to each vertex vi ∈ V. By analogy with functional analysis on continuous spaces, the integral of a function f ∈ H(V), over the set of vertices V, is defined as V f = V f . The space H(V) is endowed with the usual inner product f , h H(V) = vi ∈V f (vi )h(vi ), where f , h : V → R. Similarly, let H(E) be the space of real-valued functions defined on the edges of G. It is endowed with the inner product F, H H(E) vi ∈V vj ∼vi F(vi , vj )H(vi , vj ), where F, H : E → R are two functions of H(E). O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 9 / 30
17. Difference operators on weighted graphs · Discretization of classical continuous differential geometry. The difference operator of f , dw : H(V) → H(E), is defined on an edge eij = (vi , vj ) ∈ E by: (dw f )(eij ) = (dw f )(vi , vj ) = w(vi , vj )1/2(f (vj ) − f (vi )) . (1) The adjoint of the difference operator, noted d∗ w : H(E) → H(V), is a linear operator defined by dw f , H H(E) = f , d∗ w H H(V) for all f ∈ H(V) and all H ∈ H(E). The adjoint operator d∗ w , of a function H ∈ H(E), can by expressed at a vertex vi ∈ V by the following expression: (d∗ w H)(vi ) = −divw (H)(vi ) = vj ∼vi w(vi , vj )1/2(H(vj , vi ) − H(vi , vj )) . (2) O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 10 / 30
18. Difference operators on weighted graphs We also introduce morphological (directed) difference operators: (d+ w f )(vi , vj )=w(vi , vj )1/2 max f (vi ), f (vj ) −f (vi ) and (d− w f )(vi , vj )=w(vi , vj )1/2 f (vi )− min f (vi ), f (vj ) , (3) with the following properties (always positive) (d+ w f )(vi , vj )= max 0, (dw f )(vi , vj ) (d− w f )(vi , vj )= − min 0, (dw f )(vi , vj ) O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 11 / 30
19. Weighted gradient operator The weighted gradient operator of a function f ∈ H(V), at a vertex vi ∈ V, is the vector operator defined by (∇w f)(vi ) = [(dw f )(vi , vj ), : vj ∼ vi ]T ∀(vi , vj ) ∈ E. (4) The Lp norm of this vector represents the local variation of the function f at a vertex of the graph: (∇w f)(vi ) p = vj ∼vi wp/2 ij f (vj )−f (vi ) p 1/p . (5) Similarly, we have with M+ = max and M− = min (∇± w f)(vi )=[(d± w f )(vi , vj ), : vj ∼ vi ]T ∀(vi , vj ) ∈ E . (∇± w f)(vi ) p = vj ∼vi w(vi , vj )p/2 M± 0, f (vj )−f (vi ) p 1/p . (6) These norms exhibit the following property: (∇w f)(vi ) p p = (∇+ w f)(vi ) p p + (∇− w f)(vi ) p p . (7) O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 12 / 30
20. Anisotropic p-Laplacian The weighted p-Laplace anisotropic operator of a function f ∈ H(V), noted ∆a w,p : H(V) → H(V), is defined by: (∆a w,p f )(vi ) = 1 2 d∗ w (|(dw f )(vi , vj )|p−2(dw f )(vi , vj )) . (8) The anisotropic p-Laplace operator of f ∈ H(V), at a vertex vi ∈ V, can be computed by: (∆a w,p f )(vi ) = vj ∼vi (γa w,p f )(vi , vj )(f (vi ) − f (vj )) . (9) with (γa w,p f )(vi , vj ) = wp/2 ij |f (vi ) − f (vj )|p−2 . (10) O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 13 / 30
21. 1 Introduction 2 Graphs and difference operators 3 Variational framework of active contours on graphs 4 Nonlocal active contours models as global minimizers 5 Applications O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 14 / 30
22. Nonlocal regularization functionals We consider nonlocal regularization functionals based on Weighted Total Variation on graphs Rw,p : H(V ) → R of a function f ∈ H(V ): Rw,p (f ) = vi ∈V Dw f(vi ) p p with 0 < p < +∞ Dw is a discrete gradient operator that can be among ∇w and ∇± w . Rw,p (f ) = vi ∈V vj ∼vi w(vi , vj )p/2 f (vj )−f (vi ) p R± w,p (f ) = vj ∈V vj ∼vi w(vi , vj )p/2 M± 0, f (vj )−f (vi ) p with Rw,p (f ) = R+ w,p (f ) + R− w,p (f ). O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 15 / 30
23. Gradients of a sub-graph Proposition For a given vertex, if vi ∈ A, then χA (vi ) = 1 and χA (vi ) = 0 otherwise. The Lp -norm of the directional gradient of the indicator function is defined as: ∇± w χA (vi ) p p = χ∂±A (vi ) vj ∼vi ,vj ∈A± w(vi , vj )p/2 (11) with the notation A+=A and A−=V \ A. This leads to the following property: ∇w χA (vi ) p p = ∇+ w χA (vi ) p p + ∇− w χA (vi ) p p O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 16 / 30
24. Nonlocal functionals and nonlocal perimeters Proposition By replacing f by the indicator function χA in the regularization functionals, one has: R± w,p (χA ) = 1 2 vol(∂A). Proof. Using equation (11) and the nonlocal functionals, R+ w,p (χA ) = vi ∈V vj ∼vi ,vj ∈A w(vi , vj )p/2χ∂+A (vi ) = vi ∈∂+A vj ∈∂−A w(vi , vj )p/2 = = 1 2 vol(∂A) Similarly, one has R− w,p (χA ) = 1 2 vol(∂A), and Rw,p (χA ) = vol(∂A). - - - - - - - - + + + + + + - + + + + + + + + - All these functionals can be seen as nonlocal discrete perimeters of a sub-graph. O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 17 / 30
25. Co-area formula on weighted graphs For p = 1, the introduced nonlocal functionals do verify the co-area formula. Proposition For any function f ∈ H(V) and for p = 1, one has the following property: R∗ w,1 (f ) = +∞ −∞ R∗ w,1 (χ{f >t} )dt (12) where R∗ w,1 (f ) is among Rw,1 (f ) and R± w,1 (f ) with χ{f >t} (vi ) = 0 if f (vi ) < t and χ{f >t} (vi ) = 1 if f (vi ) ≥ t. O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 18 / 30
26. 1 Introduction 2 Graphs and difference operators 3 Variational framework of active contours on graphs 4 Nonlocal active contours models as global minimizers 5 Applications O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 19 / 30
27. Nonlocal global minimizers of active contours models on graphs We consider this very general minimization problem on graphs: min f :V→{0,1} TVw (f ) + λH(f , f 0) (13) where TVw (f ) is a weighted total variation of f . Both TVw (f ) and H(f , f 0) are considered as satisfying the co-area formula. We restrict ourselves to the following minimization problem: min f :V→{0,1} Ew,1 (f ) = R∗ w,1 (f ) + λ vi ∈V r(f 0(vi ))f (vi ) (14) where R∗ w,1 (f ) is among Rw,1 (f ) and R± w,1 (f ). Both parts of the energy satisfy the co-area formula but this minimization problem is non-convex. O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 20 / 30
28. Convex relaxation The problem can be reformulated to consider a convex relaxation problem (Bresson & Chan, 2008) to determine a global minimizing solution. A new minimization problem is considered: min ˆ f :V→[0,1] ˆ Ew,1 (f ) = R∗ w,1 (f ) + λ vi ∈V r(f 0(vi ))f (vi ) (15) Every level set of a minimizer of this minimization is a solution of the original optimization problem. To obtain a global solution f : V → {0, 1}, one thresholds any function ˆ f : V → [0, 1] that is a solution of (15) and f = χA (ˆ f ). To solve (15), we solve the following system of equations: ∂ ˆ Ew,1 (f ) ∂f (vi ) = ∂R∗ w,1 (f ) ∂f (vi ) + λr(f 0(vi )) = 0, ∀vi ∈ V, (16) O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 21 / 30
29. Algorithm One has the following relation ∂Rw,1 (f ) ∂f (vi ) = 2 ∂R+ w,1 (f ) ∂f (vi ) = 2 ∂R− w,1 (f ) ∂f (vi ) = 2∆a w,1 f (vi ). For solving, we consider the linearized Gauss-Jacobi method and obtain the following iterative algorithm: f (t+1)(vi ) = −λr(f 0(vi )) + α vj ∼vi γf (t) w (vi , vj )f (t)(vj ) α vj ∼vi γf (t) w (vi , vj ) (17) where γf (t) w (vi , vj ) = w(vi , vj )1/2|f t(vi ) − f t(vj )|−1. With α = 1 for R± w,1 (f ) and α = 2 for Rw,1 (f ), and the initial condition f (0) = f 0. O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 22 / 30
30. 1 Introduction 2 Graphs and difference operators 3 Variational framework of active contours on graphs 4 Nonlocal active contours models as global minimizers 5 Applications O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 23 / 30
31. Chan-Vese on graphs We can directly express the discrete analogue on graphs of the CV model as the following minimization, with r(f 0(vi )) = (c1 − f 0(vi ))2 − (c2 − f 0(vi ))2 and c1 and c2 the average values inside and outside the object, then ∀vi ∈ V : min f :V→{0,1}    R∗ w,1 (f ) + λ vi ∈V f (vi )m(c1, vi ) + (1 − f (vi ))m(c2, vi )    (18) where m(a, vi ) = (a − f 0(vi ))2. The minimization of CV model on graphs is performed as an alternate minimization with respect to (c1, c2 ) and with respect to f . O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 24 / 30
32. CV on images Segmentation result on local (4-adjacency graph with Gaussian weights computed on pixel values) and nonlocal (4-adjacency graph coupled with a 4-Nearest Neighbor graph selected in a 9 × 9 window and Gaussian weights computed on 3 × 3 patches) graphs. O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 25 / 30
33. CV on Region Adjacency Graphs An initial contour, the considered graph (a super-pixel graph obtained from an over-segmentation with Gaussian weights on region mean values), and the obtained partition of the super-pixel graph. O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 26 / 30
34. CV on surfaces on meshes A mesh with an initial contour (on pig’s back), the considered mesh (a triangular mesh with Gaussian weights on vertices grey values) with an image of text projected onto the surface, and the obtained partition of the mesh (all the text is extracted). O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 27 / 30
35. CV on Image Database Graphs An image database with an initial random partition, the considered graph (a 10 nearest neighbors graph weighted with Gaussian weights on 16 × 16 vectors associated to the image of each vertex). O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 28 / 30
36. CV on Point Cloud Graphs O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 29 / 30
37. Thanks. Publications available at : http://www.info.unicaen.fr/∼lezoray O. L´ ezoray, A. Elmoataz, V.-T. Ta Nonlocal PdEs on graphs for active contours models 30 / 30