Adaptive and nonlocal approaches in Mathematical Morphology Olivier L´ ezoray Universit´ e de Caen Basse Normandie [email protected] http://lezoray.users.greyc.fr 

1 Introduction 2 PDEs-based morphological processing Operators on weighted graphs Construction of graphs - non locality Adaptive mathematical morphology on graphs 3 Algebraic MM – multivariate images 4 Conclusions & Actual Works O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 2 / 41 

1 Introduction 2 PDEs-based morphological processing 3 Algebraic MM – multivariate images 4 Conclusions & Actual Works O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 3 / 41 

Introduction Mathematical morphology (MM) offers a wide range of operators to address various image processing problems. These operators can be defined in terms of algebraic (discrete) sets or as partial differential equations (PDEs). There has been much interest recently in nonlocal image processing that highlights the need for adaptivity in Image Processing Our aim: to consider adaptive and nonlocal approaches in MM for both PDEs and Algebraic sets. O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 4 / 41 

1 Introduction 2 PDEs-based morphological processing Operators on weighted graphs Construction of graphs - non locality Adaptive mathematical morphology on graphs 3 Algebraic MM – multivariate images 4 Conclusions & Actual Works O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 5 / 41 

Weighted graphs Basics A weighted graph G = (V, E, w) consists in a finite set V = {v1, . . . , vN } of N vertices O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 6 / 41 

Weighted graphs Basics A weighted graph G = (V, E, w) consists in a finite set V = {v1, . . . , vN } of N vertices and a finite set E = {e1, . . . , eN￿ } ⊂ V × V of N￿ weighted edges. We assume G to be simple, undirected, with no self-loops and no multiple edges. O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 6 / 41 

Weighted graphs Basics A weighted graph G = (V, E, w) consists in a finite set V = {v1, . . . , vN } of N vertices and a finite set E = {e1, . . . , eN￿ } ⊂ V × V of N￿ weighted edges. We assume G to be simple, undirected, with no self-loops and no multiple edges. eij = (vi , vj ) is the edge of E that connects vertices vi and vj of V. Its weight, denoted by wij = w(vi , vj ), represents the similarity between its vertices. Similarities are usually computed by using a positive symmetric function w : V × V → R+ satisfying w(vi , vj ) = 0 if (vi , vj ) / ∈ E. w w w w w w w w w w w O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 6 / 41 

Weighted graphs Basics A weighted graph G = (V, E, w) consists in a finite set V = {v1, . . . , vN } of N vertices and a finite set E = {e1, . . . , eN￿ } ⊂ V × V of N￿ weighted edges. We assume G to be simple, undirected, with no self-loops and no multiple edges. eij = (vi , vj ) is the edge of E that connects vertices vi and vj of V. Its weight, denoted by wij = w(vi , vj ), represents the similarity between its vertices. Similarities are usually computed by using a positive symmetric function w : V × V → R+ satisfying w(vi , vj ) = 0 if (vi , vj ) / ∈ E. The notation vi ∼ vj is used to denote two adjacent vertices. O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 6 / 41 

Graph boundaries Let A be a set of connected vertices with A⊂V. We denote by ∂+A and ∂−A, the external and internal boundary sets of A, respectively. Ac =V \ A is the complement of A. For a given vertex vi ∈V, ∂+A = {vi ∈Ac : ∃vj ∈A with eij = (vi , vj )∈E} ∂−A = {vi ∈A : ∃vj ∈Ac with eij = (vi , vj )∈E} - - - - - - - - + + + + + + - + + + + + + + + - - - - - - + + + + + + + + + - + Blue vertices correspond to A. ’−’ sign for the internal boundary ∂−A and ’+’ sign for the external boundary ∂+A. O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 7 / 41 

Difference operators on weighted graphs ➲ Discretization of classical continuous differential geometry. The difference operator of f , dw : H(V) → H(E), is defined on an edge eij = (vi , vj ) ∈ E by: (dw f )(eij ) = (dw f )(vi , vj ) = w(vi , vj )1/2(f (vj ) − f (vi )) . (1) We also introduce internal and external differences called morphological difference operators: (d + w f )(vi , vj )=w(vi , vj )1/2 ￿ max ￿ f (vi ), f (vj ) ￿ −f (vi ) ￿ and (d − w f )(vi , vj )=w(vi , vj )1/2 ￿ f (vi )− min ￿ f (vi ), f (vj ) ￿￿ , (2) with the following properties (always positive) (d + w f )(vi , vj )= max ￿ 0, (dw f )(vi , vj ) ￿ (d − w f )(vi , vj )= − min ￿ 0, (dw f )(vi , vj ) ￿ A. Elmoataz, O. Lezoray, S. Bougleux, Nonlocal Discrete Regularization on Weighted Graphs: a framework for Image and Manifold Processing, IEEE transactions on Image Processing, Vol. 17, n7, pp. 1047-1060, 2008. O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 8 / 41 

Weighted gradient operator The weighted gradient operator of a function f ∈ H(V), at a vertex vi ∈ V, is the vector operator defined by (∇w f)(vi ) = [(dw f )(vi , vj ) : vj ∼ vi ]T , ∀(vi , vj ) ∈ E. (3) The Lp norm of this vector represents the local variation of the function f at a vertex of the graph (It is a semi-norm for p ≥ 1): ￿(∇w f)(vi )￿p = ￿ ￿ vj ∼vi w p/2 ij ￿ ￿ f (vj )−f (vi ) ￿ ￿p ￿1/p . (4) Similarly, we have with M + = max and M − = min (∇± w f)(vi )= ￿ ∂± vj f (vi ) ￿T (vi ,vj )∈E . ￿(∇± w f)(vi )￿p = ￿ ￿ vj ∼vi w(vi , vj )p/2 ￿ ￿ M ± ￿ 0, f (vj )−f (vi ) ￿￿ ￿p ￿1/p . (5) O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 9 / 41 

Constructing graphs Any discrete domain can be modeled by a weighted graph where each data point is represented by a vertex vi ∈ V. Unorganized data An unorganized set of points V ⊂ Rn can be seen as a function f 0 : V → Rm. The set of edges is defined by in modeling the neighborhood of each vertex based on similarity relationships between feature vectors. Typical graphs: k-nearest neighbors graphs and τ-neighborhood graphs. Organized data Typical cases of organized data are signals, gray-scale or color images (in 2D or 3D). The set of edge is defined by spatial relationships. Such data can be seen as functions f 0 : V ⊂ Zn → Rm. Typical graphs: pixel or region graphs. 12 2 — GRAPHES ET OPÉRATEURS ! !! ! ! ! ! !! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !! ! ! ! ! !! ! ! ! ! ! ! !! ! !! ! !!! !! ! ! ! ! !! ! ! ! !! ! ! !! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 (a) ! !! ! ! ! ! !! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !! ! ! ! ! !! ! ! ! ! ! ! !! ! !! ! !!! !! ! ! ! ! !! ! ! ! !! ! ! !! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 (b) k=3 ! !! ! ! ! ! !! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !! ! ! ! ! !! ! ! ! ! ! ! !! ! !! ! !!! !! ! ! ! ! !! ! ! ! !! ! ! !! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 (c) k=15 Fig. 2.1 – Exemples de graphes des k plus proches voisins. (a) : données initiales dans IR2, (b) : 3-Nng, (c) : 15-Nng. O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 10 / 41 

Weighting graphs For an initial function f 0 : V → Rm, similarity relationship between data can be incorporated within edges weights according to a measure of similarity g : E → [0, 1] with w(eij ) = g(eij ), ∀eij ∈ E. Each vertex vi is associated with a feature vector Ff0 τ : V → Rm×q where q corresponds to this vector size: Ff0 τ (vi ) = ￿ f 0(vj ) : vj ∈ Nτ (vi ) ∪ {vi } ￿T (6) with Nτ (vi ) = ￿ vj ∈ V \ {vi } : µ(vi , vj ) ≤ τ ￿ . For an edge eij and a distance measure ρ : Rm×q×Rm×q → R associated to Ff0 τ , we can have: g1 (eij ) =1 (unweighted case) , g2 (eij ) = exp ￿ −ρ ￿ Ff0 τ (vi ), Ff0 τ (vj ) ￿2 /σ2 ￿ with σ > 0 , g3 (eij ) =1/ ￿ 1 + ρ ￿ Ff0 τ (vi ), Ff0 τ (vj ) ￿￿ (7) O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 11 / 41 

Local to Non Local to Graphs In Image Processing, we can divide methods according to three different models: Local Processing: usual model where local interactions around one pixel are taken into account (Vector Median Filter, Anisotropic Filtering, Wavelets, Total Variation minimization with PDE, etc.), O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 12 / 41 

Local to Non Local to Graphs In Image Processing, we can divide methods according to three different models: Local Processing: usual model where local interactions around one pixel are taken into account (Vector Median Filter, Anisotropic Filtering, Wavelets, Total Variation minimization with PDE, etc.), Semi Local Processing: one takes into account larger neighborhood interactions favored by the image geometry (Yaroslavsky and Bilateral filters), O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 12 / 41 

Local to Non Local to Graphs In Image Processing, we can divide methods according to three different models: Local Processing: usual model where local interactions around one pixel are taken into account (Vector Median Filter, Anisotropic Filtering, Wavelets, Total Variation minimization with PDE, etc.), Semi Local Processing: one takes into account larger neighborhood interactions favored by the image geometry (Yaroslavsky and Bilateral filters), Non Local Processing: model recently proposed by Buades and Morel which replaces spatial constraints by pixel blocks (i.e. patchs) constraints in a large neighborhood. O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 12 / 41 

Local to Non Local to Graphs In Image Processing, we can divide methods according to three different models: Local Processing: usual model where local interactions around one pixel are taken into account (Vector Median Filter, Anisotropic Filtering, Wavelets, Total Variation minimization with PDE, etc.), Semi Local Processing: one takes into account larger neighborhood interactions favored by the image geometry (Yaroslavsky and Bilateral filters), Non Local Processing: model recently proposed by Buades and Morel which replaces spatial constraints by pixel blocks (i.e. patchs) constraints in a large neighborhood. w O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 12 / 41 

Graph topology Digital Image 

Graph topology Digital Image 8-neighborhood : 3 × 3 

Graph topology Digital Image 8-neighborhood : 3 × 3 24-neighborhood : 5 × 5 

Graph topology Digital Image 8-neighborhood : 3 × 3 24-neighborhood : 5 × 5 Local: a value is associ- ated to vertices 

Graph topology Digital Image 8-neighborhood : 3 × 3 24-neighborhood : 5 × 5 Local: a value is associ- ated to vertices Nonlocal: a patch (vector of values in a given neigh- borhood) is associated to vertices. O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 13 / 41 

Graph topology Digital Image 8-neighborhood : 3 × 3 24-neighborhood : 5 × 5 Local: a value is associ- ated to vertices Nonlocal: a patch (vector of values in a given neigh- borhood) is associated to vertices. With Graphs Nonlocal behavior is directly expressed by the graph topology. Patches are used to measure similarity between vertices. O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 13 / 41 

Graph topology Digital Image 8-neighborhood : 3 × 3 24-neighborhood : 5 × 5 Local: a value is associ- ated to vertices Nonlocal: a patch (vector of values in a given neigh- borhood) is associated to vertices. Consequences Nonlocal processing of images becomes local processing on similarity graphs. Our difference operators on graphs naturally enable local and nonlocal configurations (with the weight function and the graph topology) O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 13 / 41 

Adaptive mathematical morphology on graphs Our proposal Transcription of PDE MM on arbitrary graphs Introduction of nonlocal schemes for images Extend MM to the processing of arbitrary data (point clouds, databases, etc.) V.T. Ta, A. Elmoataz, O. L´ ezoray, Nonlocal PDEs-based Morphology on Weighted Graphs for Image and Data Processing, IEEE transactions on Image Processing, Vol. 20, n6, pp. 1504-1516, 2011. V.T. Ta, A. Elmoataz, O. L´ ezoray, Nonlocal Graph Morphology, International Symposium on Mathematical Morphology - Abstract Book, pp. 5-9, 2009. V.T. Ta, A. Elmoataz, O. L´ ezoray, Partial difference equations on graphs for mathematical morphology operators over images and manifolds, International Conference on Image Processing (IEEE), pp. 801-804, 2008. Winner of the IBM Student-Paper Award. V.T. Ta, A. Elmoataz, O. L´ ezoray, Partial Difference Equations over Graphs: Morphological Processing of Arbitrary Discrete Data, European Conference on Computer Vision, Vol. LNCS 5304, pp. 668-680, 2008. O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 14 / 41 

Mathematical Morphology: Continuous formulation For convex structuring elements, an alternative formulation in terms of Partial Differential Equations (PDE) has also been proposed. Given an initial function f 0 : Ω ⊂ R2 → R, a disc B = {z ∈ R2 : ￿z￿p ≤ 1}, one considers the following evolution equation ∂f ∂t = ∂tf = ±￿∇f￿p Solution of f (x, y, t) at time t > 0 provides dilation (with the plus sign) or erosion (with the minus sign) within a structuring element of size n∆t: δ(f ) = ∂tf = +￿∇f￿p and ￿(f ) = ∂tf = −￿∇f￿p with a size of 100∆t, ∆t = 0.25 and p = 1, p = 2, and p = ∞. O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 15 / 41 

MM: Transcription on graphs Transcription on graphs Given G = (V, E, w), f 0 : V → R, f (., 0) = f , ∀vi ∈ V, we define: δ : ∂tf (vi , t) = +￿(∇+ w f)(vi, t)￿p ￿ : ∂tf (vi , t) = −￿(∇− w f)(vi, t)￿p O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 16 / 41 

MM: Transcription on graphs Transcription on graphs Given G = (V, E, w), f 0 : V → R, f (., 0) = f , ∀vi ∈ V, we define: δ : ∂tf (vi , t) = +￿(∇+ w f)(vi, t)￿p ￿ : ∂tf (vi , t) = −￿(∇− w f)(vi, t)￿p A⊂V O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 16 / 41 

MM: Transcription on graphs Transcription on graphs Given G = (V, E, w), f 0 : V → R, f (., 0) = f , ∀vi ∈ V, we define: δ : ∂tf (vi , t) = +￿(∇+ w f)(vi, t)￿p ￿ : ∂tf (vi , t) = −￿(∇− w f)(vi, t)￿p - - - - - + + + + + + + + + - + A⊂V ∂+A = {vi / ∈A : ∃vj ∈A with eij ∈E} ∂−A = {vi ∈A : ∃vj / ∈A with eij ∈E} O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 16 / 41 

MM: Transcription on graphs Transcription on graphs Given G = (V, E, w), f 0 : V → R, f (., 0) = f , ∀vi ∈ V, we define: δ : ∂tf (vi , t) = +￿(∇+ w f)(vi, t)￿p ￿ : ∂tf (vi , t) = −￿(∇− w f)(vi, t)￿p - - - - - + + + + + + + + + - + A⊂V ∂+A = {vi / ∈A : ∃vj ∈A with eij ∈E} ∂−A = {vi ∈A : ∃vj / ∈A with eij ∈E} Dilation: adding vertices from ∂+A to A O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 16 / 41 

MM: Transcription on graphs Transcription on graphs Given G = (V, E, w), f 0 : V → R, f (., 0) = f , ∀vi ∈ V, we define: δ : ∂tf (vi , t) = +￿(∇+ w f)(vi, t)￿p ￿ : ∂tf (vi , t) = −￿(∇− w f)(vi, t)￿p - - - - - + + + + + + + + + - + A⊂V ∂+A = {vi / ∈A : ∃vj ∈A with eij ∈E} ∂−A = {vi ∈A : ∃vj / ∈A with eij ∈E} Dilation: adding vertices from ∂+A to A Erosion: removing vertices from ∂−A to A O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 16 / 41 

MM: Transcription on graphs Transcription on graphs Given G = (V, E, w), f 0 : V → R, f (., 0) = f , ∀vi ∈ V, we define: δ : ∂tf (vi , t) = +￿(∇+ w f)(vi, t)￿p ￿ : ∂tf (vi , t) = −￿(∇− w f)(vi, t)￿p A⊂V ∂+A = {vi / ∈A : ∃vj ∈A with eij ∈E} ∂−A = {vi ∈A : ∃vj / ∈A with eij ∈E} Dilation: adding vertices from ∂+A to A Erosion: removing vertices from ∂−A to A Dilation: maximizing a surface gain proportionally to ￿(∇+ w f)(vi )￿p Erosion: minimizing a surface gain proportionally to ￿(∇− w f)(vi )￿p O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 16 / 41 

MM: Transcription on graphs PDE MM: δ : ∂tf (x, t) = +￿∇f(x, t)￿p ￿ : ∂tf (x, t) = −￿∇f(x, t)￿p Transcription on graphs Given G = (V, E, w), f 0 : V → R, f (., 0) = f , ∀vi ∈ V, we define: δ : ∂tf (vi , t) = +￿(∇+ w f)(vi, t)￿p ￿ : ∂tf (vi , t) = −￿(∇− w f)(vi, t)￿p Since we can prove that for any level f l of f , we have: ￿(∇w fl)(vi )￿p = ￿ ￿(∇+ w fl)(vi )￿p if vi ∈ ∂+Al , ￿(∇− w fl)(vi )￿p if vi ∈ ∂−Al . (8) Lp norm: ￿(∇± w f)(vi )￿p = ￿ ￿ vj ∼vi w(vi , vj )p/2 ￿ ￿ M ± ￿ 0, f (vj )−f (vi ) ￿￿ ￿p ￿1/p , 0 < p < ∞ L∞ norm: ￿(∇± w f)(vi )￿∞ = max vj ∼vi ￿ w(vi , vj )1/2 ￿ ￿ M ± ￿ 0, f (vj )−f (vi ) ￿￿ ￿ ￿ with M + = max and M − = min O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 17 / 41 

Numerical resolution Iterative algorithms with discretization in time: f 0 : V → R, f (n)(vi )≈f (vi , n∆t) ￿ f (n+1)(vi )=f (n)(vi )±∆t￿(∇± w f(n))(vi )￿p f 0(vi )=f 0(vi ) Lp norm: f (n+1)(vi )=f (n)(vi )±∆t ￿ ￿ vj ∼vi w(vi , vj )p/2 ￿ ￿ M ± ￿ 0, f (vj )−f (vi ) ￿￿ ￿p ￿1/p L∞ norm: f (n+1)(vi )=f (n)(vi )±∆t max vj ∼vi ￿ w(vi , vj )1/2 ￿ ￿ M ± ￿ 0, f (vj )−f (vi ) ￿￿ ￿ ￿ For p = 2 and w = 1 on a grid, we recover the PDE numerical scheme of Osher & Sethian. For p = ∞, ∆t = 1, and w = 1, we recover the algebraic formulation with the structuring element expressed by the graph topology. O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 18 / 41 

Why Adaptive mathematical morphology ? Varying w and graph topology, we obtain adaptivity. Adaptivity with graph weights: example of a closing φ(f ) = ￿(δ(f ))) Unweighted Weighted Non local with patchs Initial Adaptivity with graph topology: example of a dilation Initial 25-adjacency 25-NNG O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 19 / 41 

Examples Dilation Closing Algebraic PDE Dilation Closing Weighted Non local patch O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 20 / 41 

Examples: closing φ(f ) = ￿(δ(f )) Initial Local Weighted Non local / patchs O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 21 / 41 

Examples: image databases f0 : V → IR256 Dilation Erosion Opening k-NNG Initial O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 22 / 41 

1 Introduction 2 PDEs-based morphological processing 3 Algebraic MM – multivariate images 4 Conclusions & Actual Works O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 23 / 41 

Introduction Algebraic Mathematical Morphology (MM) relies on the use of a complete lattice L. A complete lattice L is a non empty set equipped with a (partial or total) ordering relation, such that every non-empty subset P of L has a lower bound ∧P and an upper bound ∨P. Images are modeled by functions mapping their domain space Ω, into a complete lattice L. This is problematic for multivariate images since there is no natural ordering for vectors. Usually, this is performed with a lexicographic ordering (comparing features in a predefined cascade) that introduces strong dissymmetry. Our contribution: a general formalism that enables nonlocal and multivariate morphological processing. O. Lezoray, A. Elmoataz, Nonlocal and Multivariate Mathematical Morphology, International Conference on Image Processing (IEEE), 2012. O. Lezoray, C. Charrier, A. Elmoataz, Learning complete lattices for manifold mathematical morphology, International Symposium on Mathematical Morphology - Abstract Book, pp. 1-4, 2009. O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 24 / 41 

Ordering of vector data: Complete Lattices in Rp A multivariate image can be represented by the mapping f : Ω ⊂ Zl → T ⊂ Rp where l is the image dimension, p the number of channels, and T is a non-empty set. One way to define an ordering relation between vectors of T is to use the framework of h-orderings (Goutsias et al.). This corresponds to defining a surjective transform h from Rp to L where L is a complete lattice equipped with the lexicographic ordering: With h : T → L, and x → h(x) then ∀(xi , xj ) ∈ T × T , xi ≤h xj ⇔ h(xi ) ≤ h(xj ) (9) Then, the following equivalences can be considered: (total ordering on T ) ⇔ (bijective application h : T → L) ⇔ (space filling curve in T ) O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 25 / 41 

Rank transform Another equivalence can be considered: (total ordering on T )⇔(rank transformation on T ) A total h-ordering ≤h orders all the vectors of the set T : sorting all the vectors and retaining their rank in the ordering corresponds to creating explicitly the complete lattice (T , ≤h ). Once the complete lattice is created, each element of the multivariate image can be replaced by its rank, creating a rank image. This (scalar) rank image is the lattice representation of the multivalued image according to the ordering strategy ≤h . Image of 256 colors Rank Image (T , ≤h) O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 26 / 41 

Complete Lattice Learning Usual approaches to mathematical morphology first define a total ordering relation (usually a specific lexicographic ordering) that induces a complete lattice. We explicitly learn the complete lattice from a multivariate image using h-ordering. The projection h cannot be linear since a distortion of the space is inevitable: use nonlinear projection techniques (Manifold Learning) to construct h. Constructing the complete lattice of an image with manifold learning directly from all the pixels is computationally unfeasible. We propose a three-step strategy towards constructing the h-ordering: Data Quantization, Manifold Learning, Out of Sample Extension O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 27 / 41 

Data Quantization We reduce the amount of data by Vector Quantization (VQ). VQ maps a vector x to another vector x￿ that belongs to n prototype vectors (a dictionary). A dictionary D is built from a training set I of size m (m ￿ n). A VQ algorithm has to produce a set D of prototypes x￿ that minimizes the distortion defined by 1 m m ￿ i=1 min 1≤j≤n ￿xi − x￿ j ￿2 LBG is one algorithm that can build such a dictionary. It is an iterative algorithm that produces n = 2k prototypes after k iterates. Given a multivariate image of m pixels, VQ is applied to construct a dictionary D = {x￿ 1 , . . . , x￿ n } where x￿ i ∈ Rp. O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 28 / 41 

Manifold Learning We construct the transformation h on D with manifold learning. Manifold learning consists in searching for a new representation {y1 , · · · , yn } with yi ∈ Rq from the n vectors {x￿ 1 , · · · , x￿ n } with x￿ i ∈ Rp of the dictionary D. Given a similarity matrix Wij = k(x￿ i , x￿ j ) = e ￿ −||x￿ i −x￿ j ||2 σ2 ￿ , the normalized Laplacian is defined by ˜ L = I − D − 1 2 WD − 1 2 Laplacian Eigenmaps manifold learning consists in searching for a new representation obtained by minimizing 1 2 ￿ ij ￿ ￿yi − yj ￿ ￿ 2 Wij = Tr(YT ˜ LY) with Y = [y1 , · · · , yn ] The low-dimensional representation is obtained by considering the q lowest eigenvectors with q ￿ p and is defined by the following operator hD : x￿ i → (φ1 (x￿ i ), · · · , φq (x￿ i ))T where φk (x￿ i ) is the i th coordinate of eigenvector φk . O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 29 / 41 

Out of sample extension The projection h has to be defined for all the vectors of the image and not only its dictionary with hD . The dictionary D is a sub-manifold of the complete lattice: extend eigenfunctions computed on D to T . The Nystr¨ om method interpolates the value of eigenvectors computed on n sample vectors x￿ i to m novel vectors xi φk (xj ) = 1 λk n ￿ i=1 φk (x￿ i )k(xj , x￿ i ) λk is an eigenvalue of of ˜ L, then 1 − λk is an eigenvalue of D − 1 2 WD − 1 2 . Applying the Nystr¨ om extension to compute the eigenvectors of the normalized Laplacian ˜ Lφk = λk φk , we get φk (xj ) = 1 1 − λk n ￿ i=1 k(xj , x￿ i ) ￿ d(xj )d(x￿ i ) We can compute the projection h for any pixel of the image. O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 30 / 41 

Complete Lattice construction f : Ω → Rp that provides a set I = {x1, · · · , xm } of m vectors in Rp, A dictionary D = {x￿ 1 , · · · , x￿ n } of n vectors in Rp is computed, Manifold Learning is performed on D and a new representation hD is obtained and extended with the Nystrm extension as h : Rp → Rq as h(x) = (φ1 (x), · · · , φq (x))T Vectors of f are sorted according to ≤h (the conditional total ordering on h(x)) and a sorted image fs : [0, m] → Rp is obtained, The rank of a vector on the complete lattice L is defined as r : Rq → [0, m] A rank image fr : Ω → [0, m] is constructed with fr (pi ) = (r ◦ h ◦ f )(pi ) and pi ∈ Ω. The original image is recovered exactly since f (pi ) = (fs ◦ fr )(pi ). Given a specific morphological processing g, the corresponding processed multivariate image is obtained by g(f (pi )) = (fs ◦ g ◦ fr )(pi ). O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 31 / 41 

Complete Lattice construction - Example f D hD h Local Nonlocal Local fr Local fs Nonlocal fr Nonlocal fs O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 32 / 41 

Initial Quantized Rank QV L QV Rank L Lexicographic Rank Lexicographic L Lexicographic Lexicographic IHSL IHSL Rank α-trimmed L α-trimmed Rank Bit-Mixing L Bit-Mixing Lexicographic Lexicographic Rank Laplacian L Laplacian Eigenmaps Eigenmaps O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 33 / 41 

IHSL Lexicographic Laplacian EigenMaps O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 34 / 41 

synthetic bpp 0 2 4 6 8 10 LE LX ATLX VQ BM LXHSL natural1 bpp 0 2 4 6 8 natural2 bpp 0 2 4 6 8 Average bpp 0 2 4 6 8 The compression ability with Lossless JPEG of the rank image is used as measure of performance (used in the literature of palette re-ordering schemes). O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 35 / 41 

Examples f Local fr Nonlocal fr The dictionary size is 32, the structuring element is a circle of radius 5, patches are 9 × 9. O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 36 / 41 

Examples ￿ δ Closing ϕ = ￿δ Local Nonlocal O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 37 / 41 

Examples ∇ = δ − ￿ WTH = f − γ Watershed Local Nonlocal O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 38 / 41 

1 Introduction 2 PDEs-based morphological processing 3 Algebraic MM – multivariate images 4 Conclusions & Actual Works O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 39 / 41 

Conclusions - & Actual Works Contributions A formulation of PDEs-based MM on graphs A formulation of Algebraic MM for multivariate data Both formulations enable to introduce nonlocal configurations Actual/Future Works Link between morphological gradients on graphs and p-Laplacian on graphs, Study the interest of nonlocal configurations for morphological processing, e.g., for adaptive morphological scale space, granulometries, O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 40 / 41 

The End. Thanks. Publications available at : http://lezoray.users.greyc.fr O. L´ ezoray (University of Caen) Adaptive and nonlocal approaches in MM 41 / 41