Nonlocal and Multivariate Mathematical Morphology Olivier L´ ezoray Universit´ e de Caen Basse Normandie [email protected] http://lezoray.users.greyc.fr ICIP 2012 Joint work with A. Elmoataz. 

1 Introduction 2 Complete Lattices in Rp 3 Complete Lattice Learning 4 Examples 5 Conclusions & Actual Works O. L´ ezoray (University of Caen) Nonlocal and Multivariate Mathematical Morphology ICIP 2012 2 / 28 

1 Introduction 2 Complete Lattices in Rp 3 Complete Lattice Learning 4 Examples 5 Conclusions & Actual Works O. L´ ezoray (University of Caen) Nonlocal and Multivariate Mathematical Morphology ICIP 2012 3 / 28 

Introduction Mathematical morphology (MM) offers a wide range of operators to address various image processing problems. These operators can be defined in terms of algebraic (discrete) sets or as partial differential equations (PDEs). There has been much interest recently in nonlocal image processing that highlights the need for adaptivity in Image Processing Our aim: to consider nonlocal and multivariate approaches in MM for Algebraic sets. O. L´ ezoray (University of Caen) Nonlocal and Multivariate Mathematical Morphology ICIP 2012 4 / 28 

Mathematical Morphology: Algebraic formulation Nonlinear scale-space approaches based on Mathematical Morphology (MM) operators are one of the most important tools in image processing. The two fundamental operators in Mathematical Morphology are dilation and erosion. Dilation δ of a function f 0 : Ω ⊂ R2 → R consists in replacing the function value by the maximum value within a structuring element B such that: δB f 0(x, y) = max f 0(x + x , y + y )|(x , y ) ∈ B Erosion is computed by: B f 0(x, y) = min f 0(x + x , y + y )|(x , y ) ∈ B O. L´ ezoray (University of Caen) Nonlocal and Multivariate Mathematical Morphology ICIP 2012 5 / 28 

1 Introduction 2 Complete Lattices in Rp 3 Complete Lattice Learning 4 Examples 5 Conclusions & Actual Works O. L´ ezoray (University of Caen) Nonlocal and Multivariate Mathematical Morphology ICIP 2012 6 / 28 

Complete Lattices Algebraic Mathematical Morphology (MM) relies on the use of a complete lattice L. A complete lattice L is a non empty set equipped with a (partial or total) ordering relation, such that every non-empty subset P of L has a lower bound ∧P and an upper bound ∨P. Images are modeled by functions mapping their domain space Ω, into a complete lattice L. This is problematic for multivariate images since there is no natural ordering for vectors. Usually, this is performed with a lexicographic ordering (comparing features in a predefined cascade) that introduces strong dissymmetry. Our contribution: a general formalism that enables nonlocal and multivariate morphological processing. O. L´ ezoray (University of Caen) Nonlocal and Multivariate Mathematical Morphology ICIP 2012 7 / 28 

Ordering of vector data: Complete Lattices in Rp A multivariate image can be represented by the mapping f : Ω ⊂ Zl → T ⊂ Rp where l is the image dimension, p the number of channels, and T is a non-empty set. One way to define an ordering relation between vectors of T is to use the framework of h-orderings (Goutsias et al.). This corresponds to defining a surjective transform h from Rp to L where L is a complete lattice equipped with the lexicographic ordering: With h : T → L, and x → h(x) then ∀(xi , xj ) ∈ T × T , xi ≤h xj ⇔ h(xi ) ≤ h(xj ) (1) Then, the following equivalences can be considered: (total ordering on T ) ⇔ (bijective application h : T → L) ⇔ (space filling curve in T ) O. L´ ezoray (University of Caen) Nonlocal and Multivariate Mathematical Morphology ICIP 2012 8 / 28 

Rank transform Another equivalence can be considered: (total ordering on T )⇔(rank transformation on T ) A total h-ordering ≤h orders all the vectors of the set T : sorting all the vectors and retaining their rank in the ordering corresponds to creating explicitly the complete lattice (T , ≤h ). Once the complete lattice is created, each element of the multivariate image can be replaced by its rank, creating a rank image. Image of 256 colors Rank Image (T , ≤h) This (scalar) rank image is the lattice representation of the multivalued image according to the ordering strategy ≤h . O. L´ ezoray (University of Caen) Nonlocal and Multivariate Mathematical Morphology ICIP 2012 9 / 28 

1 Introduction 2 Complete Lattices in Rp 3 Complete Lattice Learning 4 Examples 5 Conclusions & Actual Works O. L´ ezoray (University of Caen) Nonlocal and Multivariate Mathematical Morphology ICIP 2012 10 / 28 

Complete Lattice Learning Usual approaches to mathematical morphology first define a total ordering relation (usually a specific lexicographic ordering) that induces a complete lattice. We explicitly learn the complete lattice from a multivariate image using h-ordering. The projection h cannot be linear since a distortion of the space is inevitable. Solution: use nonlinear projection techniques (Manifold Learning) to construct h. Constructing the complete lattice of an image with manifold learning directly from all the pixels is computationally unfeasible. We propose a three-step strategy towards constructing the h-ordering: Data Quantization, Manifold Learning, Out of Sample Extension O. L´ ezoray (University of Caen) Nonlocal and Multivariate Mathematical Morphology ICIP 2012 11 / 28 

Data Quantization We reduce the amount of data by Vector Quantization (VQ). VQ maps a vector x to another vector x that belongs to n prototype vectors (a dictionary). A dictionary D is built from a training set I of size m (m n). A VQ algorithm has to produce a set D of prototypes x that minimizes the distortion defined by 1 m m i=1 min 1≤j≤n xi − xj 2 LBG is one algorithm that can build such a dictionary. It is an iterative algorithm that produces n = 2k prototypes after k iterates. Given a multivariate image of m pixels, VQ is applied to construct a dictionary D = {x1 , . . . , xn } where xi ∈ Rp. O. L´ ezoray (University of Caen) Nonlocal and Multivariate Mathematical Morphology ICIP 2012 12 / 28 

Manifold Learning We construct the transformation h on D with manifold learning. Manifold learning consists in searching for a new representation {y1 , · · · , yn } with yi ∈ Rq from the n vectors {x1 , · · · , xn } with xi ∈ Rp of the dictionary D. Given a similarity matrix Wij = k(xi , xj ) = e −||xi −xj ||2 σ2 , the normalized Laplacian is defined by ˜ L = I − D− 1 2 WD− 1 2 Laplacian Eigenmaps manifold learning consists in searching for a new representation obtained by minimizing 1 2 ij yi − yj 2 Wij = Tr(YT ˜ LY) with Y = [y1 , · · · , yn ] The low-dimensional representation is obtained by considering the q lowest eigenvectors with q p and is defined by the following operator hD : xi → (φ1 (xi ), · · · , φq (xi ))T where φk (xi ) is the ith coordinate of eigenvector φk . O. L´ ezoray (University of Caen) Nonlocal and Multivariate Mathematical Morphology ICIP 2012 13 / 28 

Out of sample extension The projection h has to be defined for all the vectors of the image and not only its dictionary with hD . The dictionary D is a sub-manifold of the complete lattice: extend eigenfunctions computed on D to T . The Nystr¨ om method interpolates the value of eigenvectors computed on n sample vectors xi to m novel vectors xi φk (xj ) = 1 λk n i=1 φk (xi )k(xj , xi ) λk is an eigenvalue of of ˜ L, then 1 − λk is an eigenvalue of D− 1 2 WD− 1 2 . Applying the Nystr¨ om extension to compute the eigenvectors of the normalized Laplacian ˜ Lφk = λk φk , we get φk (xj ) = 1 1 − λk n i=1 k(xj , xi ) d(xj )d(x i ) We can compute the projection h for any pixel of the image. O. L´ ezoray (University of Caen) Nonlocal and Multivariate Mathematical Morphology ICIP 2012 14 / 28 

Complete Lattice construction – summary f : Ω → Rp that provides a set I = {x1, · · · , xm} of m vectors in Rp, A dictionary D = {x1 , · · · , xn } of n vectors in Rp is computed, Manifold Learning is performed on D and a new representation hD is obtained and extended with the Nystr¨ om extension as h : Rp → Rq as h(x) = (φ1 (x), · · · , φq (x))T Vectors of f are sorted according to ≤h (the conditional total ordering on h(x)) and a sorted image fs : [0, m] → Rp is obtained, The rank of a vector on the complete lattice L is defined as r : Rq → [0, m] A rank image fr : Ω → [0, m] is constructed with fr (pi ) = (r ◦ h ◦ f )(pi ) and pi ∈ Ω. The original image is recovered exactly since f (pi ) = (fs ◦ fr )(pi ). Given a specific morphological processing g, the corresponding processed multivariate image is obtained by g(f (pi )) = (fs ◦ g ◦ fr )(pi ). O. L´ ezoray (University of Caen) Nonlocal and Multivariate Mathematical Morphology ICIP 2012 15 / 28 

1 Introduction 2 Complete Lattices in Rp 3 Complete Lattice Learning 4 Examples 5 Conclusions & Actual Works O. L´ ezoray (University of Caen) Nonlocal and Multivariate Mathematical Morphology ICIP 2012 16 / 28 

Complete Lattice construction - Example f D hD h Local Nonlocal Local fr Local fs Nonlocal fr Nonlocal fs O. L´ ezoray (University of Caen) Nonlocal and Multivariate Mathematical Morphology ICIP 2012 17 / 28 

Initial Quantized Rank QV L QV Rank L Lexicographic Rank Lexicographic L Lexicographic Lexicographic IHSL IHSL Rank α-trimmed L α-trimmed Rank Bit-Mixing L Bit-Mixing Lexicographic Lexicographic Rank Laplacian L Laplacian Eigenmaps Eigenmaps O. L´ ezoray (University of Caen) Nonlocal and Multivariate Mathematical Morphology ICIP 2012 18 / 28 

IHSL Lexicographic Laplacian EigenMaps O. L´ ezoray (University of Caen) Nonlocal and Multivariate Mathematical Morphology ICIP 2012 19 / 28 

Examples f Local fr Nonlocal fr The dictionary size is 32, the structuring element is a circle of radius 5, patches are 9 × 9. O. L´ ezoray (University of Caen) Nonlocal and Multivariate Mathematical Morphology ICIP 2012 20 / 28 

Examples δ Closing ϕ = δ Local Nonlocal O. L´ ezoray (University of Caen) Nonlocal and Multivariate Mathematical Morphology ICIP 2012 21 / 28 

Examples δ Closing ϕ = δ Local Nonlocal O. L´ ezoray (University of Caen) Nonlocal and Multivariate Mathematical Morphology ICIP 2012 22 / 28 

Examples ∇ = δ − WTH = f − γ Watershed Local Nonlocal O. L´ ezoray (University of Caen) Nonlocal and Multivariate Mathematical Morphology ICIP 2012 23 / 28 

Examples ∇ = δ − WTH = f − γ Watershed Local Nonlocal O. L´ ezoray (University of Caen) Nonlocal and Multivariate Mathematical Morphology ICIP 2012 24 / 28 

1 Introduction 2 Complete Lattices in Rp 3 Complete Lattice Learning 4 Examples 5 Conclusions & Actual Works O. L´ ezoray (University of Caen) Nonlocal and Multivariate Mathematical Morphology ICIP 2012 25 / 28 

Conclusions & Actual Works Contributions Unsupervised approach towards the construction of complete lattices for multivariate images Lead to a framework for unsupervised multivariate mathematical morphology No prior information (e.g., component prioritization) is required Introduced novel nonlocal flat algebraic mathematical morphology operators Actual/Future Works Learn Complete lattices for image databases Introduce user constraints in the complete lattice construction (aka semi-supervised complete lattice learning) Study the interest of nonlocal configurations for morphological processing, e.g., for adaptive morphological scale space, granulometries, O. L´ ezoray (University of Caen) Nonlocal and Multivariate Mathematical Morphology ICIP 2012 26 / 28 

The End. Thanks. Publications available at : http://lezoray.users.greyc.fr O. L´ ezoray (University of Caen) Nonlocal and Multivariate Mathematical Morphology ICIP 2012 27 / 28