High Dynamic Range Image processing using manifold-based ordering Olivier L´ ezoray Normandie Univ, UNICAEN, ENSICAEN, CNRS, GREYC, 14000 Caen, France [email protected] https://lezoray.users.greyc.fr 

1 Introduction and Motivations 2 Learning an order of HDR vectors 3 HDR image representation and processing 4 Tone Mapping 5 Conclusions and perspectives 2 / 35 

1 Introduction and Motivations 2 Learning an order of HDR vectors 3 HDR image representation and processing 4 Tone Mapping 5 Conclusions and perspectives 3 / 35 

High Dynamic Range Imaging Dynamic range : gap between the highest and lowest recorded level of light In real scenes the dynamic range is very large and it is impossible to capture them with digital cameras One solution can be to vary exposure to capture much dynamic range 4 / 35 

Exposure Bracketing Exposure bracketing : Within a short period of time, multiple shots with different exposures are captured An HDR radiance map is recovered by combining the images (using the correlations among the bracketing images) 5 / 35 

Tone mapping HDR images cannot be displayed directly so Tone Mapping is applied to transform it into a displayable 24 bits image 6 / 35 

Processing HDR images ? Why ? to perform filtering, simplification, enhancement, . . . Most algorithms for the processing of HDR images (including Tone Mapping) operate on the log-luminance of an HDR image This projection on the log-luminance curve is sub-optimal since the manifold where HDR vectors live is obviously much more nonlinear than a simple projection onto a log-luminance curve Can we directly perform image processing of raw HDR images rather than on their log-luminance or their tone mapped LDR ones ? 7 / 35 

Processing RAW HDR images We want to be able to perform classical image processing operation directly on RAW HDR images We propose an approach that consists in learning the manifold of HDR image vectors to build a new HDR image representation in the form of an index image associated with an ordering of the HDR pixels’ vectors. 8 / 35 

1 Introduction and Motivations 2 Learning an order of HDR vectors 3 HDR image representation and processing 4 Tone Mapping 5 Conclusions and perspectives 9 / 35 

1 Introduction and Motivations 2 Learning an order of HDR vectors 3 HDR image representation and processing 4 Tone Mapping 5 Conclusions and perspectives 10 / 35 

Notations We consider HDR images encoded with the Open EXR standard that associates three floating point values to pixels (RGB values of radiance). An HDR image is represented by the mapping f : Ω ⊂ Z2 → T ⊂ R3 where T is a non-empty set of image multivariate HDR vectors. To each pixel pi ∈ Ω of an image is associated a vector vi = f (pi ). The set T = {v1, · · · , vm} denotes all the HDR vectors associated to all pixels in the image, and is of size m. T [i] = vi will denote the i-th element of a set. 11 / 35 

Complete Lattice To process HDR images, we build a new HDR image representation in the form of an index image associated with an ordering of the HDR pixels’ vectors T of an image. Ordering all the values of the set T can be done with the use of an ordering relation within HDR vectors. This amounts to dispose of a complete lattice (T , ≤) but there is no universal order for vectorial data The framework of h-orderings can be considered for that : construct a surjective mapping h from T to L where L is a complete lattice equipped with the conditional total ordering h : T → L and v → h(v), ∀(vi , vj ) ∈ T × T vi ≤h vj ⇔ h(vi ) ≤ h(vj ) . ≤h will denote such an h-ordering 12 / 35 

Manifold-based ordering × Problem : the projection operator h cannot be linear since a distortion of the space is inevitable ! 13 / 35 

Manifold-based ordering × Problem : the projection operator h cannot be linear since a distortion of the space is inevitable ! Solution : Consider non-linear dimensionality reduction with Laplacian Eigenmaps that corresponds to learn the manifold where the HDR vectors live. 13 / 35 

Manifold-based ordering × Problem : the projection operator h cannot be linear since a distortion of the space is inevitable ! Solution : Consider non-linear dimensionality reduction with Laplacian Eigenmaps that corresponds to learn the manifold where the HDR vectors live. × Problem : Non-linear dimensionality reduction directly on the set T of HDR vectors of the image is not tractable in reasonable time, especially for large images ! 13 / 35 

Manifold-based ordering × Problem : the projection operator h cannot be linear since a distortion of the space is inevitable ! Solution : Consider non-linear dimensionality reduction with Laplacian Eigenmaps that corresponds to learn the manifold where the HDR vectors live. × Problem : Non-linear dimensionality reduction directly on the set T of HDR vectors of the image is not tractable in reasonable time, especially for large images ! Solution : Consider a more efficient strategy. 13 / 35 

Manifold-based ordering × Problem : the projection operator h cannot be linear since a distortion of the space is inevitable ! Solution : Consider non-linear dimensionality reduction with Laplacian Eigenmaps that corresponds to learn the manifold where the HDR vectors live. × Problem : Non-linear dimensionality reduction directly on the set T of HDR vectors of the image is not tractable in reasonable time, especially for large images ! Solution : Consider a more efficient strategy. Proposed Three-Step Strategy Dictionary Learning to produce a set D from the set of initial vectors T Laplacian Eigenmaps Manifold Learning on the dictionary D to obtain a projection operator hD Out of sample extension to extrapolate hD to T and define h 13 / 35 

Learning the manifold of HDR vectors Step 1: Dictionary Construction Build from T , by Vector Quantization, a dictionary D = {x1 , . . . , xp } with p m Step 2: Manifold Learning on the dictionary Laplacian Eigenmaps Manifold Learning searches Φ such that 1 2 ij Φ(xi ) − Φ(xj ) 2 KD (i, j) = Tr(ΦT LΦ) with ΦT DD Φ = I. Compute the similarity matrix KD between vectors xi ∈ D with KD (i, j) = k(xi , xj ) = exp − xi −xj 2 2 σ2 with σ = max (x i ,x j )∈D xi − xj 2 2 Compute the degree diagonal matrix DD of KD Solution is obtained with the eigen-decomposition of the normalized Laplacian L = I − D− 1 2 D KD D− 1 2 D as L = ΦD ΠD ΦT D with eigenvectors ΦD = [Φ1 D , · · · , Φp D ] and eigenvalues ΠD = diag[λ1, · · · , λp ] 14 / 35 

Learning the manifold of HDR vectors A new representation hD (xi ) is obtained for each element xi of the dictionary D: hD : xi → (φ1 D (xi ), · · · , φp D (xi ))T ∈ Rp . This constructs the lattice (D, ≤hD ) with a hD -ordering, valid only on D. Step 3: Extrapolation of the projection ΦD to all the vectors of T Compute similarity matrices KT on T and KDT between sets D and T Compute the degree diagonal matrix DDP of KDP Extrapolate eigenvectors obtained from D to T with ˜ Φ = D− 1 2 DT KT DT D− 1 2 D ΦD (diag[1] − ΠD )−1 Output: The final projection h : T ⊂ R3 → L ⊂ Rp on the manifold is given by ˜ Φ and defined as h(x) = ( ˜ φ1 (x), · · · , ˜ φp (x))T . The complete lattice (T , ≤h ) is obtained by using the conditional ordering on h. 15 / 35 

1 Introduction and Motivations 2 Learning an order of HDR vectors 3 HDR image representation and processing 4 Tone Mapping 5 Conclusions and perspectives 16 / 35 

1 Introduction and Motivations 2 Learning an order of HDR vectors 3 HDR image representation and processing 4 Tone Mapping 5 Conclusions and perspectives 17 / 35 

HDR Image representation Given the complete lattice (T , ≤h ), a sorted permutation P of T is constructed P = {v1 , · · · , vm } with vi ≤h vi+1 , ∀i ∈ [1, (m − 1)]. From the ordering, an index image I : Ω ⊂ Z2 → [1, m] is defined as: I(pi ) = {k | vk = f (pi ) = vi } . The pair (I, P) provides a new HDR image representation (the index and the palette of ordered HDR vectors). The original image f can be directly recovered since one has f (pi ) = P[I(pi )] = T [i] = vi 18 / 35 

HDR Image representation Figure: From left to right : the learned manifold (shown on the three first eigenvectors) on HDR image, the deduced ordering of HDR vectors (converted in colors for visualization, line by line from top-left to bottom right), the resulting index image I, and the usual log image of the luminance of HDR pixels. 19 / 35 

HDR Image processing The index image I can be directly used to process the HDR image, however, to be able to reconstruct the result, the values have to be kept within [1, m]. · A processing g operating on I must be a vector preserving one : g(f (pi )) = P[g(I(pi ))] . Typical vector preserving image processing operations: morphological and median. Erosion and dilation of an HDR image f at pixel pi ∈ Ω by a structuring element B ⊂ Ω as: B (f )(pi ) = {P[∧I(pj )], pj ∈ B(pi )} δB (f )(pi ) = {P[∨I(pj )], pj ∈ B(pi )} · To illustrate the processing of HDR images with the proposed representation, we present the result after Tone Mapping with the approach of Durand and Dorsey. F. Durand and J. Dorsey, “Fast bilateral filtering for the display of high-dynamic-range images,” ACM Trans. Graph., vol. 21, no. 3, pp. 257–266, 2002. 20 / 35 

Examples - Erosion, dilation Original Image B δB Figure: HDR Morphological Processing (B is a square of side 9 pixels) 21 / 35 

Examples - Gradient, Opening, Closing δB − B γB φB Figure: HDR Morphological Processing (B is a square of side 9 pixels) 22 / 35 

Median Filtering, Leveling Figure: HDR image simplification with Median filtering, Leveling. 23 / 35 

Detail enhancement Figure: HDR image detail manipulation with Contrast Mapping (morphological shock filter) and OCCO detail manipulation (filter with an OCCO filter and boost the residual by a factor of 1.5). 24 / 35 

1 Introduction and Motivations 2 Learning an order of HDR vectors 3 HDR image representation and processing 4 Tone Mapping 5 Conclusions and perspectives 25 / 35 

1 Introduction and Motivations 2 Learning an order of HDR vectors 3 HDR image representation and processing 4 Tone Mapping 5 Conclusions and perspectives 26 / 35 

Tone Mapping by layer decomposition Classical Approach of Durand and Dorsey Compute the log-luminance Ll = log(L(f )) with L(f ) = 20fR +40fG +fB 61 Filter Ll with a bilateral filter to obtain a base layer Bl = BF(Ll ), and a detail layer Dl = Ll − Bl The tone-mapped image is obtained with f∗ = f∗ L(f ) exp (τBl + (Ll − Bl )) 1 γ with with f the RGB tone mapped image, τ a contrast parameter and γ a gamma correction parameter (fixed to 50 and 2.2). Our proposal : · Operate directly on the HDR new representation and NOT on the log-luminance image · replace the bilateral filter by a Morphological Filter · The base layer is replaced by Bl = log(L(MF(f ))) where MF(f ) is a morphological filter on the proposed HDR image representation 27 / 35 

Result - Base Layer DD Ours 28 / 35 

Result - Detail Layer DD Ours 29 / 35 

Result - Final Tone Mapping with log-PSNR values DD - 154.12dB Ours - 157.61dB 30 / 35 

Result - Final Tone Mapping 31 / 35 

1 Introduction and Motivations 2 Learning an order of HDR vectors 3 HDR image representation and processing 4 Tone Mapping 5 Conclusions and perspectives 32 / 35 

1 Introduction and Motivations 2 Learning an order of HDR vectors 3 HDR image representation and processing 4 Tone Mapping 5 Conclusions and perspectives 33 / 35 

Conclusions and perspectives Proposed a new representation of HDR images in the form of an index and a palette The palette is obtained by ordering the HDR vectors using a manifold-based ordering, and an index is deduced from it. Vector-preserving operations have been proposed for the processing of raw HDR images. The processing can be used to filter or enhance the images or even for Tone Mapping. · Consider non vector-preserving methods by processing the palette instead of the index image. 34 / 35 

The end Publications available at : https://lezoray.users.greyc.fr Work funded under ANR-14-CE27-0001 GRAPHSIP. 35 / 35