Optimal Transport for Image Assimilation

by npapadakis

Embed

Start on current slide

Slide 1

Slide 1 text

Optimal Transport for Image Assimilation Nicolas Papadakis Optimal Transport for Image Assimilation 1/38

Slide 2

Slide 2 text

1 Introduction 2 Image Assimilation 3 Wasserstein distance 4 Wasserstein Image Assimilation 5 Conclusion Optimal Transport for Image Assimilation Introduction 2/38

Slide 3

Slide 3 text

How to use images for ocean state prediction? Optimal Transport for Image Assimilation Introduction 3/38

Slide 4

Slide 4 text

Assimilation: how to predict the state of a dynamical system? Meteorology Glaciology Optimal Transport for Image Assimilation Health sciences Introduction 4/38

Slide 5

Slide 5 text

Heat diffusion Physical variable: u(x, t) room temperature Ω at time t Initial conditions Homogeneous temperature u(., 0) = 15◦ Add an Heater at xH : u(xH , 0) = 20◦ Optimal Transport for Image Assimilation Introduction 5/38

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Heat diffusion Dynamical model: heat equation during 3 hours   ∂t u(x, t) = ∆u(x, t) x ∈ Ω\xh u(xH , t) = 20◦  ∂u (x) =0 x ∈ ∂Ω ∂⃗n Optimal Transport for Image Assimilation Introduction 5/38

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Heat diffusion Dynamical model: heat equation during 3 hours   ∂t u(x, t) = ∆u(x, t) x ∈ Ω\xh u(xH , t) = 20◦  ∂u (x) =0 x ∈ ∂Ω ∂⃗n Observations: Thermometer measure at θ: 17◦ ̸= u(xθ , 3) Optimal Transport for Image Assimilation Introduction 5/38

Slide 10

Slide 10 text

Heat diffusion Dynamical model: heat equation during 3 hours   ∂t u(x, t) = ∆u(x, t) x ∈ Ω\xh u(xH , t) = 20◦  ∂u (x) = g(x) x ∈ ∂Ω ∂⃗n Observations: Thermometer measure at θ: 17◦ ̸= u(xθ , 3) Uncertainties on boundary conditions g (ventilation...) Optimal Transport for Image Assimilation Introduction 5/38

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Heat diffusion Dynamical model: heat equation during 3 hours   ∂t u(x, t) = ∆u(x, t) x ∈ Ω\xh u(xH , t) = 20◦  ∂u (x) = g(x) x ∈ ∂Ω ∂⃗n Observations: Thermometer measure at θ: 17◦ ̸= u(xθ , 3) Uncertainties on boundary conditions g (ventilation...) Correction of boundary conditions g - Learning of g - Better knowledge of x - More reliable predictions Optimal Transport for Image Assimilation Introduction 5/38

Slide 13

Slide 13 text

Oceanography Variables: velocity, surface height, temperature, salinity, phytoplankton... Numerical model of ocean dynamic Non homogeneous spatial interactions Complex boundary conditions - Littoral - Seabed - Atmosphere Observations - in situ: Drifting buoys/ships... - Bathymetry - Satellite data Optimal Transport for Image Assimilation Introduction 6/38

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Satellite observations of ocean surface Information spatially structured (fronts, vortex...) Altimetry Temperature Chlorophyll Temporal deformations (occlusions) SST image Temporal variation Source: [Béréziat and Herlin ‘11] Optimal Transport for Image Assimilation Introduction 7/38

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Satellite observations Jason Resolution ≈ 10km SWOT High resolution ≈ 1km (CNES/NASA, 2021-) (CNES/NASA, 2001-) Optimal Transport for Image Assimilation Introduction 8/38

Slide 20

Slide 20 text

Satellite observations Jason Resolution ≈ 10km (CNES/NASA, 2001-) Optimal Transport for Image Assimilation SWOT High resolution ≈ 1km (CNES/NASA, 2021-) Introduction 8/38

Slide 21

Slide 21 text

Ocean images Optimal Transport for Image Assimilation Introduction 9/38

Slide 22

Slide 22 text

Prediction with data assimilation Constrained minimization problem Z ||y (t) − H(u(t))||2R dt + ||u(t0 ) − f ||2B min ∂t u=M(u) T Physical variable u: velocity, temperature, pressure... Dynamical model ∂t u = M(u): temporal evolution Background information at t0 : u(t0 ) ≈ f Observations y : radar, stations, balloons... Observation operator H Choice of a norm ||.|| Modeling of error covariance matrices on the background B and the observations R Optimal Transport for Image Assimilation Introduction 10/38

Slide 23

Slide 23 text

Assimilation of ocean surface images Satellite images y Challenges Choice of the norm to compare the relevant information contained in images Assimilation Loss function: ||y − H(u)||2R Optimal Transport for Image Assimilation Introduction 11/38

Slide 24

Slide 24 text

Assimilation of ocean surface images Satellite images y Challenges Observed quantities y not in the state u : Observation operator H? Choice of the norm to compare the relevant information contained in images Assimilation Loss function: ||y − H(u)||2R Optimal Transport for Image Assimilation Introduction 11/38

Slide 25

Slide 25 text

Assimilation of ocean surface images Satellite images y Challenges Observed quantities y not in the state u : Observation operator H? Error modeling R: KaRIN noise, satellite roll, atmospheric conditions... Assimilation Loss function: ||y − H(u)||2R Denoised image y [Gómez-Navarro, Cosme, Le Sommer et al. ‘18- ] Choice of the norm to compare the relevant information contained in images Optimal Transport for Image Assimilation Introduction 11/38

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

1 Introduction 2 Image Assimilation 3 Wasserstein distance 4 Wasserstein Image Assimilation 5 Conclusion Optimal Transport for Image Assimilation Image Assimilation 12/38

Slide 29

Slide 29 text

Image assimilation for oceanography Short review : Assimilation of structured information contained in images with standard Euclidean norm Main contributors: groups of R. Fablet, I. Herlin, F.-X. Le Dimet, É. Mémin, J. Verron... Optimal Transport for Image Assimilation Image Assimilation 13/38

Slide 30

Slide 30 text

Image assimilation in situ configuration - Image I: observation of a subset of the state variable u - Occlusion mask M, observation operator H - Pixelwise assimilation: ||M(I − Hu)||2 Example [Corpetti et al. ‘09] - Estimate the velocity fields of a multilayer shallow-water model - Assimilation of pressure images ✗ The structured information contained in images is not used Optimal Transport for Image Assimilation Image Assimilation 14/38

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Slide 33

Slide 33 text

How to assimilate the structures contained in images? Indirect assimilation: use image processing techniques to produce pseudo-observations Direct comparison of images and state variables in a dedicated space Optimal Transport for Image Assimilation Image Assimilation 15/38

Slide 34

Slide 34 text

Indirect assimilation 1 Extract information y from image observations I 2 Assimilation of pseudo-observations y : ||H(u) − y ||2 Example: fronts [Gaultier et al. ‘13] - I: SST or SSS image (a) SST I1 (b) SSS I2 (c) SPICE u - y : thresholded image gradients - H(u): Binarized Finite-Size Lyapunov Exponents (FLSE) (d) gradient SST y1 (e) gradient SSS y2 Optimal Transport for Image Assimilation (f) FLSE H(u) Image Assimilation 16/38

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Direct assimilation Define a coupling between image I and state variable u: ||H(u, I)||2 Optimal Transport for Image Assimilation Image Assimilation 17/38

Slide 37

Slide 37 text

Direct assimilation Define a coupling between image I and state variable u: ||H(u, I)||2 Temporal information [P. and Mémin ‘07, Béréziat and Herlin ‘09] - Layer velocity u(x, t), image sequence I(x, t) - Coupling with optical flow equation: ||I(x + u(x, t), t + 1) − I(x, t)||2 I(x, t) Optimal Transport for Image Assimilation I(x, t + 1) u(x, t) Image Assimilation 17/38

Slide 38

Slide 38 text

Slide 39

Slide 39 text

Direct assimilation Define a coupling between image I and state variable u: ||H(u, I)||2 Temporal information [P. and Mémin ‘07, Béréziat and Herlin ‘09] - Layer velocity u(x, t), image sequence I(x, t) - Coupling with optical flow equation: ||I(x + u(x, t), t + 1) − I(x, t)||2 I(x, t) I(x, t + 1) u(x, t) ✗ Not a norm on I or u : hard to model observation errors in a covariance matrix Optimal Transport for Image Assimilation Image Assimilation 17/38

Slide 40

Slide 40 text

Direct assimilation Define a coupling between image I and state variable u: ||H(u, I)||2 Multiscale decomposition [Titaud et al. ‘10] - Additional passive state variable u - Project u and I in a structured space S (Wavelets, Curvelets...): ||PS (u) − PS (I)||2 ✓ Threshold coefficients: dimension reduction, robustness to noise Optimal Transport for Image Assimilation Image Assimilation 17/38

Slide 41

Slide 41 text

Direct assimilation Define a coupling between image I and state variable u: ||H(u, I)||2 Multiscale decomposition [Titaud et al. ‘10] - Additional passive state variable u - Project u and I in a structured space S (Wavelets, Curvelets...): ||PS (u) − PS (I)||2 ✓ Threshold coefficients: dimension reduction, robustness to noise ✓ For observation noise correlated in space: diagonal error covariance matrix in wavelet space [Chabot et al. ‘15] Optimal Transport for Image Assimilation Image Assimilation 17/38

Slide 42

Slide 42 text

Norm for assimilation Type of compromise between prediction p and observation y : min ||u − p||2B + ||u − y ||2R u Standard assimilation: Euclidean distance weighted by a positive definite matrix A: ||x||2A = ⟨x, Ax⟩ For a confidence of degree α ∈ [0; 1], B = αId and R = (1 − α)Id: min α||u − p||2 + (1 − α)||u − y ||2 u Solution: linear interpolation u ∗ = αp + (1 − α)y - Prediction p = 18◦ , observation y = 17◦ - Assimilation is expected to produce something within [17◦ , 18◦ ] Optimal Transport for Image Assimilation Image Assimilation 18/38

Slide 43

Slide 43 text

Slide 44

Slide 44 text

Slide 45

Slide 45 text

Slide 46

Slide 46 text

Euclidean distance Data defined on a 1D domain Prediction Observation Euclidean Interpolation Interpolation → computation of geodesic in Euclidean space Not adapted to errors in the position of structures Optimal Transport for Image Assimilation Image Assimilation 19/38

Slide 47

Slide 47 text

Slide 48

Slide 48 text

Slide 49

Slide 49 text

Slide 50

Slide 50 text

Wasserstein distance Data defined on a 1D domain Prediction Observation Wasserstein Interpolation Interpolation → computation of geodesic in Wasserstein space Adapted to errors in the position of structures Optimal Transport for Image Assimilation Image Assimilation 19/38

Slide 51

Slide 51 text

Wasserstein in 2D Sea Surface Height creation of vortexes in Cap Point (output of NEMO model) How to deal with the coast? Optimal Transport for Image Assimilation Image Assimilation 20/38

Slide 52

Slide 52 text

Wasserstein in 2D Sea Surface Height creation of vortexes in Cap Point (output of NEMO model) How to deal with the coast? Optimal Transport for Image Assimilation Image Assimilation 20/38

Slide 53

Slide 53 text

1 Introduction 2 Image Assimilation 3 Wasserstein distance 4 Wasserstein Image Assimilation 5 Conclusion Optimal Transport for Image Assimilation Wasserstein distance 21/38

Slide 54

Slide 54 text

Optimal transport and Wasserstein distance Transport T : application moving a density ρ0 onto another one ρ1 (same mass), defined for x ∈ Ω as: ρ1 = T #ρ0 Optimal transport: Application of minimal cost R Ω c(x, T (x))dx Wasserstein distance Wp : c(x, y ) = ||x − y ||p Concave cost (Economy), Truncated cost (Computer Vision), W1 (Deep Learning) Optimal Transport for Image Assimilation Wasserstein distance 22/38

Slide 55

Slide 55 text

Slide 56

Slide 56 text

Slide 57

Slide 57 text

Applications in IP, CV, ML and DL Robust dissimilarity measure (Optimal transport cost) • Image retrieval (EMD) [Rubner et al. ’00] • 3D shape recognitions [Ruzon and Tomasi, ’01] • SIFT matching [Pele and Werman ’08] • Object segmentation [Ni et al. ’09, Rabin et al. ’11, ’15], • Denoising [Burger et al. ’12, Tartavel et al. ’16] • Generative models [Arjovsky et al. ’17, Genevay et al. ’18] Optimal Transport for Image Assimilation Wasserstein distance 23/38

Slide 58

Slide 58 text

Applications in IP, CV, ML and DL Robust dissimilarity measure (Optimal transport cost) • Image retrieval (EMD) [Rubner et al. ’00] • 3D shape recognitions [Ruzon and Tomasi, ’01] • SIFT matching [Pele and Werman ’08] • Object segmentation [Ni et al. ’09, Rabin et al. ’11, ’15], • Denoising [Burger et al. ’12, Tartavel et al. ’16] • Generative models [Arjovsky et al. ’17, Genevay et al. ’18] Why is it robust? Discrete bin-to-bin metrics are not informative for disjoint supports Optimal Transport for Image Assimilation Wasserstein distance 23/38

Slide 59

Slide 59 text

Applications in IP, CV, ML and DL Robust dissimilarity measure (Optimal transport cost) • Image retrieval (EMD) [Rubner et al. ’00] • 3D shape recognitions [Ruzon and Tomasi, ’01] • SIFT matching [Pele and Werman ’08] • Object segmentation [Ni et al. ’09, Rabin et al. ’11, ’15], • Denoising [Burger et al. ’12, Tartavel et al. ’16] • Generative models [Arjovsky et al. ’17, Genevay et al. ’18] Why is it robust? Discrete bin-to-bin metrics are not informative for disjoint supports Transport map T explains how far are the distributions Optimal Transport for Image Assimilation Wasserstein distance 23/38

Slide 60

Slide 60 text

Optimal Transport Map • The Monge transport map: • Interpolate between densities, compute barycenters or geodesics in the Wasserstein space ρ0 Optimal Transport for Image Assimilation ρ1 Wasserstein distance 24/38

Slide 61

Slide 61 text

Applications in IP, CV, ML and DL Tool for matching/interpolation (Optimal transport map) • Image interpolation, registration [Angenent et al. ’04] Medical image registration [Rehman et al. ’09] • Color transfer [Delon, ’04, Pitié et al. ’07, Bonneel et al. ‘11] • Shape matching [Rabin et al. ’10, Schmitzer and Schnörr ’14] • Texture synthesis [Ferradans et al. ’13, Leclaire et al. ’19] • Geodesic PCA [Bigot et al. ‘13, Seguy et al. ‘15, Cazelles et al. ‘18] • Domain adaptation [Courty et al. ’15, Redko et al. ’17] • Generative models [Seguy et al. ’18] Optimal Transport for Image Assimilation Wasserstein distance 25/38

Slide 62

Slide 62 text

Formulations Continuous Optimal Transport for Image Assimilation Semi-discrete Wasserstein distance Discrete 26/38

Slide 63

Slide 63 text

Formulations Continuous Semi-discrete Discrete [J.-D. Benamou] • Images: densities on support Ω • Mass transport in a fluid mechanics framework on Ω Velocity field T : Ω → Ω Optimal Transport for Image Assimilation Wasserstein distance 26/38

Slide 64

Slide 64 text

Formulations Continuous Semi-discrete Discrete [Q. Mérigot] Wasserstein for GAN Optimal Transport for Image Assimilation Wasserstein distance 26/38

Slide 65

Slide 65 text

Formulations Continuous Semi-discrete Discrete [Q. Mérigot] Wasserstein for GAN Optimal Transport for Image Assimilation Wasserstein distance 26/38

Slide 66

Slide 66 text

Formulations Continuous Semi-discrete Discrete [Q. Mérigot] Wasserstein for GAN Optimal Transport for Image Assimilation Wasserstein distance 26/38

Slide 67

Slide 67 text

Formulations Continuous Semi-discrete Discrete [M. Cuturi] • Distributions of image features • Transport between normalized histograms of size M and N Coupling matrix P of size M × N Optimal Transport for Image Assimilation Wasserstein distance 26/38

Slide 68

Slide 68 text

Formulations Continuous Optimal Transport for Image Assimilation Semi-discrete Wasserstein distance Discrete 26/38

Slide 69

Slide 69 text

Continuous Optimal Transport Densities ρ0 and ρ1 defined from x ∈ [0, 1]d to [0, 1] Mass preserving transport map T : T (ρ0 , ρ1 ) := {T : [0, 1]d 7→ [0, 1]d such that ρ1 = T ♯ρ0 } T ∈ T (ρ0 , ρ1 ) satisfies the gradient equation ρ0 (x) = ρ1 (T (x))| det(∂T (x))| An optimal transport T solves Z min T ∈T (ρ0 ,ρ1 ) C(x, T (x))ρ0 (x) dx where C(x, y ) ⩾ 0 is the cost of assigning x ∈ [0, 1]d to y ∈ [0, 1]d For C(x, y ) = ||x − y ||p : distance Wp , T unique if p > 1 Optimal Transport for Image Assimilation Wasserstein distance 27/38

Slide 70

Slide 70 text

Slide 71

Slide 71 text

Slide 72

Slide 72 text

Estimation of optimal transport map For p = 2,T = ∇ψ, with ψ convex [Brenier ’91] and optimal mass transfer follows straight lines → Monge-Ampère equation: det(D 2 ψ) = ρ0 (x) ρ1 (∇ψ(x)) [Oliker and Prussner ’88, Oberman ’08, Froese ’12, Benamou et al. ’16] ✓ Fast algorithms (second order methods) ✗ ρ1 should be lipschitz continuous with convex support ρ0 ρ1 ✓ Optimal Transport for Image Assimilation Wasserstein distance 28/38

Slide 73

Slide 73 text

Estimation of optimal transport map For p = 2,T = ∇ψ, with ψ convex [Brenier ’91] and optimal mass transfer follows straight lines → Monge-Ampère equation: det(D 2 ψ) = ρ0 (x) ρ1 (∇ψ(x)) [Oliker and Prussner ’88, Oberman ’08, Froese ’12, Benamou et al. ’16] ✓ Fast algorithms (second order methods) ✗ ρ1 should be lipschitz continuous with convex support ρ0 ρ1 ρ0 ρ1 ✓ ✗ Optimal Transport for Image Assimilation Wasserstein distance 28/38

Slide 74

Slide 74 text

Estimation of optimal transport map For p = 2,T = ∇ψ, with ψ convex [Brenier ’91] and optimal mass transfer follows straight lines → Monge-Ampère equation: det(D 2 ψ) = ρ0 (x) ρ1 (∇ψ(x)) [Oliker and Prussner ’88, Oberman ’08, Froese ’12, Benamou et al. ’16] ✓ Fast algorithms (second order methods) ✗ ρ1 should be lipschitz continuous with convex support ρ0 ρ0 ρ1 ✓ ρ1 ✗ • Regularized potential ψ [Paty et al. ’19] Optimal Transport for Image Assimilation Wasserstein distance 28/38

Slide 75

Slide 75 text

Estimation of optimal transport map For p = 2,T = ∇ψ, with ψ convex [Brenier ’91] and optimal mass transfer follows straight lines → Monge-Ampère equation: det(D 2 ψ) = ρ0 (x) ρ1 (∇ψ(x)) [Oliker and Prussner ’88, Oberman ’08, Froese ’12, Benamou et al. ’16] ✓ Fast algorithms (second order methods) ✗ ρ1 should be lipschitz continuous with convex support ρ0 ρ0 ρ1 ✓ ρ1 ✗ • Regularized potential ψ [Paty et al. ’19] All these methods are limited to non vanishing densities Optimal Transport for Image Assimilation Wasserstein distance 28/38

Slide 76

Slide 76 text

Fluid mechanics formulation [Benamou-Brenier ’00] • Parameterization with t ∈ [0, 1] of the geodesic path ρ(x, t): ρ(x, t) = ((1 − t)Id + tT (x)) ♯ρ0 Optimal Transport for Image Assimilation Wasserstein distance 29/38

Slide 77

Slide 77 text

Fluid mechanics formulation [Benamou-Brenier ’00] • Parameterization with t ∈ [0, 1] of the geodesic path ρ(x, t): ρ(x, t) = ((1 − t)Id + tT (x)) ♯ρ0 • Non-convex problem over ρ(x, t) ∈ R and velocity field v (x, t) ∈ R2 : Z Z 1 1 2 W2 (ρ0 , ρ1 ) = min ρ(x, t)||v (x, t)||2 dtdx, (v ,ρ)∈Cv 2 [0,1]2 0 under the set of non-linear constraints Cv = (v , ρ) \ ∂t ρ + divx (ρv ) = 0, v (0, ·) = v (1, ·) = 0, ρ(·, 0) = ρ0 , ρ(·, 1) = ρ1 Optimal Transport for Image Assimilation Wasserstein distance 29/38

Slide 78

Slide 78 text

Fluid mechanics formulation [Benamou-Brenier ’00] • Parameterization with t ∈ [0, 1] of the geodesic path ρ(x, t): ρ(x, t) = ((1 − t)Id + tT (x)) ♯ρ0 • Non-convex problem over ρ(x, t) ∈ R and velocity field v (x, t) ∈ R2 : Z Z 1 1 2 W2 (ρ0 , ρ1 ) = min ρ(x, t)||v (x, t)||2 dtdx, (v ,ρ)∈Cv 2 [0,1]2 0 under the set of non-linear constraints Cv = (v , ρ) \ ∂t ρ + divx (ρv ) = 0, v (0, ·) = v (1, ·) = 0, ρ(·, 0) = ρ0 , ρ(·, 1) = ρ1 ✓ Change of variable (v , ρ) 7→ (m, ρ), with m = ρv : Convex cost J and linear constraints C Optimal Transport for Image Assimilation Wasserstein distance 29/38

Slide 79

Slide 79 text

Fluid mechanics formulation [Benamou-Brenier ’00] • Parameterization with t ∈ [0, 1] of the geodesic path ρ(x, t): ρ(x, t) = ((1 − t)Id + tT (x)) ♯ρ0 • Non-convex problem over ρ(x, t) ∈ R and velocity field v (x, t) ∈ R2 : Z Z 1 1 2 W2 (ρ0 , ρ1 ) = min ρ(x, t)||v (x, t)||2 dtdx, (v ,ρ)∈Cv 2 [0,1]2 0 under the set of non-linear constraints Cv = (v , ρ) \ ∂t ρ + divx (ρv ) = 0, v (0, ·) = v (1, ·) = 0, ρ(·, 0) = ρ0 , ρ(·, 1) = ρ1 ✓ Change of variable (v , ρ) 7→ (m, ρ), with m = ρv : Convex cost J and linear constraints C ✗ No estimation of the transport map T , only the geodesic ρ(x, t) Optimal Transport for Image Assimilation Wasserstein distance 29/38

Slide 80

Slide 80 text

Wasserstein distance Pollutant on a convex 2D domain Prédiction Optimal Transport for Image Assimilation Observation Interpolation Wasserstein distance 30/38

Slide 81

Slide 81 text

Wasserstein distance Pollutant on a non-convex 2D domain Prédiction Observation Interpolation Obstacles with w : x → {1; +∞} [P., Peyré and Oudet, ‘14] Z w(x)c(x, T (x))dx Ω Optimal Transport for Image Assimilation Wasserstein distance 30/38

Slide 82

Slide 82 text

Wasserstein distance Optimal Transport for Image Assimilation Wasserstein distance 31/38

Slide 83

Slide 83 text

Wasserstein distance ✓ Deal with structured data in complex domain Optimal Transport for Image Assimilation Wasserstein distance 31/38

Slide 84

Slide 84 text

Wasserstein distance ✓ Deal with structured data in complex domain ✗ Computational cost Optimal Transport for Image Assimilation Wasserstein distance 31/38

Slide 85

Slide 85 text

1 Introduction 2 Image Assimilation 3 Wasserstein distance 4 Wasserstein Image Assimilation 5 Conclusion Optimal Transport for Image Assimilation Wasserstein Image Assimilation 32/38

Slide 86

Slide 86 text

Image assimilation with Wasserstein distance Quantity observed I included in the state [Feyeux et al. ‘18]: W2 (u, I) Clean formulation: gradient flow in Wasserstein space Initial time Optimal Transport for Image Assimilation Final time Wasserstein Image Assimilation 33/38

Slide 87

Slide 87 text

Image assimilation with Wasserstein distance Quantity observed I included in the state [Feyeux et al. ‘18]: W2 (u, I) Clean formulation: gradient flow in Wasserstein space Initial time Final time ✗ Only defined for densities: same mass and non negative Optimal Transport for Image Assimilation Wasserstein Image Assimilation 33/38

Slide 88

Slide 88 text

Contour assimilation with Wasserstein distance Deal with pollutant position shift [Li et al. ‘19] Level set representation [Ba et al. ‘10, Gautama et al. ‘16] of a contour C(u) of interest, transported by the model Pseudo-observations: extraction of a contour C(I) from the image Normalization of the mass: W2 Optimal Transport for Image Assimilation C(u) C(I) , |C(u)| |C(I)| Wasserstein Image Assimilation 34/38

Slide 89

Slide 89 text

Contour assimilation with Wasserstein distance Deal with pollutant position shift [Li et al. ‘19] Level set representation [Ba et al. ‘10, Gautama et al. ‘16] of a contour C(u) of interest, transported by the model Pseudo-observations: extraction of a contour C(I) from the image Normalization of the mass: W2 C(u) C(I) , |C(u)| |C(I)| C(I) Reduced non-linearities [P. and Rabin ‘17]: W2 C(u), |C(I)| |C(u)| Optimal Transport for Image Assimilation Wasserstein Image Assimilation 34/38

Slide 90

Slide 90 text

Contour assimilation with Wasserstein distance Deal with pollutant position shift [Li et al. ‘19] Level set representation [Ba et al. ‘10, Gautama et al. ‘16] of a contour C(u) of interest, transported by the model Pseudo-observations: extraction of a contour C(I) from the image Normalization of the mass: W2 C(u) C(I) , |C(u)| |C(I)| C(I) Reduced non-linearities [P. and Rabin ‘17]: W2 C(u), |C(I)| |C(u)| ✗ Limited to segmentation masks Optimal Transport for Image Assimilation Wasserstein Image Assimilation 34/38

Slide 91

Slide 91 text

Generalized Wasserstein models Extended W1 for seismograms of varying mass [Métivier et al. ‘16] Full waveform inversion Optimal Transport for Image Assimilation Wasserstein Image Assimilation 35/38

Slide 92

Slide 92 text

Generalized Wasserstein models Extended W1 for seismograms of varying mass [Métivier et al. ‘16] Full waveform inversion Unbalanced transport [Chizat et al. ‘18] - Deal with mass variations (source term in transport equation) - Still a distance Euclidean Optimal Transport for Image Assimilation Wasserstein Unbalanced Wasserstein Image Assimilation 35/38

Slide 93

Slide 93 text

Generalized Wasserstein models Extended W1 for seismograms of varying mass [Métivier et al. ‘16] Full waveform inversion Unbalanced transport [Chizat et al. ‘18] - Deal with mass variations (source term in transport equation) - Still a distance Euclidean Wasserstein Unbalanced Transport-based distance [Thorpe et al. ‘17] - Can deal with negative values (lifting) - Adapted to phase shifts Optimal Transport for Image Assimilation Wasserstein Image Assimilation 35/38

Slide 94

Slide 94 text

Slide 95

Slide 95 text

Slide 96

Slide 96 text

Slide 97

Slide 97 text

Computational schemes Dynamical Optimal Transport (geodesic estimation) Fixed grid [Benamou and Brenier ‘00, P. et al. ‘14, Osher et al. ‘17, Chizat et al ‘18, Hug et al. ‘20] Discrete surfaces [Lavenant et al. ’18] Parametric: sphere [Lang and P. ‘20?] - Multi-scale representation, adaptive meshes - Controlling errors on a reduced number of quadrature points Fast Optimal Transport map estimation Machine Learning: discrete setting [Cuturi & Peyré ‘19, Flamary, Courty et al. ‘19, P. ‘19, Weed & Rigolet ‘19...] Deep Learning: density estimation, mass transport [Dinh et al. ‘16, Zhang et al. ‘18, Grathwohl et al. ‘19...] Optimal Transport for Image Assimilation Wasserstein Image Assimilation 36/38

Slide 98

Slide 98 text

1 Introduction 2 Image Assimilation 3 Wasserstein distance 4 Wasserstein Image Assimilation 5 Conclusion Optimal Transport for Image Assimilation Conclusion 37/38

Slide 99

Slide 99 text

Conclusions and current challenges Wasserstein distance ✓ Measure between structured images ✓ Generalization: mass variation, negative ✗ Efficient computation ✗ Dealing with covariance matrices for assimilation Ocean images (SWOT mission, 2021) ✓ Observation of small scale dynamic ✗ Modeling observation errors Optimal Transport for Image Assimilation Conclusion 38/38