Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Optimal Transport for Image Assimilation

npapadakis
January 20, 2020

Optimal Transport for Image Assimilation

npapadakis

January 20, 2020
Tweet

More Decks by npapadakis

Other Decks in Technology

Transcript

  1. Optimal Transport for Image Assimilation Nicolas Papadakis Workshop on AI

    for Ocean, Atmosphere and Climate Dynamics January 20th 2020 Optimal Transport for Image Assimilation 1/38
  2. 1 Introduction 2 Image Assimilation 3 Wasserstein distance 4 Wasserstein

    Image Assimilation 5 Conclusion Optimal Transport for Image Assimilation Introduction 2/38
  3. How to use images for ocean state prediction? Optimal Transport

    for Image Assimilation Introduction 3/38
  4. Assimilation: how to predict the state of a dynamical system?

    Meteorology Glaciology Health sciences Optimal Transport for Image Assimilation Introduction 4/38
  5. Heat diffusion Physical variable: u(x, t) room temperature Ω at

    time t Initial conditions Homogeneous temperature u(., 0) = 15◦ Add an Heater at xH : u(xH, 0) = 20◦ Optimal Transport for Image Assimilation Introduction 5/38
  6. Heat diffusion Physical variable: u(x, t) room temperature Ω at

    time t Initial conditions Homogeneous temperature u(., 0) = 15◦ Add an Heater at xH : u(xH, 0) = 20◦ Optimal Transport for Image Assimilation Introduction 5/38
  7. Heat diffusion Dynamical model: heat equation during 3 hours 

      ∂t u(x, t) = ∆u(x, t) x ∈ Ω\xh u(xH, t) = 20◦ ∂u ∂n (x) = 0 x ∈ ∂Ω Optimal Transport for Image Assimilation Introduction 5/38
  8. Heat diffusion Dynamical model: heat equation during 3 hours 

      ∂t u(x, t) = ∆u(x, t) x ∈ Ω\xh u(xH, t) = 20◦ ∂u ∂n (x) = 0 x ∈ ∂Ω Optimal Transport for Image Assimilation Introduction 5/38
  9. Heat diffusion Dynamical model: heat equation during 3 hours 

      ∂t u(x, t) = ∆u(x, t) x ∈ Ω\xh u(xH, t) = 20◦ ∂u ∂n (x) = 0 x ∈ ∂Ω Observations: Thermometer measure at θ: 17◦ = u(xθ, 3) Optimal Transport for Image Assimilation Introduction 5/38
  10. Heat diffusion Dynamical model: heat equation during 3 hours 

      ∂t u(x, t) = ∆u(x, t) x ∈ Ω\xh u(xH, t) = 20◦ ∂u ∂n (x) = g(x) x ∈ ∂Ω Observations: Thermometer measure at θ: 17◦ = u(xθ, 3) Uncertainties on boundary conditions g (ventilation...) Optimal Transport for Image Assimilation Introduction 5/38
  11. Heat diffusion Dynamical model: heat equation during 3 hours 

      ∂t u(x, t) = ∆u(x, t) x ∈ Ω\xh u(xH, t) = 20◦ ∂u ∂n (x) = g(x) x ∈ ∂Ω Observations: Thermometer measure at θ: 17◦ = u(xθ, 3) Uncertainties on boundary conditions g (ventilation...) Optimal Transport for Image Assimilation Introduction 5/38
  12. Heat diffusion Dynamical model: heat equation during 3 hours 

      ∂t u(x, t) = ∆u(x, t) x ∈ Ω\xh u(xH, t) = 20◦ ∂u ∂n (x) = g(x) x ∈ ∂Ω Observations: Thermometer measure at θ: 17◦ = u(xθ, 3) Uncertainties on boundary conditions g (ventilation...) Correction of boundary conditions g - Learning of g - Better knowledge of x - More reliable predictions Optimal Transport for Image Assimilation Introduction 5/38
  13. Oceanography Variables: velocity, surface height, temperature, salinity, phytoplankton... Numerical model

    of ocean dynamic Non homogeneous spatial interactions Complex boundary conditions - Littoral - Seabed - Atmosphere Observations - in situ: Drifting buoys/ships... - Bathymetry - Satellite data Optimal Transport for Image Assimilation Introduction 6/38
  14. Oceanography Variables: velocity, surface height, temperature, salinity, phytoplankton... Numerical model

    of ocean dynamic Non homogeneous spatial interactions Complex boundary conditions - Littoral - Seabed - Atmosphere Observations - in situ: Drifting buoys/ships... - Bathymetry - Satellite data Optimal Transport for Image Assimilation Introduction 6/38
  15. Oceanography Variables: velocity, surface height, temperature, salinity, phytoplankton... Numerical model

    of ocean dynamic Non homogeneous spatial interactions Complex boundary conditions - Littoral - Seabed - Atmosphere Observations - in situ: Drifting buoys/ships... - Bathymetry - Satellite data Optimal Transport for Image Assimilation Introduction 6/38
  16. Oceanography Variables: velocity, surface height, temperature, salinity, phytoplankton... Numerical model

    of ocean dynamic Non homogeneous spatial interactions Complex boundary conditions - Littoral - Seabed - Atmosphere Observations - in situ: Drifting buoys/ships... - Bathymetry - Satellite data Optimal Transport for Image Assimilation Introduction 6/38
  17. Satellite observations of ocean surface Information spatially structured (fronts, vortex...)

    Altimetry Temperature Chlorophyll Temporal deformations (occlusions) SST image Temporal variation Source: [Béréziat and Herlin ‘11] Optimal Transport for Image Assimilation Introduction 7/38
  18. Satellite observations of ocean surface Information spatially structured (fronts, vortex...)

    Altimetry Temperature Chlorophyll Temporal deformations (occlusions) SST image Temporal variation Source: [Béréziat and Herlin ‘11] Optimal Transport for Image Assimilation Introduction 7/38
  19. Satellite observations Jason Resolution ≈ 10km (CNES/NASA, 2001-) SWOT High

    resolution ≈ 1km (CNES/NASA, 2021-) Optimal Transport for Image Assimilation Introduction 8/38
  20. Satellite observations Jason Resolution ≈ 10km (CNES/NASA, 2001-) SWOT High

    resolution ≈ 1km (CNES/NASA, 2021-) Optimal Transport for Image Assimilation Introduction 8/38
  21. Prediction with data assimilation Constrained minimization problem min ∂t u=M(u)

    T ||y(t) − H(u(t))||2 R dt + ||u(t0) − f||2 B Physical variable u: velocity, temperature, pressure... Dynamical model ∂t u = M(u): temporal evolution Background information at t0 : u(t0) ≈ f Observations y: radar, stations, balloons... Observation operator H Choice of a norm ||.|| Modeling of error covariance matrices on the background B and the observations R Optimal Transport for Image Assimilation Introduction 10/38
  22. Assimilation of ocean surface images Satellite images y Assimilation Loss

    function: ||y − H(u)||2 R Challenges Choice of the norm to compare the relevant information contained in images Optimal Transport for Image Assimilation Introduction 11/38
  23. Assimilation of ocean surface images Satellite images y Assimilation Loss

    function: ||y − H(u)||2 R Challenges Observed quantities y not in the state u : Observation operator H? Choice of the norm to compare the relevant information contained in images Optimal Transport for Image Assimilation Introduction 11/38
  24. Assimilation of ocean surface images Satellite images y Assimilation Loss

    function: ||y − H(u)||2 R Challenges Observed quantities y not in the state u : Observation operator H? Error modeling R: KaRIN noise, satellite roll, atmospheric conditions... Denoised image y [Gómez-Navarro, Cosme, Le Sommer et al. ‘18- ] Choice of the norm to compare the relevant information contained in images Optimal Transport for Image Assimilation Introduction 11/38
  25. Assimilation of ocean surface images Satellite images y Assimilation Loss

    function: ||y − H(u)||2 R Challenges Observed quantities y not in the state u : Observation operator H? Error modeling R: KaRIN noise, satellite roll, atmospheric conditions... Denoised image y [Gómez-Navarro, Cosme, Le Sommer et al. ‘18- ] Choice of the norm to compare the relevant information contained in images Optimal Transport for Image Assimilation Introduction 11/38
  26. Assimilation of ocean surface images Satellite images y Assimilation Loss

    function: ||y − H(u)||2 R Challenges Observed quantities y not in the state u : Observation operator H? Error modeling R: KaRIN noise, satellite roll, atmospheric conditions... Denoised image y [Gómez-Navarro, Cosme, Le Sommer et al. ‘18- ] Choice of the norm to compare the relevant information contained in images Optimal Transport for Image Assimilation Introduction 11/38
  27. 1 Introduction 2 Image Assimilation 3 Wasserstein distance 4 Wasserstein

    Image Assimilation 5 Conclusion Optimal Transport for Image Assimilation Image Assimilation 12/38
  28. Image assimilation for oceanography Short review : Assimilation of structured

    information contained in images with standard Euclidean norm Main contributors: groups of R. Fablet, I. Herlin, F.-X. Le Dimet, É. Mémin, J. Verron... Optimal Transport for Image Assimilation Image Assimilation 13/38
  29. Image assimilation in situ configuration - Image I: observation of

    a subset of the state variable u - Occlusion mask M, observation operator H - Pixelwise assimilation: ||M(I − Hu)||2 Example [Corpetti et al. ‘09] - Estimate the velocity fields of a multilayer shallow-water model - Assimilation of pressure images  The structured information contained in images is not used Optimal Transport for Image Assimilation Image Assimilation 14/38
  30. Image assimilation in situ configuration - Image I: observation of

    a subset of the state variable u - Occlusion mask M, observation operator H - Pixelwise assimilation: ||M(I − Hu)||2 Example [Corpetti et al. ‘09] - Estimate the velocity fields of a multilayer shallow-water model - Assimilation of pressure images  The structured information contained in images is not used Optimal Transport for Image Assimilation Image Assimilation 14/38
  31. Image assimilation in situ configuration - Image I: observation of

    a subset of the state variable u - Occlusion mask M, observation operator H - Pixelwise assimilation: ||M(I − Hu)||2 Example [Corpetti et al. ‘09] - Estimate the velocity fields of a multilayer shallow-water model - Assimilation of pressure images  The structured information contained in images is not used Optimal Transport for Image Assimilation Image Assimilation 14/38
  32. How to assimilate the structures contained in images? Indirect assimilation:

    use image processing techniques to produce pseudo-observations Direct comparison of images and state variables in a dedicated space Optimal Transport for Image Assimilation Image Assimilation 15/38
  33. Indirect assimilation 1 Extract information y from image observations I

    2 Assimilation of pseudo-observations y: ||H(u) − y||2 Example: fronts [Gaultier et al. ‘13] (a) SST I1 (b) SSS I2 (c) SPICE u (d) gradient SST y1 (e) gradient SSS y2 (f) FLSE H(u) - I: SST or SSS image - y: thresholded image gradients - H(u): Binarized Finite-Size Lyapunov Exponents (FLSE) Optimal Transport for Image Assimilation Image Assimilation 16/38
  34. Indirect assimilation 1 Extract information y from image observations I

    2 Assimilation of pseudo-observations y: ||H(u) − y||2 Example: fronts [Gaultier et al. ‘13] (a) SST I1 (b) SSS I2 (c) SPICE u (d) gradient SST y1 (e) gradient SSS y2 (f) FLSE H(u) - I: SST or SSS image - y: thresholded image gradients - H(u): Binarized Finite-Size Lyapunov Exponents (FLSE) Optimal Transport for Image Assimilation Image Assimilation 16/38
  35. Direct assimilation Define a coupling between image I and state

    variable u: ||H(u, I)||2 Optimal Transport for Image Assimilation Image Assimilation 17/38
  36. Direct assimilation Define a coupling between image I and state

    variable u: ||H(u, I)||2 Temporal information [P. and Mémin ‘07, Béréziat and Herlin ‘09] - Layer velocity u(x, t), image sequence I(x, t) - Coupling with optical flow equation: ||I(x + u(x, t), t + 1) − I(x, t)||2 I(x, t) I(x, t + 1) u(x, t) Optimal Transport for Image Assimilation Image Assimilation 17/38
  37. Direct assimilation Define a coupling between image I and state

    variable u: ||H(u, I)||2 Temporal information [P. and Mémin ‘07, Béréziat and Herlin ‘09] - Layer velocity u(x, t), image sequence I(x, t) - Coupling with optical flow equation: ||I(x + u(x, t), t + 1) − I(x, t)||2 I(x, t) I(x, t + 1) u(x, t) Optimal Transport for Image Assimilation Image Assimilation 17/38
  38. Direct assimilation Define a coupling between image I and state

    variable u: ||H(u, I)||2 Temporal information [P. and Mémin ‘07, Béréziat and Herlin ‘09] - Layer velocity u(x, t), image sequence I(x, t) - Coupling with optical flow equation: ||I(x + u(x, t), t + 1) − I(x, t)||2 I(x, t) I(x, t + 1) u(x, t)  Not a norm on I or u : hard to model observation errors in a covariance matrix Optimal Transport for Image Assimilation Image Assimilation 17/38
  39. Direct assimilation Define a coupling between image I and state

    variable u: ||H(u, I)||2 Multiscale decomposition [Titaud et al. ‘10] - Additional passive state variable u - Project u and I in a structured space S (Wavelets, Curvelets...): ||PS (u) − PS (I)||2  Threshold coefficients: dimension reduction, robustness to noise Optimal Transport for Image Assimilation Image Assimilation 17/38
  40. Direct assimilation Define a coupling between image I and state

    variable u: ||H(u, I)||2 Multiscale decomposition [Titaud et al. ‘10] - Additional passive state variable u - Project u and I in a structured space S (Wavelets, Curvelets...): ||PS (u) − PS (I)||2  Threshold coefficients: dimension reduction, robustness to noise  For observation noise correlated in space: diagonal error covariance matrix in wavelet space [Chabot et al. ‘15] Optimal Transport for Image Assimilation Image Assimilation 17/38
  41. Norm for assimilation Type of compromise between prediction p and

    observation y: min u ||u − p||2 B + ||u − y||2 R Standard assimilation: Euclidean distance weighted by a positive definite matrix A: ||x||2 A = x, Ax For a confidence of degree α ∈ [0; 1], B = αId and R = (1 − α)Id: min u α||u − p||2 + (1 − α)||u − y||2 Solution: linear interpolation u∗ = αp + (1 − α)y - Prediction p = 18◦, observation y = 17◦ - Assimilation is expected to produce something within [17◦, 18◦] Optimal Transport for Image Assimilation Image Assimilation 18/38
  42. Norm for assimilation Type of compromise between prediction p and

    observation y: min u ||u − p||2 B + ||u − y||2 R Standard assimilation: Euclidean distance weighted by a positive definite matrix A: ||x||2 A = x, Ax For a confidence of degree α ∈ [0; 1], B = αId and R = (1 − α)Id: min u α||u − p||2 + (1 − α)||u − y||2 Solution: linear interpolation u∗ = αp + (1 − α)y - Prediction p = 18◦, observation y = 17◦ - Assimilation is expected to produce something within [17◦, 18◦] Optimal Transport for Image Assimilation Image Assimilation 18/38
  43. Norm for assimilation Type of compromise between prediction p and

    observation y: min u ||u − p||2 B + ||u − y||2 R Standard assimilation: Euclidean distance weighted by a positive definite matrix A: ||x||2 A = x, Ax For a confidence of degree α ∈ [0; 1], B = αId and R = (1 − α)Id: min u α||u − p||2 + (1 − α)||u − y||2 Solution: linear interpolation u∗ = αp + (1 − α)y - Prediction p = 18◦, observation y = 17◦ - Assimilation is expected to produce something within [17◦, 18◦] Optimal Transport for Image Assimilation Image Assimilation 18/38
  44. Norm for assimilation Type of compromise between prediction p and

    observation y: min u ||u − p||2 B + ||u − y||2 R Standard assimilation: Euclidean distance weighted by a positive definite matrix A: ||x||2 A = x, Ax For a confidence of degree α ∈ [0; 1], B = αId and R = (1 − α)Id: min u α||u − p||2 + (1 − α)||u − y||2 Solution: linear interpolation u∗ = αp + (1 − α)y - Prediction p = 18◦, observation y = 17◦ - Assimilation is expected to produce something within [17◦, 18◦] Optimal Transport for Image Assimilation Image Assimilation 18/38
  45. Euclidean distance Data defined on a 1D domain Prediction Observation

    Euclidean Interpolation Interpolation → computation of geodesic in Euclidean space Not adapted to errors in the position of structures Optimal Transport for Image Assimilation Image Assimilation 19/38
  46. Euclidean distance Data defined on a 1D domain Prediction Observation

    Euclidean Interpolation Interpolation → computation of geodesic in Euclidean space Not adapted to errors in the position of structures Optimal Transport for Image Assimilation Image Assimilation 19/38
  47. Euclidean distance Data defined on a 1D domain Prediction Observation

    Euclidean Interpolation Interpolation → computation of geodesic in Euclidean space Not adapted to errors in the position of structures Optimal Transport for Image Assimilation Image Assimilation 19/38
  48. Euclidean distance Data defined on a 1D domain Prediction Observation

    Euclidean Interpolation Interpolation → computation of geodesic in Euclidean space Not adapted to errors in the position of structures Optimal Transport for Image Assimilation Image Assimilation 19/38
  49. Wasserstein distance Data defined on a 1D domain Prediction Observation

    Wasserstein Interpolation Interpolation → computation of geodesic in Wasserstein space Adapted to errors in the position of structures Optimal Transport for Image Assimilation Image Assimilation 19/38
  50. Wasserstein in 2D Sea Surface Height creation of vortexes in

    Cap Point (output of NEMO model) How to deal with the coast? Optimal Transport for Image Assimilation Image Assimilation 20/38
  51. Wasserstein in 2D Sea Surface Height creation of vortexes in

    Cap Point (output of NEMO model) How to deal with the coast? Optimal Transport for Image Assimilation Image Assimilation 20/38
  52. 1 Introduction 2 Image Assimilation 3 Wasserstein distance 4 Wasserstein

    Image Assimilation 5 Conclusion Optimal Transport for Image Assimilation Wasserstein distance 21/38
  53. Optimal transport and Wasserstein distance Transport T: application moving a

    density ρ0 onto another one ρ1 (same mass), defined for x ∈ Ω as: ρ1 = T#ρ0 Optimal transport: Application of minimal cost Ω c(x, T(x))dx Wasserstein distance Wp : c(x, y) = ||x − y||p Concave cost (Economy), Truncated cost (Computer Vision), W1 (Deep Learning) Optimal Transport for Image Assimilation Wasserstein distance 22/38
  54. Optimal transport and Wasserstein distance Transport T: application moving a

    density ρ0 onto another one ρ1 (same mass), defined for x ∈ Ω as: ρ1 = T#ρ0 Optimal transport: Application of minimal cost Ω c(x, T(x))dx Wasserstein distance Wp : c(x, y) = ||x − y||p Concave cost (Economy), Truncated cost (Computer Vision), W1 (Deep Learning) Optimal Transport for Image Assimilation Wasserstein distance 22/38
  55. Optimal transport and Wasserstein distance Transport T: application moving a

    density ρ0 onto another one ρ1 (same mass), defined for x ∈ Ω as: ρ1 = T#ρ0 Optimal transport: Application of minimal cost Ω c(x, T(x))dx Wasserstein distance Wp : c(x, y) = ||x − y||p Concave cost (Economy), Truncated cost (Computer Vision), W1 (Deep Learning) Optimal Transport for Image Assimilation Wasserstein distance 22/38
  56. Applications in IP, CV, ML and DL Robust dissimilarity measure

    (Optimal transport cost) • Image retrieval (EMD) [Rubner et al. ’00] • 3D shape recognitions [Ruzon and Tomasi, ’01] • SIFT matching [Pele and Werman ’08] • Object segmentation [Ni et al. ’09, Rabin et al. ’11, ’15], • Denoising [Burger et al. ’12, Tartavel et al. ’16] • Generative models [Arjovsky et al. ’17, Genevay et al. ’18] Optimal Transport for Image Assimilation Wasserstein distance 23/38
  57. Applications in IP, CV, ML and DL Robust dissimilarity measure

    (Optimal transport cost) • Image retrieval (EMD) [Rubner et al. ’00] • 3D shape recognitions [Ruzon and Tomasi, ’01] • SIFT matching [Pele and Werman ’08] • Object segmentation [Ni et al. ’09, Rabin et al. ’11, ’15], • Denoising [Burger et al. ’12, Tartavel et al. ’16] • Generative models [Arjovsky et al. ’17, Genevay et al. ’18] Why is it robust? Discrete bin-to-bin metrics are not informative for disjoint supports Optimal Transport for Image Assimilation Wasserstein distance 23/38
  58. Applications in IP, CV, ML and DL Robust dissimilarity measure

    (Optimal transport cost) • Image retrieval (EMD) [Rubner et al. ’00] • 3D shape recognitions [Ruzon and Tomasi, ’01] • SIFT matching [Pele and Werman ’08] • Object segmentation [Ni et al. ’09, Rabin et al. ’11, ’15], • Denoising [Burger et al. ’12, Tartavel et al. ’16] • Generative models [Arjovsky et al. ’17, Genevay et al. ’18] Why is it robust? Discrete bin-to-bin metrics are not informative for disjoint supports Transport map T explains how far are the distributions Optimal Transport for Image Assimilation Wasserstein distance 23/38
  59. Optimal Transport Map • The Monge transport map: • Interpolate

    between densities, compute barycenters or geodesics in the Wasserstein space ρ0 ρ1 Optimal Transport for Image Assimilation Wasserstein distance 24/38
  60. Applications in IP, CV, ML and DL Tool for matching/interpolation

    (Optimal transport map) • Image interpolation, registration [Angenent et al. ’04] Medical image registration [Rehman et al. ’09] • Color transfer [Delon, ’04, Pitié et al. ’07, Bonneel et al. ‘11] • Shape matching [Rabin et al. ’10, Schmitzer and Schnörr ’14] • Texture synthesis [Ferradans et al. ’13, Leclaire et al. ’19] • Geodesic PCA [Bigot et al. ‘13, Seguy et al. ‘15, Cazelles et al. ‘18] • Domain adaptation [Courty et al. ’15, Redko et al. ’17] • Generative models [Seguy et al. ’18] Optimal Transport for Image Assimilation Wasserstein distance 25/38
  61. Formulations Continuous [J.-D. Benamou] Semi-discrete Discrete • Images: densities on

    support Ω • Mass transport in a fluid mechanics framework on Ω Velocity field T : Ω → Ω Optimal Transport for Image Assimilation Wasserstein distance 26/38
  62. Formulations Continuous Semi-discrete [Q. Mérigot] Discrete Wasserstein for GAN Optimal

    Transport for Image Assimilation Wasserstein distance 26/38
  63. Formulations Continuous Semi-discrete [Q. Mérigot] Discrete Wasserstein for GAN Optimal

    Transport for Image Assimilation Wasserstein distance 26/38
  64. Formulations Continuous Semi-discrete [Q. Mérigot] Discrete Wasserstein for GAN Optimal

    Transport for Image Assimilation Wasserstein distance 26/38
  65. Formulations Continuous Semi-discrete Discrete [M. Cuturi] • Distributions of image

    features • Transport between normalized histograms of size M and N Coupling matrix P of size M × N Optimal Transport for Image Assimilation Wasserstein distance 26/38
  66. Continuous Optimal Transport Densities ρ0 and ρ1 defined from x

    ∈ [0, 1]d to [0, 1] Mass preserving transport map T: T (ρ0, ρ1) := {T : [0, 1]d → [0, 1]d such that ρ1 = T ρ0} T ∈ T (ρ0, ρ1) satisfies the gradient equation ρ0(x) = ρ1(T(x))| det(∂T(x))| An optimal transport T solves min T∈T (ρ0,ρ1) C(x, T(x))ρ0(x) dx where C(x, y) 0 is the cost of assigning x ∈ [0, 1]d to y ∈ [0, 1]d For C(x, y) = ||x − y||p: distance Wp, T unique if p > 1 Optimal Transport for Image Assimilation Wasserstein distance 27/38
  67. Estimation of optimal transport map For p = 2,T =

    ∇ψ, with ψ convex [Brenier ’91] and optimal mass transfer follows straight lines → Monge-Ampère equation: det(D2ψ) = ρ0(x) ρ1(∇ψ(x)) [Oliker and Prussner ’88, Oberman ’08, Froese ’12, Benamou et al. ’16] Optimal Transport for Image Assimilation Wasserstein distance 28/38
  68. Estimation of optimal transport map For p = 2,T =

    ∇ψ, with ψ convex [Brenier ’91] and optimal mass transfer follows straight lines → Monge-Ampère equation: det(D2ψ) = ρ0(x) ρ1(∇ψ(x)) [Oliker and Prussner ’88, Oberman ’08, Froese ’12, Benamou et al. ’16]  Fast algorithms (second order methods) Optimal Transport for Image Assimilation Wasserstein distance 28/38
  69. Estimation of optimal transport map For p = 2,T =

    ∇ψ, with ψ convex [Brenier ’91] and optimal mass transfer follows straight lines → Monge-Ampère equation: det(D2ψ) = ρ0(x) ρ1(∇ψ(x)) [Oliker and Prussner ’88, Oberman ’08, Froese ’12, Benamou et al. ’16]  Fast algorithms (second order methods)  ρ1 should be lipschitz continuous with convex support ρ0 ρ1  Optimal Transport for Image Assimilation Wasserstein distance 28/38
  70. Estimation of optimal transport map For p = 2,T =

    ∇ψ, with ψ convex [Brenier ’91] and optimal mass transfer follows straight lines → Monge-Ampère equation: det(D2ψ) = ρ0(x) ρ1(∇ψ(x)) [Oliker and Prussner ’88, Oberman ’08, Froese ’12, Benamou et al. ’16]  Fast algorithms (second order methods)  ρ1 should be lipschitz continuous with convex support ρ0 ρ1 ρ0 ρ1   Optimal Transport for Image Assimilation Wasserstein distance 28/38
  71. Estimation of optimal transport map For p = 2,T =

    ∇ψ, with ψ convex [Brenier ’91] and optimal mass transfer follows straight lines → Monge-Ampère equation: det(D2ψ) = ρ0(x) ρ1(∇ψ(x)) [Oliker and Prussner ’88, Oberman ’08, Froese ’12, Benamou et al. ’16]  Fast algorithms (second order methods)  ρ1 should be lipschitz continuous with convex support ρ0 ρ1 ρ0 ρ1   • Regularized potential ψ [Paty et al. ’19] Optimal Transport for Image Assimilation Wasserstein distance 28/38
  72. Estimation of optimal transport map For p = 2,T =

    ∇ψ, with ψ convex [Brenier ’91] and optimal mass transfer follows straight lines → Monge-Ampère equation: det(D2ψ) = ρ0(x) ρ1(∇ψ(x)) [Oliker and Prussner ’88, Oberman ’08, Froese ’12, Benamou et al. ’16]  Fast algorithms (second order methods)  ρ1 should be lipschitz continuous with convex support ρ0 ρ1 ρ0 ρ1   • Regularized potential ψ [Paty et al. ’19] All these methods are limited to non vanishing densities Optimal Transport for Image Assimilation Wasserstein distance 28/38
  73. Fluid mechanics formulation [Benamou-Brenier ’00] • Parameterization with t ∈

    [0, 1] of the geodesic path ρ(x, t): ρ(x, t) = ((1 − t)Id + tT(x)) ρ0 Optimal Transport for Image Assimilation Wasserstein distance 29/38
  74. Fluid mechanics formulation [Benamou-Brenier ’00] • Parameterization with t ∈

    [0, 1] of the geodesic path ρ(x, t): ρ(x, t) = ((1 − t)Id + tT(x)) ρ0 • Non-convex problem over ρ(x, t) ∈ R and velocity field v(x, t) ∈ R2: W2(ρ0, ρ1)2 = min (v,ρ)∈Cv 1 2 [0,1]2 1 0 ρ(x, t)||v(x, t)||2dtdx, under the set of non-linear constraints Cv = (v, ρ) \ ∂t ρ + divx (ρv) = 0, v(0, ·) = v(1, ·) = 0, ρ(·, 0) = ρ0, ρ(·, 1) = ρ1 Optimal Transport for Image Assimilation Wasserstein distance 29/38
  75. Fluid mechanics formulation [Benamou-Brenier ’00] • Parameterization with t ∈

    [0, 1] of the geodesic path ρ(x, t): ρ(x, t) = ((1 − t)Id + tT(x)) ρ0 • Non-convex problem over ρ(x, t) ∈ R and velocity field v(x, t) ∈ R2: W2(ρ0, ρ1)2 = min (v,ρ)∈Cv 1 2 [0,1]2 1 0 ρ(x, t)||v(x, t)||2dtdx, under the set of non-linear constraints Cv = (v, ρ) \ ∂t ρ + divx (ρv) = 0, v(0, ·) = v(1, ·) = 0, ρ(·, 0) = ρ0, ρ(·, 1) = ρ1  Change of variable (v, ρ) → (m, ρ), with m = ρv: Convex cost J and linear constraints C Optimal Transport for Image Assimilation Wasserstein distance 29/38
  76. Fluid mechanics formulation [Benamou-Brenier ’00] • Parameterization with t ∈

    [0, 1] of the geodesic path ρ(x, t): ρ(x, t) = ((1 − t)Id + tT(x)) ρ0 • Non-convex problem over ρ(x, t) ∈ R and velocity field v(x, t) ∈ R2: W2(ρ0, ρ1)2 = min (v,ρ)∈Cv 1 2 [0,1]2 1 0 ρ(x, t)||v(x, t)||2dtdx, under the set of non-linear constraints Cv = (v, ρ) \ ∂t ρ + divx (ρv) = 0, v(0, ·) = v(1, ·) = 0, ρ(·, 0) = ρ0, ρ(·, 1) = ρ1  Change of variable (v, ρ) → (m, ρ), with m = ρv: Convex cost J and linear constraints C  No estimation of the transport map T, only the geodesic ρ(x, t) Optimal Transport for Image Assimilation Wasserstein distance 29/38
  77. Wasserstein distance Pollutant on a convex 2D domain Prédiction Observation

    Interpolation Optimal Transport for Image Assimilation Wasserstein distance 30/38
  78. Wasserstein distance Pollutant on a non-convex 2D domain Prédiction Observation

    Interpolation Obstacles with w : x → {1; +∞} [P., Peyré and Oudet, ‘14] Ω w(x)c(x, T(x))dx Optimal Transport for Image Assimilation Wasserstein distance 30/38
  79. Wasserstein distance  Deal with structured data in complex domain

    Optimal Transport for Image Assimilation Wasserstein distance 31/38
  80. Wasserstein distance  Deal with structured data in complex domain

     Computational cost Optimal Transport for Image Assimilation Wasserstein distance 31/38
  81. 1 Introduction 2 Image Assimilation 3 Wasserstein distance 4 Wasserstein

    Image Assimilation 5 Conclusion Optimal Transport for Image Assimilation Wasserstein Image Assimilation 32/38
  82. Image assimilation with Wasserstein distance Quantity observed I included in

    the state [Feyeux et al. ‘18]: W2(u, I) Clean formulation: gradient flow in Wasserstein space Initial time Final time Optimal Transport for Image Assimilation Wasserstein Image Assimilation 33/38
  83. Image assimilation with Wasserstein distance Quantity observed I included in

    the state [Feyeux et al. ‘18]: W2(u, I) Clean formulation: gradient flow in Wasserstein space Initial time Final time  Only defined for densities: same mass and non negative Optimal Transport for Image Assimilation Wasserstein Image Assimilation 33/38
  84. Contour assimilation with Wasserstein distance Deal with pollutant position shift

    [Li et al. ‘19] Level set representation [Ba et al. ‘10, Gautama et al. ‘16] of a contour C(u) of interest, transported by the model Pseudo-observations: extraction of a contour C(I) from the image Normalization of the mass: W2 C(u) |C(u)| , C(I) |C(I)| Optimal Transport for Image Assimilation Wasserstein Image Assimilation 34/38
  85. Contour assimilation with Wasserstein distance Deal with pollutant position shift

    [Li et al. ‘19] Level set representation [Ba et al. ‘10, Gautama et al. ‘16] of a contour C(u) of interest, transported by the model Pseudo-observations: extraction of a contour C(I) from the image Normalization of the mass: W2 C(u) |C(u)| , C(I) |C(I)| Reduced non-linearities [P. and Rabin ‘17]: W2 C(u), C(I) |C(I)| |C(u)| Optimal Transport for Image Assimilation Wasserstein Image Assimilation 34/38
  86. Contour assimilation with Wasserstein distance Deal with pollutant position shift

    [Li et al. ‘19] Level set representation [Ba et al. ‘10, Gautama et al. ‘16] of a contour C(u) of interest, transported by the model Pseudo-observations: extraction of a contour C(I) from the image Normalization of the mass: W2 C(u) |C(u)| , C(I) |C(I)| Reduced non-linearities [P. and Rabin ‘17]: W2 C(u), C(I) |C(I)| |C(u)|  Limited to segmentation masks Optimal Transport for Image Assimilation Wasserstein Image Assimilation 34/38
  87. Generalized Wasserstein models Extended W1 for seismograms of varying mass

    [Métivier et al. ‘16] Full waveform inversion Optimal Transport for Image Assimilation Wasserstein Image Assimilation 35/38
  88. Generalized Wasserstein models Extended W1 for seismograms of varying mass

    [Métivier et al. ‘16] Full waveform inversion Unbalanced transport [Chizat et al. ‘18] - Deal with mass variations (source term in transport equation) - Still a distance Euclidean Wasserstein Unbalanced Optimal Transport for Image Assimilation Wasserstein Image Assimilation 35/38
  89. Generalized Wasserstein models Extended W1 for seismograms of varying mass

    [Métivier et al. ‘16] Full waveform inversion Unbalanced transport [Chizat et al. ‘18] - Deal with mass variations (source term in transport equation) - Still a distance Euclidean Wasserstein Unbalanced Transport-based distance [Thorpe et al. ‘17] - Can deal with negative values (lifting) - Adapted to phase shifts Optimal Transport for Image Assimilation Wasserstein Image Assimilation 35/38
  90. Computational schemes Dynamical Optimal Transport (geodesic estimation) Fixed grid [Benamou

    and Brenier ‘00, P. et al. ‘14, Osher et al. ‘17, Chizat et al ‘18, Hug et al. ‘20] Discrete surfaces [Lavenant et al. ’18] Optimal Transport for Image Assimilation Wasserstein Image Assimilation 36/38
  91. Computational schemes Dynamical Optimal Transport (geodesic estimation) Fixed grid [Benamou

    and Brenier ‘00, P. et al. ‘14, Osher et al. ‘17, Chizat et al ‘18, Hug et al. ‘20] Discrete surfaces [Lavenant et al. ’18] Parametric: sphere [Lang and P. ‘20?] Optimal Transport for Image Assimilation Wasserstein Image Assimilation 36/38
  92. Computational schemes Dynamical Optimal Transport (geodesic estimation) Fixed grid [Benamou

    and Brenier ‘00, P. et al. ‘14, Osher et al. ‘17, Chizat et al ‘18, Hug et al. ‘20] Discrete surfaces [Lavenant et al. ’18] Parametric: sphere [Lang and P. ‘20?] - Multi-scale representation, adaptive meshes - Controlling errors on a reduced number of quadrature points Optimal Transport for Image Assimilation Wasserstein Image Assimilation 36/38
  93. Computational schemes Dynamical Optimal Transport (geodesic estimation) Fixed grid [Benamou

    and Brenier ‘00, P. et al. ‘14, Osher et al. ‘17, Chizat et al ‘18, Hug et al. ‘20] Discrete surfaces [Lavenant et al. ’18] Parametric: sphere [Lang and P. ‘20?] - Multi-scale representation, adaptive meshes - Controlling errors on a reduced number of quadrature points Fast Optimal Transport map estimation Machine Learning: discrete setting [Cuturi & Peyré ‘19, Flamary, Courty et al. ‘19, P. ‘19, Weed & Rigolet ‘19...] Deep Learning: density estimation, mass transport [Dinh et al. ‘16, Zhang et al. ‘18, Grathwohl et al. ‘19...] Optimal Transport for Image Assimilation Wasserstein Image Assimilation 36/38
  94. 1 Introduction 2 Image Assimilation 3 Wasserstein distance 4 Wasserstein

    Image Assimilation 5 Conclusion Optimal Transport for Image Assimilation Conclusion 37/38
  95. Conclusions and current challenges Wasserstein distance  Measure between structured

    images  Generalization: mass variation, negative  Efficient computation  Dealing with covariance matrices for assimilation Ocean images (SWOT mission, 2021)  Observation of small scale dynamic  Modeling observation errors Optimal Transport for Image Assimilation Conclusion 38/38