of distances between densities of probability • Transport a mass ρ0 onto ρ1: • Define a cost C(x, y) of mass transport between locations x and y • OT: application with mimimal global cost that transfers ρ0 onto ρ1 • If C(x, y) = ||x − y||p, Lp Wasserstein distance • Concave cost (Economy), Truncated cost (Computer Vision) 1 / 50
transport cost) • Image retrieval (EMD) [Rubner et al. ’00] • 3D shape recognitions [Ruzon and Tomasi, ’01] • SIFT matching [Pele and Werman ’08] • Object segmentation [Ni et al. ’09, Rabin et al. ’11, ’15], • Denoising [Burger et al. ’12, Tartavel et al. ’16] • Loss function [Frogner et al. ’15, Genevay et al. ’17] • Generative models [Arjovsky et al. ’17] 2 / 50
transport cost) • Image retrieval (EMD) [Rubner et al. ’00] • 3D shape recognitions [Ruzon and Tomasi, ’01] • SIFT matching [Pele and Werman ’08] • Object segmentation [Ni et al. ’09, Rabin et al. ’11, ’15], • Denoising [Burger et al. ’12, Tartavel et al. ’16] • Loss function [Frogner et al. ’15, Genevay et al. ’17] • Generative models [Arjovsky et al. ’17] Why is it robust? Discrete bin-to-bin metrics are not informative for disjoint supports 2 / 50
transport cost) • Image retrieval (EMD) [Rubner et al. ’00] • 3D shape recognitions [Ruzon and Tomasi, ’01] • SIFT matching [Pele and Werman ’08] • Object segmentation [Ni et al. ’09, Rabin et al. ’11, ’15], • Denoising [Burger et al. ’12, Tartavel et al. ’16] • Loss function [Frogner et al. ’15, Genevay et al. ’17] • Generative models [Arjovsky et al. ’17] Why is it robust? Discrete bin-to-bin metrics are not informative for disjoint supports Transport map T explains how far are the distributions 2 / 50
transport map) • Image interpolation, registration [Angenent et al. ’04] Medical image registration [Rehman et al. ’09] • Color transfer [Delon, ’04, Pitié et al. ’07, Bonneel et al. ‘11] • Shape matching [Rabin et al. ’10, Schmitzer and Schnörr ’14] • Texture synthesis [Xia et al. ’13, Galerne et al. ’18, Leclaire et al. ’19] • Geodesic PCA [Bigot et al. ‘13, Seguy et al. ‘15, Cazelles et al. ‘18] • Domain adaptation [Courty et al. ’15, Redko et al. ’17] • Generative models [Seguy et al. ’18] 4 / 50
transport map) • Image interpolation, registration [Angenent et al. ’04] Medical image registration [Rehman et al. ’09] • Color transfer [Delon, ’04, Pitié et al. ’07, Bonneel et al. ‘11] • Shape matching [Rabin et al. ’10, Schmitzer and Schnörr ’14] • Texture synthesis [Xia et al. ’13, Galerne et al. ’18, Leclaire et al. ’19] • Geodesic PCA [Bigot et al. ‘13, Seguy et al. ‘15, Cazelles et al. ‘18] • Domain adaptation [Courty et al. ’15, Redko et al. ’17] • Generative models [Seguy et al. ’18] Today: Use of the transport map for Image Processing applications 4 / 50
on support Ω • Mass transport in a fluid mechanics framework on Ω Velocity field T : Ω → Ω • Distributions of image features • Transport between normalized histograms of size N and M Coupling matrix P of size M × N 5 / 50
• Generalization of the transport cost • Non-convex model with physical priors ⇒ Application to data interpolation Part II - Discrete formulation • Relaxation and regularization of static transport matrix • Non-convex model to cancel mass spreading ⇒ Application to color transfer 6 / 50
• Collaborations with oceanographers: Sea Surface Height: creation of vortexes in Cap Point (output of model NEMO) • Objective: Image interpolation • Problem: How to deal with the coast?p 8 / 50
• Collaborations with oceanographers: Sea Surface Height: creation of vortexes in Cap Point (output of model NEMO) • Objective: Image interpolation • Problem: How to deal with the coast?pOptical flow 8 / 50
• Collaborations with oceanographers: Sea Surface Height: creation of vortexes in Cap Point (output of model NEMO) • Objective: Image interpolation • Problem: How to deal with the coast?p (((((( hhhhhh Optical flow 8 / 50
• Collaborations with oceanographers: Sea Surface Height: creation of vortexes in Cap Point (output of model NEMO) • Objective: Image interpolation • Problem: How to deal with the coast?p (((((( hhhhhh Optical flow Discrete OT 8 / 50
• Collaborations with oceanographers: Sea Surface Height: creation of vortexes in Cap Point (output of model NEMO) • Objective: Image interpolation • Problem: How to deal with the coast?p (((((( hhhhhh Optical flow (((((( hhhhhh Discrete OT 8 / 50
x ∈ [0, 1]d to [0, 1] • Mass preserving transport map T: T (ρ0, ρ1) := {T : [0, 1]d → [0, 1]d such that ρ1 = T ρ0} • An optimal transport T solves min T∈T (ρ0,ρ1) C(x, T(x))ρ0(x) dx where C(x, y) 0 is the cost of assigning x ∈ [0, 1]d to y ∈ [0, 1]d 10 / 50
⇒ p−Wasserstein distance between ρ0 and ρ1 • For p > 1, T is unique • For p = 2,T = ∇ψ, with ψ convex [Brenier ’91] and optimal mass transfer follows straight lines 11 / 50
⇒ p−Wasserstein distance between ρ0 and ρ1 • For p > 1, T is unique • For p = 2,T = ∇ψ, with ψ convex [Brenier ’91] and optimal mass transfer follows straight lines Explicit computation in 1D with cumulative functions 11 / 50
⇒ p−Wasserstein distance between ρ0 and ρ1 • For p > 1, T is unique • For p = 2,T = ∇ψ, with ψ convex [Brenier ’91] and optimal mass transfer follows straight lines Explicit computation in 1D with cumulative functions 11 / 50
⇒ p−Wasserstein distance between ρ0 and ρ1 • For p > 1, T is unique • For p = 2,T = ∇ψ, with ψ convex [Brenier ’91] and optimal mass transfer follows straight lines Explicit computation in 1D with cumulative functions 11 / 50
⇒ p−Wasserstein distance between ρ0 and ρ1 • For p > 1, T is unique • For p = 2,T = ∇ψ, with ψ convex [Brenier ’91] and optimal mass transfer follows straight lines Explicit computation in 1D with cumulative functions 11 / 50
⇒ p−Wasserstein distance between ρ0 and ρ1 • For p > 1, T is unique • For p = 2,T = ∇ψ, with ψ convex [Brenier ’91] and optimal mass transfer follows straight lines Explicit computation in 1D with cumulative functions 11 / 50
⇒ p−Wasserstein distance between ρ0 and ρ1 • For p > 1, T is unique • For p = 2,T = ∇ψ, with ψ convex [Brenier ’91] and optimal mass transfer follows straight lines Explicit computation in 1D with cumulative functions 11 / 50
⇒ p−Wasserstein distance between ρ0 and ρ1 • For p > 1, T is unique • For p = 2,T = ∇ψ, with ψ convex [Brenier ’91] and optimal mass transfer follows straight lines Explicit computation in 1D with cumulative functions 11 / 50
⇒ p−Wasserstein distance between ρ0 and ρ1 • For p > 1, T is unique • For p = 2,T = ∇ψ, with ψ convex [Brenier ’91] and optimal mass transfer follows straight lines Explicit computation in 1D with cumulative functions 11 / 50
⇒ p−Wasserstein distance between ρ0 and ρ1 • For p > 1, T is unique • For p = 2,T = ∇ψ, with ψ convex [Brenier ’91] and optimal mass transfer follows straight lines Explicit computation in 1D with cumulative functions 11 / 50
⇒ p−Wasserstein distance between ρ0 and ρ1 • For p > 1, T is unique • For p = 2,T = ∇ψ, with ψ convex [Brenier ’91] and optimal mass transfer follows straight lines Explicit computation in 1D with cumulative functions 11 / 50
⇒ p−Wasserstein distance between ρ0 and ρ1 • For p > 1, T is unique • For p = 2,T = ∇ψ, with ψ convex [Brenier ’91] and optimal mass transfer follows straight lines Explicit computation in 1D with cumulative functions Can not be extended to higher dimensions x ∈ Rd , d > 1 11 / 50
∈ T (ρ0, ρ1) satisfies the gradient equation ρ0(x) = ρ1(T(x))| det(∂T(x))| • For p = 2, T = ∇ψ ⇒ Monge-Ampere equation: det(D2ψ) = ρ0(x) ρ1(∇ψ(x)) [Oliker and Prussner ’88, Oberman ’08, Froese ’12, Benamou et al. ’12, ’16] Fast algorithms (second order methods) ρ1 should be lipschitz continuous with convex support ρ0 ρ1 12 / 50
∈ T (ρ0, ρ1) satisfies the gradient equation ρ0(x) = ρ1(T(x))| det(∂T(x))| • For p = 2, T = ∇ψ ⇒ Monge-Ampere equation: det(D2ψ) = ρ0(x) ρ1(∇ψ(x)) [Oliker and Prussner ’88, Oberman ’08, Froese ’12, Benamou et al. ’12, ’16] Fast algorithms (second order methods) ρ1 should be lipschitz continuous with convex support ρ0 ρ1 ρ0 ρ1 12 / 50
∈ T (ρ0, ρ1) satisfies the gradient equation ρ0(x) = ρ1(T(x))| det(∂T(x))| • For p = 2, T = ∇ψ ⇒ Monge-Ampere equation: det(D2ψ) = ρ0(x) ρ1(∇ψ(x)) [Oliker and Prussner ’88, Oberman ’08, Froese ’12, Benamou et al. ’12, ’16] Fast algorithms (second order methods) ρ1 should be lipschitz continuous with convex support ρ0 ρ1 ρ0 ρ1 • Regularized potential ψ [Paty et al. ’19] 12 / 50
The Knothe transport solves: min T∈T (ρ0,ρ1) d i=1 x C(xi , T(x)i ) • Can be computed explicitly PDE initialized with Knothe rearrangement • T = ∇ψ, then curl(T) = ∇ × T = 0 . Penalization of curl(T) [Angenent et al. ’03, Haber et al. ’10] . PDE on T in the transport map space • Lagrangian formulation using straight lines [Iollo and Lombardi, ’11] • PDE on ψ on the torus [Carlier et al. ’10, Bonnotte ’13] 13 / 50
The Knothe transport solves: min T∈T (ρ0,ρ1) d i=1 x C(xi , T(x)i ) • Can be computed explicitly PDE initialized with Knothe rearrangement • T = ∇ψ, then curl(T) = ∇ × T = 0 . Penalization of curl(T) [Angenent et al. ’03, Haber et al. ’10] . PDE on T in the transport map space • Lagrangian formulation using straight lines [Iollo and Lombardi, ’11] • PDE on ψ on the torus [Carlier et al. ’10, Bonnotte ’13] All these methods are limited to non vanishing densities 13 / 50
[0, 1] of the geodesic path ρ(x, t): ρ(x, t) = ((1 − t)Id + tT(x)) ρ0 • Non-convex problem over ρ(x, t) ∈ R and velocity field v(x, t) ∈ R2: W2(ρ0, ρ1)2 = min (v,ρ)∈Cv 1 2 [0,1]2 1 0 ρ(x, t)||v(x, t)||2dtdx, under the set of non-linear constraints Cv = (v, ρ) ; ∂t ρ + divx (ρv) = 0, v(0, ·) = v(1, ·) = 0, ρ(·, 0) = ρ0, ρ(·, 1) = ρ1 Change of variable (v, µ) → (m, µ), with m = µv: Convex cost J and linear constraints C No estimation of the transport map T, only the geodesic ρ(x, t) 14 / 50
follows straight lines • Can we include other physical priors on the transport ? Reintroduction of the velocity v [Hug et al. ’15, Maas et al. ’15] • Coupling with a smooth and non-convex penalization: K(m, ρ, v) = 1 2 [0;1]2 1 0 ||m − ρv||2dtdx • Regularity priors R(v): incompressibility (div(v) = 0), rigidity... 28 / 50
Adding constraints and generalizing cost function Extension of Dynamic OT • Discrete surfaces [Lavenant et al. ’18] • Sphere [Lang and P. ’19?] 31 / 50
Adding constraints and generalizing cost function Extension of Dynamic OT • Discrete surfaces [Lavenant et al. ’18] • Sphere [Lang and P. ’19?] Open problems • Study the existence of solution for non static domains • Modeling of data occlusions (clouds) for data assimilation in oceanography 31 / 50
i=1 µi δXi et ν = N j=1 νj δYj , Xi , Yj ∈ Rd • Wasserstein distance W2(µ, ν)2 = min P∈Pµ,ν { P , C = i,j Pi,j Ci,j } Pµ,ν = P ∈ RM×N , Pi,j 0, i,j Pi,j = 1, j Pi,j = µi , i Pi,j = νj • Pi,j is the mass transported from µi to νj • Cost matrix between locations Xi and Yj : Ci,j = d k=1 ||Xk i − Yk j ||2 Do not depend on feature dimension d Limited to low dimensions M and N 38 / 50
i=1 µi δXi et ν = N j=1 νj δYj , Xi , Yj ∈ Rd • Wasserstein distance W2(µ, ν)2 = min P∈Pµ,ν { P , C = i,j Pi,j Ci,j } Pµ,ν = P ∈ RM×N , Pi,j 0, i,j Pi,j = 1, j Pi,j = µi , i Pi,j = νj • Pi,j is the mass transported from µi to νj • Cost matrix between locations Xi and Yj : Ci,j = d k=1 ||Xk i − Yk j ||2 Do not depend on feature dimension d Limited to low dimensions M and N 38 / 50
Vi = TP(Xi) − Xi • Spatial consistency: graph of similarity ωij between Xi and Xj ⇒ Close pixels with similar colors should be matched together • Graph-laplacian of the mean transport field V (∆V)i := j∈EX (i) ωij d =1 (Vi − Vj ), • Color consistency penalize color shift to avoid artifacts R(P) = i |∆V|i • Still a linear program • Symetric formulation • Barycenter computation Graph 44 / 50
Vi = TP(Xi) − Xi • Spatial consistency: graph of similarity ωij between Xi and Xj ⇒ Close pixels with similar colors should be matched together • Graph-laplacian of the mean transport field V (∆V)i := j∈EX (i) ωij d =1 (Vi − Vj ), • Color consistency penalize color shift to avoid artifacts R(P) = i |∆V|i • Still a linear program • Symetric formulation • Barycenter computation Graph 44 / 50
Vi = TP(Xi) − Xi • Spatial consistency: graph of similarity ωij between Xi and Xj ⇒ Close pixels with similar colors should be matched together • Graph-laplacian of the mean transport field V (∆V)i := j∈EX (i) ωij d =1 (Vi − Vj ), • Color consistency penalize color shift to avoid artifacts R(P) = i |∆V|i • Still a linear program • Symetric formulation • Barycenter computation Graph 44 / 50
Vi = TP(Xi) − Xi • Spatial consistency: graph of similarity ωij between Xi and Xj ⇒ Close pixels with similar colors should be matched together • Graph-laplacian of the mean transport field V (∆V)i := j∈EX (i) ωij d =1 (Vi − Vj ), • Color consistency penalize color shift to avoid artifacts R(P) = i |∆V|i • Still a linear program • Symetric formulation • Barycenter computation Relaxed OT 44 / 50
Vi = TP(Xi) − Xi • Spatial consistency: graph of similarity ωij between Xi and Xj ⇒ Close pixels with similar colors should be matched together • Graph-laplacian of the mean transport field V (∆V)i := j∈EX (i) ωij d =1 (Vi − Vj ), • Color consistency penalize color shift to avoid artifacts R(P) = i |∆V|i • Still a linear program • Symetric formulation • Barycenter computation Relaxed and regularized OT 44 / 50
Giraud et al. ’18] Image Superpixels 2. Graph built from superpixel similarities (spatial+color) 3. Estimation of relaxed and regularised transport map 4. Final synthesis at pixel scale 45 / 50
→ ¯ Yi • Amplified with explicit regularisation Implicit regularisation of the transport matrix elements does not help • Entropic regularisation [Cuturi ’13] • Convex/sparse regularisation [Blondel et al. ’17, Dessein et al. ’19] Solution: deal with the color dispersion 47 / 50
→ ¯ Yi • Amplified with explicit regularisation Implicit regularisation of the transport matrix elements does not help • Entropic regularisation [Cuturi ’13] • Convex/sparse regularisation [Blondel et al. ’17, Dessein et al. ’19] Solution: deal with the color dispersion 47 / 50
→ ¯ Yi • Amplified with explicit regularisation Implicit regularisation of the transport matrix elements does not help • Entropic regularisation [Cuturi ’13] • Convex/sparse regularisation [Blondel et al. ’17, Dessein et al. ’19] Solution: deal with the color dispersion 47 / 50
P. ’15] Var(Y)i := Y − Yi 2 i • Minimisation of a concave term of mass dispersion: α i µi Var(Y)i ⇒ Associate a single color to Xi • Non-smooth and non-convex problem Forward-Backard [Attouch et al. ’13, Ochs et al. ’14], DC programming [Tao ’05] 48 / 50
P. ’15] Var(Y)i := Y − Yi 2 i • Minimisation of a concave term of mass dispersion: α i µi Var(Y)i ⇒ Associate a single color to Xi • Non-smooth and non-convex problem Forward-Backard [Attouch et al. ’13, Ochs et al. ’14], DC programming [Tao ’05] 48 / 50
constraint is necessary • Spatial regularization into transport map deals with artifacts • Non convex models prevent from creating new dull colors To be fair: • Doing color transfer with optimal transport is currently time consuming (1 minute for HD image) • Semi-automatic methods (high level segmentation, semantic analysis, simple optimal transport) [Bonneel et al. ’13, Frigo et al. ’14] give fast and accurate color transfer results even for videos But: • Enhancing OT framework will improve semi-automatic methods • Dealing with artifacts allows defining robust dissimilarity measures 50 / 50
npapadakis
genomethica
mickey_0226
rajeshrinet
trycycle
hashimoto_kei
calpis10000
nearme_tech
.png){kind=avatar}
statditto
shucho0103
gkazunii
csswizardry
addyosmani
eileencodes
kimpetersen
62gerente
techseoconnect
keavy
geoffreycrofte
scottboms
rkendrick25
baasie
reverentgeek
![Formulations, Numerical methods, Limitations Continuous [Benamou - Brenier ’00] Semi-discrete](https://files.speakerdeck.com/presentations/29747bcada0c4fc0809e3c85db8e97b0/slide_8.jpg){kind=link}
![Fluid mechanics formulation [Benamou-Brenier ’00] • Parameterization with t ∈](https://files.speakerdeck.com/presentations/29747bcada0c4fc0809e3c85db8e97b0/slide_49.jpg){kind=link}
![Fluid mechanics formulation [Benamou-Brenier ’00] • Parameterization with t ∈](https://files.speakerdeck.com/presentations/29747bcada0c4fc0809e3c85db8e97b0/slide_50.jpg){kind=link}
![Fluid mechanics formulation [Benamou-Brenier ’00] • Parameterization with t ∈](https://files.speakerdeck.com/presentations/29747bcada0c4fc0809e3c85db8e97b0/slide_51.jpg){kind=link}
![Fluid mechanics formulation [Benamou-Brenier ’00] • Parameterization with t ∈](https://files.speakerdeck.com/presentations/29747bcada0c4fc0809e3c85db8e97b0/slide_52.jpg){kind=link}
