Introduction 1 / 32
Adaptive color transfer
with relaxed optimal transport
Julien Rabin1, Sira Ferradans2 and Nicolas Papadakis3
1GREYC, University of Caen, 2 Duke University, 3 CNRS, Institut de Mathématiques de Bordeaux
IEEE International Conference on Image Processing
SS1 Variational and Morphological Optimizations:
A Tribute to Vicent Caselles
October 30th 2014, Paris
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Introduction 2 / 32
Optimal transport on histograms
Monge-Kantorovitch (MK) discrete mass transportation problem:
Map µ0
onto µ1
while minimizing the total transport cost
�������������
The two histograms must have the same mass.
Optimal transport cost is called the Wasserstein distance (Earth Mover’s
Distance)
Optimal transport map is the application mapping µ0
onto µ1
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Introduction 3 / 32
Applications in Image Processing and Computer Vision
Optimal transport as a framework to deﬁne statistical-based tools
Applications to many imaging and computer vision problems:
• Robust dissimilarity measure (Optimal transport cost):
Image retrieval [Rubner et al., 2000] [Pele and Werman, 2009]
SIFT matching [Pele and Werman, 2008] [Rabin et al., 2009]
3D shape recognition, Feature detection [Tomasi]
Object segmentation [Ni et al., 2009] [Swoboda and Schnorr, 2013]
• Tool for matching/interpolation (Optimal transport map):
Non-rigid shape matching, image registration [Angenent et al., 2004]
Texture synthesis and mixing [Ferradans et al., 2013]
Histogram speciﬁcation and averaging [Delon, 2004]
Color transfer [Pitié et al., 2007], [Rabin et al., 2011b]
Not to mention other applications (physics, economy, etc).
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Introduction 4 / 32
Color transfer
Target image (µ)
Source image (ν)
Optimal transport of µ onto ν
Target image after color transfer
Limitations:
• Mass conservation artifacts
• Irregularity of optimal transport map
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Introduction 5 / 32
Vicent’s Caselles legacy
Optimal transport:
• Optimal transportation networks [Bernot, Caselles and Morel, 2009]
Histogram transfer and color enhancement:
• Shape preserving local histogram modiﬁcation [Caselles, Lisani, Morel,
Sapiro, 1999]
• A perceptually inspired variational framework for color enhancement
[Palma-Amestoy, Provenzi, Bertalmío, Caselles, 2009]
• An analysis of visual adaptation and contrast perception for tone mapping
[Ferradans, Bertalmio, Provenzi, Caselles, 2011]
• A Variational Model for Histogram Transfer of Color Images [Papadakis,
Provenzi, Caselles, 2011]
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Introduction 6 / 32
Outline
Outline:
Part I. Computation of optimal transport between histograms
Part II. Optimal transport relaxation and regularization
Application to color transfer
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Optimal transport framework 7 / 32
Part I
Wasserstein distance between histograms
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Optimal transport framework 8 / 32
Formulation for clouds of points
Deﬁnition: L2-Wasserstein Distance Given two clouds of points
X, Y ⊂ Rd×N of N elements in Rd with equal masses 1
N
, the quadratic
Wasserstein distance is deﬁned as
W2
(X, Y)2 = min
σ∈ΣN
1
N
N
i=1
Xi
− Yσ(i)
2 (1)
where ΣN
is the set of all permutations of N elements.
⇔ Optimal Assignment problem
Can be computed using standard sorting algorithms when d = 1
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Optimal transport framework 9 / 32
Exact solution in unidimensional case (d = 1) for histograms
Histograms may be seen as clouds of points with non-uniform masses, so
that
µ(x) =
M
i=1
mi
δXi
(x), s.t.
i
mi
= 1, mi
≥ 0 ∀i
Computing the Lp-Wasserstein distance for one-dimensional histograms is
still simple for p ≥ 1.
Optimal transport cost writes [Villani, 2003]
Wp(µ, ν) = H−1
µ
− H−1
ν
p
=
1
0
H−1
µ
(t) − H−1
ν
(t) p
dt
1
p
where Hµ
(t) = t
−∞
dµ =
Xi t
mi
is the cumulative distribution function of µ
and H−1
µ
(t) = inf {s \ Rµ
(s) t} its pseudo-inverse.
Time complexity: O(N) operations if bins are already sorted.
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Optimal transport framework 10 / 32
Exact solution in unidimensional case (d = 1) for histograms
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Optimal transport framework 10 / 32
Exact solution in unidimensional case (d = 1) for histograms
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Optimal transport framework 10 / 32
Exact solution in unidimensional case (d = 1) for histograms
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Optimal transport framework 10 / 32
Exact solution in unidimensional case (d = 1) for histograms
Can not be extended to higher dimensions
as the cumulative function Hµ
: x ∈ Rd → Hµ
(x) ∈ R is not invertible for
d > 1
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Optimal transport framework 11 / 32
Exact solution in general case (d>1)
Transport cost between normalized histograms µ and ν, where
µ = M
i=1
mi
δXi
, ν = N
j=1
nj
δYj
, mi
, nj
≥ 0 and
i
mi
=
j
nj
= 1.
• mi
, nj
are the masses at locations Xi
, Yj
It can be recasted as a linear programming problem: linear cost + linear
constraints
W2
(µ, ν)2 = min
P∈Pµ,ν
P , C =
i,j
Pi,j
Xi
− Yj
2
= min
A·p=b
pT c
• C is the ﬁxed cost assignment matrix between histograms bins:
Ci,j
=
d
k=1
||Xk
i
− Yk
j
||2
• Pµ,ν
is the set of non negative matrices P with marginals µ and ν, ie
P(µ, ν) =
P ∈ RM×N ,
Pi,j
0,
i,j
Pi,j
= 1,
j
Pi,j
= mi
,
i
Pi,j
= nj
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Optimal transport framework 12 / 32
Illustration in unidimensional case (d = 1) for histograms
Two histograms µ = {1
3
, 2
3
} and ν = {1
3
, 1
6
, 1
2
}
Example: µi
is the production at plant i and νj
is the storage capacity of
storehouse j
Matrix C deﬁnes the transport cost from i to j:
C11
= 22 C21
= 62
C12
= 12 C22
= 52
C13
= 52 C23
= 12
The set of admissible matrices P is
µ1
= 1/3 µ2
= 2/3
P11
P21
ν1
= 1/3
P12
P22
ν2
= 1/6
P13
P23
ν3
= 1/2
Pij
is the mass that is transported from µi
to νj
.
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Optimal transport framework 12 / 32
Illustration in unidimensional case (d = 1) for histograms
Two histograms µ = {1
3
, 2
3
} and ν = {1
3
, 1
6
, 1
2
}
Example: µi
is the production at plant i and νj
is the storage capacity of
storehouse j
Matrix C deﬁnes the transport cost from i to j:
C11
= 22 C21
= 62
C12
= 12 C22
= 52
C13
= 52 C23
= 12
The set of admissible matrices P is
µ1
= 1/3 µ2
= 2/3
1/9 2/9 ν1
= 1/3
1/18 1/9 ν2
= 1/6
1/6 1/3 ν3
= 1/2
Pij
is the mass that is transported from µi
to νj
.
The transport cost is W(µ, ν) =
ij
Pij
Cij
= 15
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Optimal transport framework 12 / 32
Illustration in unidimensional case (d = 1) for histograms
Two histograms µ = {1
3
, 2
3
} and ν = {1
3
, 1
6
, 1
2
}
Example: µi
is the production at plant i and νj
is the storage capacity of
storehouse j
Matrix C deﬁnes the transport cost from i to j:
C11
= 22 C21
= 62
C12
= 12 C22
= 52
C13
= 52 C23
= 12
The set of admissible matrices P is
µ1
= 1/3 µ2
= 2/3
1/3 0 ν1
= 1/3
0 1/6 ν2
= 1/6
0 1/2 ν3
= 1/2
Pij
is the mass that is transported from µi
to νj
.
The transport cost is W(µ, ν) =
ij
Pij
Cij
= 6
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Optimal transport framework 13 / 32
Optimal transport solution illustration in 1D
Histograms µ and ν (on uniform grid Ω) Optimal ﬂow P
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Optimal transport framework 13 / 32
Optimal transport solution illustration in 1D
Histograms µ and ν (on uniform grid Ω) Optimal ﬂow P
Remark: Masses can be splitted by transport
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Optimal transport framework 13 / 32
Optimal transport solution illustration in 1D
Histograms µ and ν (on uniform grid Ω) Optimal ﬂow P
Remark: Masses can be splitted by transport
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Optimal transport framework 14 / 32
Optimal transport solution with linear programming method
Discrete mass transportation problem for histograms can be solved with
standard linear programming algorithms (simplex, interior point methods).
Dedicated algorithms are more efﬁcient for optimal assignment problem (e.g
Hungarian and Auction algorithms in O(N3))
Computation can be (slightly) accelerated when using other costs than L2
(e.g. L1
[Ling and Okada, 2007], Truncated L1
[Pele and Werman, 2008])
Advantages
Complexity does not depend on feature dimension d
Limitation
Intractable for signal processing applications where N 103
(considering time complexity & memory limitation)
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Relaxation Regularization Conclusion 15 / 32
Part II
Relaxation and regularization
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Relaxation Regularization Conclusion 16 / 32
Problem Statement
Histogram speciﬁcation exhibits strong limitations of optimal transport when
dealing with image processing:
• Color artifacts due to the exact speciﬁcation (histograms can have very
different shapes)
• Irregularities: Transport map is not consistent in the color domain
It does not take into account spatial information
Histogram equalization + Filtering
Proposed solution
• Relax mass conservation constraint
• Promote regular transport ﬂows (color consistency)
• Include spatial information (spatial consistency)
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Relaxation Regularization Conclusion 17 / 32
Constraint Relaxation
Idea 1: Relaxation of mass conservation constraints [Ferradans et al., 2013]
We consider the transport cost between normalized histograms µ and ν,
µ(x) =
M
i=1
mi
δXi
(x), s.t.
i
mi
= 1, mi
≥ 0 ∀i
Relaxed Formulation :
P ∈ arg min
P∈Pκ(µ,ν)
P, C =
1 i N,1 j M
Pi,j
Ci,j
• with Ci,j
= Xi
− Yj
2, where Xi
∈ Ω ⊂ Rd is bin centroid of µ for index i;
• with new (linear) constraints:
Pκ
(µ, ν) =
Pi,j
0,
i,j
Pi,j
= 1,
j
Pi,j
= mi
,
κnj
≤
i
Pi,j
≤ Knj
where capacity parameters are such that κ ≤ 1 ≤ K: hard to tune
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Relaxation Regularization Conclusion 17 / 32
Constraint Relaxation
Idea 1: Relaxation of mass conservation constraints [Ferradans et al., 2013]
We consider the transport cost between normalized histograms µ and ν,
µ(x) =
M
i=1
mi
δXi
(x), s.t.
i
mi
= 1, mi
≥ 0 ∀i
Relaxed Formulation :
P ∈ arg min
P∈Pκ(µ,ν)
P, C =
1 i N,1 j M
Pi,j
Ci,j
• with Ci,j
= Xi
− Yj
2, where Xi
∈ Ω ⊂ Rd is bin centroid of µ for index i;
• with new (linear) constraints:
P(µ, ν) =
Pi,j
0,
i,j
Pi,j
= 1,
j
Pi,j
= mi
,
i
Pi,j
= nj
where capacity parameters are such that κ ≤ 1 ≤ K: hard to tune
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Relaxation Regularization Conclusion 18 / 32
Proposed relaxed histogram matching
Idea 2: Use capacity variables as unknowns
{P , κ } ∈ arg min
P∈Pκ(µ,ν)
κ∈RN ,κ≥0, κ, n =1
P, C + ρ||κ − 1||
1
where
Pκ
(µ, ν) =
Pi,j
0,
i,j
Pi,j
= 1,
j
Pi,j
= mi
,
i
Pi,j
= κj
nj
⇒ Still a linear program
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Relaxation Regularization Conclusion 19 / 32
Illustration of relaxed transport for histograms
Two histograms µ = {1
3
, 2
3
} and ν = {1
3
, 1
6
, 1
2
}
The set of admissible matrices P is
1/3 2/3
P11
P21
= κ1
/3
P12
P22
= κ2
/6
P13
P23
= κ3
/2
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Relaxation Regularization Conclusion 20 / 32
Illustration of relaxed transport
Optimal transport
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Relaxation Regularization Conclusion 20 / 32
Illustration of relaxed transport
Relaxed optimal transport
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Relaxation Regularization Conclusion 21 / 32
Relaxed color transfer: comparison with raw OT
Target Raw OT Relaxed OT Source
No color or spatial regularization
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Relaxation Regularization Conclusion 22 / 32
Proposed relaxed and regularized histogram matching
Idea 3: Add regularization prior
{P , κ } ∈ arg min
P∈Pκ(µ,ν)
κ∈RN ,κ≥0, κ, n =1
P, C + ρ||κ − 1||
1
+ λR(P).
where
Pκ
(µ, ν) =
Pi,j
0,
i,j
Pi,j
= 1,
j
Pi,j
= mi
,
i
Pi,j
= κj
nj
and R(P) models some regularity priors
⇒ Still a linear program
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Relaxation Regularization Conclusion 23 / 32
Regularity of transport map
• Global regularization: Deﬁning the regularity of the ﬂow matrix is a
NP-hard problem
• Average transport map Instead, we use the Posterior mean to estimate a
one-to-one transfer function T between µ and ν
T(Xi
) = Yi
=
1
j
Pij
j
Pij
Yj
= (Dµ
PY)i
Flow P
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Relaxation Regularization Conclusion 23 / 32
Regularity of transport map
• Global regularization: Deﬁning the regularity of the ﬂow matrix is a
NP-hard problem
• Average transport map Instead, we use the Posterior mean to estimate a
one-to-one transfer function T between µ and ν
T(Xi
) = Yi
=
1
j
Pij
j
Pij
Yj
= (Dµ
PY)i
Flow P
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Relaxation Regularization Conclusion 23 / 32
Regularity of transport map
• Global regularization: Deﬁning the regularity of the ﬂow matrix is a
NP-hard problem
• Average transport map Instead, we use the Posterior mean to estimate a
one-to-one transfer function T between µ and ν
T(Xi
) = Yi
=
1
j
Pij
j
Pij
Yj
= (Dµ
PY)i
Flow P
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Relaxation Regularization Conclusion 24 / 32
Regularity of transport map for color transfer
• Average transport vector ﬁeld : Vi
= T(Xi
) − Xi
= (Dµ
PY − X)i
• Graph is built from the similarity ωij
between bins Xi
, Xj
• Graph-laplacian
(∆V)i
:=
j∈EX (i)
ωij
d
=1
(Vi
− Vj
),
• Penalization: Color shift does not introduce artifacts and should be
preserved (piecewise constant prior)
R(P) =
i
|∆(Dµ
PY − X)|i
• Spatial information is required and introduced via graph structure
⇒ Close pixels with similar colors should be matched to similar colors
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Relaxation Regularization Conclusion 25 / 32
Illustration of relaxed and regularized transport
Graph
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Relaxation Regularization Conclusion 25 / 32
Illustration of relaxed and regularized transport
Relaxed optimal transport
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Relaxation Regularization Conclusion 25 / 32
Illustration of relaxed and regularized transport
Relaxed and Regularized optimal transport
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Relaxation Regularization Conclusion 26 / 32
Final process
•Pre-processing: Spatio-color clustering
Super-pixel clustering to decrease the dimension
Graph built from super-pixel self-similarities (colorimetric and spatial)
Image Super-pixels
•Estimation of the relaxed and regularized optimal transport map
•Image synthesis
Computation of the likelihood of each pixel (position and color) w.r.t each
super-pixels
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Relaxation Regularization Conclusion 27 / 32
Relaxed color transfer: comparisons
Target Source
Our method [Ferradans et al., 2013] [Papadakis et al., 2011]
[Pitié and Kokaram, 2006] [Pitié and Kokaram, 2006]
+[Rabin et al., 2011a]
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Relaxation Regularization Conclusion 28 / 32
Relaxed and adaptive color transfer
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Relaxation Regularization Conclusion 28 / 32
Relaxed and adaptive color transfer
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Relaxation Regularization Conclusion 28 / 32
Relaxed and adaptive color transfer
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Relaxation Regularization Conclusion 29 / 32
Conclusion
What can be remembered for image processing applications:
• Relaxation of mass conservation constraint is necessary !
• Including spatial regularization into transport map deals with artifacts
To be fair:
• Doing all the color transfer with optimal transport is currently unrealistic
(1 minute for an HR image)
• Semi-automatic methods (high level segmentation, semantic analysis,
simple optimal transport) give fast and accurate color transfer results.
But:
• Enhancing the optimal transport framework will improve semi-automatic
methods
• Dealing with artifacts is important for other issues (dissimilarity
measures...)
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Relaxation Regularization Conclusion 30 / 32
Future works
• Extension to other statistics (patches, wavelet coefﬁcients...)
• Faster implementation
• Other regularization (entropy regularization [Cuturi 2013])
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

Relaxation Regularization Conclusion 31 / 32
Question time
Thank you for your attention !
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

32 / 32
Bibliography
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

32 / 32
Delon, J. (2004).
Midway image equalization.
JMIV, 21(2):119–134.
Ferradans, S., Papadakis, N., Peyré, G., and Aujol, J.-F. (2013).
Regularized discrete optimal transport.
ArXiv e-prints 1307.5551.
Ling, H. and Okada, K. (2007).
An Efﬁcient Earth Mover’s Distance Algorithm for Robust Histogram Comparison.
IEEE Trans. PAMI, 29(5):840–853.
Ni, K., Bresson, X., Chan, T., and Esedoglu, S. (2009).
Local histogram based segmentation using the wasserstein distance.
IJCV, 84:97–111.
Papadakis, N., Provenzi, E., and Caselles, V. (2011).
A variational model for histogram transfer of color images.
IEEE Transaction on Image Processing, 19(12).
Pele, O. and Werman, M. (2008).
A Linear Time Histogram Metric for Improved SIFT Matching.
In ECCV08.
Pele, O. and Werman, M. (2009).
Fast and robust earth mover’s distances.
In ICCV.
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

32 / 32
Pitié, F. and Kokaram, A. (2006).
The Linear Monge-Kantorovitch Colour Mapping for Example-Based Colour Transfer.
In Proceedings of 3rd European Conference on Visual Media Production (CVMP’06), London.
Pitié, F., Kokaram, A., and Dahyot, R. (2007).
Automated colour grading using colour distribution transfer.
Computer Vision and Image Understanding.
Rabin, J., Delon, J., and Gousseau, Y. (2009).
A statistical approach to the matching of local features.
SIAM Journal on Imaging Sciences, 2(3):931–958.
Rabin, J., Delon, J., and Gousseau, Y. (2011a).
Removing artefacts from color and contrast modiﬁcation.
IEEE Trans. Image Proc.
http://hal.archives-ouvertes.fr/hal-00505966/en/.
Rabin, J., Peyré, G., Delon, J., and Bernot, M. (2011b).
Wasserstein barycenter and its application to texture mixing.
Proc. SSVM’11.
Rubner, Y., Tomasi, C., and Guibas, L. J. (2000).
The earth mover’s distance as a metric for image retrieval.
IJCV, 40(2):99–121.
Swoboda, P. and Schnorr, C. (2013).
Convex variational image restoration with histogram priors.
SIAM Journal on Imaging Sciences, 6(3):1719–1735.
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport

32 / 32
Villani, C. (2003).
Topics in Optimal Transportation.
American Mathematical Society.
J. Rabin, S. Ferradans, N. Papadakis Adaptive color transfer with relaxed optimal transport