yN } sampled from Y ∼ ν • Synthetic distribution µθ = gθ ♯ζ Goal: find the best θ s.t. µθ is close in some sense to ν N. Papadakis Wasserstein Generative Models for Texture Synthesis 5 / 54
as generative model gθ GAN [Goodfellow et al. ’14] • Discriminator dη between fake gθ (Z ) and true Y samples min max Eν [log(dη (Y ))] + Eζ [log(1 − dη (gθ (Z )))] θ η WGAN [Arjovsky et al. ’17] • Compare fake Z ∼ µθ = gθ ♯ζ and true Y ∼ ν sample distributions min D(µθ , ν) θ • Duality of Wasserstein distance D = W1 yields min max Eν [ψ(Y )] − Eζ [ψ(gθ (Z ))] θ ψ∈Lip1 • Parameterization of the dual variable ψ with dη Questions: Other Wasserstein costs? Training strategies? N. Papadakis Wasserstein Generative Models for Texture Synthesis 6 / 54
’99] → • Iterative refinement with nearest neighbors [Kwatra, ’05] • Impose patch distribution at different scales [Gutierrez et al. ’17, Leclaire and Rabin ’19] ✗ Image composed of patches processed independently ✗ Apply the algorithm for each new synthesis N. Papadakis Wasserstein Generative Models for Texture Synthesis 7 / 54
features at different scales [Gatys et al. ’15] min ||Gu − Gv ||2 u → Prescribe features of an example image v ✓ Process the whole image u and not its patches independently • Train a feedforward generative network gθ [Ulyanov et al, ’16] min Eζ ||Ggθ (Z ) − Gv ||2 θ ✓ Real time synthesis Questions: Wasserstein metric between feature distributions? Dealing with patches? N. Papadakis Wasserstein Generative Models for Texture Synthesis 8 / 54
between densities of probability • Transport a mass µ(x) onto ν(y) x y Euclidean • Define a cost c(x, y ) of mass transport between locations x and y • OT: application with mimimal global cost that transfers µ onto ν • If c(x, y ) = ||x − y ||p , Lp Wasserstein distance • Interpolation with transport map T N. Papadakis Wasserstein Generative Models for Texture Synthesis 10 / 54
between densities of probability • Transport a mass µ(x) onto ν(y) x y Euclidean • Define a cost c(x, y ) of mass transport between locations x and y • OT: application with mimimal global cost that transfers µ onto ν • If c(x, y ) = ||x − y ||p , Lp Wasserstein distance • Interpolation with transport map T N. Papadakis Wasserstein Generative Models for Texture Synthesis 10 / 54
between densities of probability • Transport a mass µ(x) onto ν(y) x y Wasserstein • Define a cost c(x, y ) of mass transport between locations x and y • OT: application with mimimal global cost that transfers µ onto ν • If c(x, y ) = ||x − y ||p , Lp Wasserstein distance • Interpolation with transport map T N. Papadakis Wasserstein Generative Models for Texture Synthesis 10 / 54
[Mérigot ’11, et al. ’17] What’s next • More on the (semi-discrete formulation) of the optimal transport cost OT(µ, ν) • Differentiability and regularization of the cost min OT(µθ , ν) θ • Application to patch-based texture synthesis N. Papadakis Wasserstein Generative Models for Texture Synthesis 11 / 54
× Rd → R • µ, ν probability measures supported on compacts X , Y ⊂ Rd , let Z c(x, y )dπ(x, y ) OTc (µ, ν) = min π∈Π(µ,ν) Π(µ, ν) : set of probability measures on X × Y with marginals µ, ν. Theorem [Villani ’03, Santambrogio ’15] Strong duality holds i.e. Z OTc (µ, ν) = max φ,ψ Z φdµ + X ψdν Y where max is taken on all functions φ ∈ L1 (µ), ψ ∈ L1 (ν) such that φ(x) + ψ(y ) ⩽ c(x, y ) dµ(x) a.e., dν(y ) a.e. N. Papadakis Wasserstein Generative Models for Texture Synthesis 12 / 54
min [c(x, y ) − φ(x)] x∈X ψ c (x) = min [c(x, y ) − ψ(y )] y ∈Y Semi-dual Z OTc (µ, ν) = max φ,ψ Z φdµ + X Z ψdν = max Y φ∈C (X ) ZX = max ψ∈C (Y) N. Papadakis Z φc (y )dν(y ) φ(x)dµ(x) + c ZY ψ (x)dµ(x) + X Wasserstein Generative Models for Texture Synthesis ψ(y )dν(y ) Y 13 / 54
min [c(x, y ) − φ(x)] x∈X ψ c (x) = min [c(x, y ) − ψ(y )] y ∈Y Semi-dual Z OTc (µ, ν) = max φ,ψ Z φdµ + X Z ψdν = max Y φ∈C (X ) Z φc (y )dν(y ) φ(x)dµ(x) + ZX = max ψ∈C (Y) c ZY ψ (x)dµ(x) + X ψ(y )dν(y ) Y • c-transforms inherit regularity from c N. Papadakis Wasserstein Generative Models for Texture Synthesis 13 / 54
min [c(x, y ) − ψ(y )] φ (y ) = min [c(x, y ) − φ(x)] y ∈Y x∈X Semi-dual Z OTc (µ, ν) = max φ,ψ Z Z φdµ + X ψdν = max φ∈C (X ) Y ZX = max ψ∈C (Y) Z φc (y )dν(y ) φ(x)dµ(x) + ZY c ψ (x)dµ(x) + X ψ(y )dν(y ) Y • If c(x, y ) = ||x − y ||, then ψ c = −ψ and ψ is 1-lipschitz [Kantorovich and Rubinstein, ’58] Z ψ∈Lip1 N. Papadakis Z ψ(y )dν(y ) − OTc (µ, ν) = max Y ψ(x)dµ(x) X Wasserstein Generative Models for Texture Synthesis 13 / 54
min [c(x, y ) − ψ(y )] φ (y ) = min [c(x, y ) − φ(x)] y ∈Y x∈X Semi-dual Z OTc (µ, ν) = max φ,ψ Z φdµ + X Z ψdν = max φ∈C (X ) Y ZX = max ψ∈C (Y) • For a discrete ν = X φc (y )dν(y ) ZY c ψ (x)dµ(x) + X ψ(y )dν(y ) Y J j=1 νj δyj and ψj = ψ(yj ) ψ c (x) = min j∈{1,··· ,J} c(x, yj ) − ψj Z OTc (µ, ν) = max {ψj }Jj=1 N. Papadakis Z φ(x)dµ(x) + X ψ c (x)dµ(x) + J X νj ψj j=1 Wasserstein Generative Models for Texture Synthesis 13 / 54
OTc (µθ , ν) = inf max θ θ ψ c ψ dµθ + Z ψdν • Is the loss function OTc (µθ , ν) regular? • If not, what kind of problems happen? • Do these problems appear in discrete/semi-discrete cases? • Does this scale up in order to address image synthesis problems? N. Papadakis Wasserstein Generative Models for Texture Synthesis 15 / 54
inf max θ θ ψ ψ c dµθ + Z ψdν [Goodfellow et al. ’14] GAN (Jensen-Shannon divergence) [Arjovsky et al. ’17] Wasserstein GAN (Wasserstein distance with L1 -cost) [Gulrajani et al. ’17] WGAN-GP: Wasserstein GAN with Gradient Penalty [Genevay et al. ’18] Generative models with Sinkhorn divergences [Salimans et al. ’18] Improving GANs using Optimal transport [Liu et al. ’18] WGAN-TS (for Two Steps) [Chen et al. ’19] Semi-discrete Wasserstein generative network training Differential properties of OT [Burger et al. ’12] Wasserstein distance and regularized densitiy [Cuturi and Peyré ’15] Gradient of regularized Wasserstein distance [Cazelles et al. ’19] Proof of differentiability in both previous settings [Degournay et al. ’19] Differentiation w.r.t. the discrete target measure N. Papadakis Wasserstein Generative Models for Texture Synthesis 17 / 54
c continuous, X , Y ⊂ Rd and fix ν • µ 7→ OTc (µ, ν) is convex • For all subgradient φ ∈ ∂µ OTc (µ, ν) Z Z OTc (µ, ν) = φdµ + φc dν Hence Z OTc (µ + χ, ν) ⩾ OTc (µ, ν) + φdχ • If φ is unique up to additive constants, then one can show Gateaux-differentiability at (µ, ν) NB: Extension to entropy-regularized optimal transport [Feydy et al. ’18] Sufficient condition [Santambrogio ’15] c is C 1 and Supp(µ) (or ν) is the closure of a bounded connected open set Does not include c(x, y ) = ||x − y || N. Papadakis Wasserstein Generative Models for Texture Synthesis 18 / 54
c continuous, X , Y ⊂ Rd and fix ν • µ 7→ OTc (µ, ν) is convex • For all subgradient φ ∈ ∂µ OTc (µ, ν) Z Z OTc (µ, ν) = φdµ + φc dν Hence Z OTc (µ + χ, ν) ⩾ OTc (µ, ν) + φdχ • If φ is unique up to additive constants, then one can show Gateaux-differentiability at (µ, ν) NB: Extension to entropy-regularized optimal transport [Feydy et al. ’18] Sufficient condition [Santambrogio ’15] c is C 1 and Supp(µ) (or ν) is the closure of a bounded connected open set Does not include c(x, y ) = ||x − y || N. Papadakis Wasserstein Generative Models for Texture Synthesis 18 / 54
c continuous, X , Y ⊂ Rd and fix ν • µ 7→ OTc (µ, ν) is convex • For all subgradient φ ∈ ∂µ OTc (µ, ν) Z Z OTc (µ, ν) = φdµ + φc dν Hence Z OTc (µ + χ, ν) ⩾ OTc (µ, ν) + φdχ • If φ is unique up to additive constants, then one can show Gateaux-differentiability at (µ, ν) NB: Extension to entropy-regularized optimal transport [Feydy et al. ’18] Sufficient condition [Santambrogio ’15] c is C 1 and Supp(µ) (or ν) is the closure of a bounded connected open set Does not include c(x, y ) = ||x − y || N. Papadakis Wasserstein Generative Models for Texture Synthesis 18 / 54
c continuous, X , Y ⊂ Rd and fix ν • µ 7→ OTc (µ, ν) is convex • For all subgradient φ ∈ ∂µ OTc (µ, ν) Z Z OTc (µ, ν) = φdµ + φc dν Hence Z OTc (µ + χ, ν) ⩾ OTc (µ, ν) + φdχ • If φ is unique up to additive constants, then one can show Gateaux-differentiability at (µ, ν) NB: Extension to entropy-regularized optimal transport [Feydy et al. ’18] Sufficient condition [Santambrogio ’15] c is C 1 and Supp(µ) (or ν) is the closure of a bounded connected open set Does not include c(x, y ) = ||x − y || N. Papadakis Wasserstein Generative Models for Texture Synthesis 18 / 54
Z inf OTc (µθ , ν) = inf max θ For F (ψ, θ) = θ ψ c ψ dµθ + Z ψdν c X ψ (x)dµθ (x) + Y ψ(y )dν(y ) we have R R W (θ) := OTc (µθ , ν) = max F (ψ, θ) ψ Potential ψ acts as a discriminator between µθ and ν N. Papadakis Wasserstein Generative Models for Texture Synthesis 19 / 54
Z inf OTc (µθ , ν) = inf max θ For F (ψ, θ) = θ ψ Z c ψ dµθ + ψdν c X ψ (x)dµθ (x) + Y ψ(y )dν(y ) we have R R W (θ) := OTc (µθ , ν) = max F (ψ, θ) ψ Theorem [Arjovsky et al., 2017] Let θ0 and ψ0∗ satisfying W (θ0 ) = F (ψ0∗ , θ0 ). If W and θ 7→ F (ψ0∗ , θ) are both differentiable at θ0 , then ∇W (θ0 ) = ∇θ F (ψ0∗ , θ0 ) (Grad-OT) " There are cases where no such couple (ψ ∗ , θ0 ) exists 0 N. Papadakis Wasserstein Generative Models for Texture Synthesis 19 / 54
Z inf OTc (µθ , ν) = inf max θ For F (ψ, θ) = θ ψ Z c ψ dµθ + ψdν c X ψ (x)dµθ (x) + Y ψ(y )dν(y ) we have R R W (θ) := OTc (µθ , ν) = max F (ψ, θ) ψ Theorem [Arjovsky et al., 2017] Let θ0 and ψ0∗ satisfying W (θ0 ) = F (ψ0∗ , θ0 ). If W and θ 7→ F (ψ0∗ , θ) are both differentiable at θ0 , then ∇W (θ0 ) = ∇θ F (ψ0∗ , θ0 ) (Grad-OT) " There are cases where no such couple (ψ ∗ , θ0 ) exists 0 N. Papadakis Wasserstein Generative Models for Texture Synthesis 19 / 54
∈ Rd , and let ν = 21 δy1 + 21 δy2 with y1 , y2 ∈ Rd distinct. Let c(x, y ) = ∥x − y ∥pp , p > 1. Then • θ 7→ W (θ) is differentiable everywhere. 2 and any ψ0∗ ∈ argmaxψ F (ψ, θ0 ), θ 7→ F (ψ0∗ , θ) is not • For θ0 ̸= y1 +y 2 differentiable at θ0 . 2 Hence (Grad-OT) relation does not hold (except for θ0 = y1 +y 2 ). Proof • W (θ) = 21 c(θ, y1 ) + c(θ, y2 ) = ∥θ − y1 ∥pp + ∥θ − y2 ∥pp N. Papadakis Wasserstein Generative Models for Texture Synthesis 20 / 54
∈ Rd , and let ν = 21 δy1 + 21 δy2 with y1 , y2 ∈ Rd distinct. Let c(x, y ) = ∥x − y ∥pp , p > 1. Then • θ 7→ W (θ) is differentiable everywhere. 2 and any ψ0∗ ∈ argmaxψ F (ψ, θ0 ), θ 7→ F (ψ0∗ , θ) is not • For θ0 ̸= y1 +y 2 differentiable at θ0 . 2 Hence (Grad-OT) relation does not hold (except for θ0 = y1 +y 2 ). Proof • W (θ) = 21 c(θ, y1 ) + c(θ, y2 ) = ∥θ − y1 ∥pp + ∥θ − y2 ∥pp N. Papadakis Wasserstein Generative Models for Texture Synthesis 20 / 54
W (θ) θ need an estimation of the gradient. • For the L2 -cost ∇W (θ) = θ − y1 + θ − y2 is estimated by ( θ − y1 ∇θ F (ψ, θ) = θ − y2 if θ ∈ L1 (ψ) if θ ∈ L2 (ψ) Solution 1 Regularization of optimal transport 2 Assumption on the generator N. Papadakis Wasserstein Generative Models for Texture Synthesis 21 / 54
W (θ) θ need an estimation of the gradient. • For the L2 -cost ∇W (θ) = θ − y1 + θ − y2 is estimated by ( θ − y1 ∇θ F (ψ, θ) = θ − y2 if θ ∈ L1 (ψ) if θ ∈ L2 (ψ) Solution 1 Regularization of optimal transport 2 Assumption on the generator N. Papadakis Wasserstein Generative Models for Texture Synthesis 21 / 54
> 0, the regularized OT cost is defined by Z λ c(x, y )dπ(x, y ) + λKL(π|µ ⊗ ν) OTc (µ, ν) = inf π∈Π(µ,ν) where KL is the Kullback-Leibler divergence: (R dπ(x,y ) dπ log dµ(x)dν(y ) dπ(x, y ) if dµdν exists KL(π|µ ⊗ ν) = . +∞ otherwise N. Papadakis Wasserstein Generative Models for Texture Synthesis 22 / 54
× Y), then Z Z OTλc (µ, ν) = max ψ c,λ (x)dµ(x) + ψ(y )dν(y ) ψ∈L∞ (Y) X Y where ψ c,λ (x) = Soft min c(x, yj ) − ψj j∈{1,··· ,J} Z ψ(y ) − c(x, y ) dν(y ) exp = −λ log λ Y Theorem [Genevay ’19, Chizat et al. ’19] For c ∈ L∞ (X × Y), the semi-dual problem admits a solution ψ ∗ ∈ L∞ (ν) which is unique ν − a.e. up to an additive constant NB: Solutions are characterized by the fixed point equation (ψ c,λ )c,λ = ψ N. Papadakis Wasserstein Generative Models for Texture Synthesis 23 / 54
g(θ, Z ) • Z : r.v. in Z ⊂ Rp with distribution ζ min OTλc (µθ , ν) = min max E[ψ c,λ (g(θ, Z ))] + θ θ ψ∈L∞ (Y) | N. Papadakis {z :=F λ (ψ,θ) Wasserstein Generative Models for Texture Synthesis Z Y ψdν } 24 / 54
g(θ, Z ) • Z : r.v. in Z ⊂ Rp with distribution ζ min OTλc (µθ , ν) = min max E[ψ c,λ (g(θ, Z ))] + θ θ ψ∈L∞ (Y) | Z Y {z :=F λ (ψ,θ) ψdν } Hypothesis (H) There exists L : Θ × Z → R+ such that, for any θ ∈ Θ, there is a neighborhood Vθ of θ such that ∀θ′ ∈ Vθ Z − a.s., ∥g(θ, Z ) − g(θ′ , Z )∥ ⩽ L(θ, Z )∥θ − θ′ ∥ with E[L(θ, Z )] < ∞. Proposition Let λ > 0. Assume that c is C 1 , and g satisfies (H). For any θ0 ∈ Θ and any ψ ∈ L∞ (Y), θ 7→ F λ (ψ, θ) is differentiable at θ0 h i T ∇θ F λ (ψ, θ0 ) = E (∂θ g(θ0 , Z )) ∇ψ c,λ (g(θ0 , Z )) If g is C 1 , then so is F λ (ψ, ·) N. Papadakis Wasserstein Generative Models for Texture Synthesis 24 / 54
g(θ, Z ) • Z : r.v. in Z ⊂ Rp with distribution ζ min OTλc (µθ , ν) = min max E[ψ c,λ (g(θ, Z ))] + θ θ ψ∈L∞ (Y) | Z Y {z :=F λ (ψ,θ) ψdν } Hypothesis (H) There exists L : Θ × Z → R+ such that, for any θ ∈ Θ, there is a neighborhood Vθ of θ such that ∀θ′ ∈ Vθ Z − a.s., ∥g(θ, Z ) − g(θ′ , Z )∥ ⩽ L(θ, Z )∥θ − θ′ ∥ with E[L(θ, Z )] < ∞. Proposition Let λ > 0. Assume that c is C 1 , and g satisfies (H). For any θ0 ∈ Θ and any ψ ∈ L∞ (Y), θ 7→ F λ (ψ, θ) is differentiable at θ0 h i T ∇θ F λ (ψ, θ0 ) = E (∂θ g(θ0 , Z )) ∇ψ c,λ (g(θ0 , Z )) If g is C 1 , then so is F λ (ψ, ·) N. Papadakis Wasserstein Generative Models for Texture Synthesis 24 / 54
0. Assume that c is C 1 , g is C 1 and satisfies (H). Then W λ : θ 7→ OTλc (µθ , ν) is C 1 , and for any θ ∈ Θ, h i ∇θ W λ (θ) = ∇θ F λ (ψ ∗ , θ) = E (∂θ g(θ, Z ))T ∇ψ ∗,c,λ (g(θ, Z )) where ψ ∗ satisfies W λ (θ) = F λ (ψ ∗ , θ). N. Papadakis Wasserstein Generative Models for Texture Synthesis 25 / 54
Assume that c is C 1 . Then for any λ ⩾ 0, and any θ, θ′ ∈ Ω, |W λ (θ) − W λ (θ′ )| ⩽ ∥c ′ ∥∞ E[∥g(θ, Z ) − g(θ′ , Z )∥]. Theorem Let λ > 0. Assume that c is C 1 and g satisfies (H). Then W λ is locally Lipschitz and thus differentiable a.e.. For almost any θ, ∇θ W λ (θ) = ∇θ F λ (ψ ∗ , θ) with ψ ∗ such that W λ (θ) = F λ (ψ ∗ , θ) NB: One cannot expect more regularity in W λ than there is in the ground cost c or the generator g N. Papadakis Wasserstein Generative Models for Texture Synthesis 26 / 54
ν = Jj=1 νj δyj assume c is C 1 . Let θ ∈ Θ such that ∂θ g(θ, Z ) exists almost surely and such that g satisfies (H) at θ. J [ Lj (ψ). Let also ψ ∈ RJ such that, almost surely, g(θ, Z ) ∈ j=1 Then h i ∇θ F (ψ, θ) = E (∂θ g(θ, Z ))T ∇ψ c (g(θ, Z )) . S • Fourth assumption: µθ X \ y ∈Y Lψ (y ) = 0 • If µθ (Y) = 0, deal with lipschitz costs (c(x) = ||x − y ||) → Does not require regularization N. Papadakis Wasserstein Generative Models for Texture Synthesis 29 / 54
ν = Jj=1 νj δyj assume c is C 1 . Let θ ∈ Θ such that ∂θ g(θ, Z ) exists almost surely and such that g satisfies (H) at θ. J [ Lj (ψ). Let also ψ ∈ RJ such that, almost surely, g(θ, Z ) ∈ j=1 Then h i ∇θ F (ψ, θ) = E (∂θ g(θ, Z ))T ∇ψ c (g(θ, Z )) . S • Fourth assumption: µθ X \ y ∈Y Lψ (y ) = 0 • If µθ (Y) = 0, deal with lipschitz costs (c(x) = ||x − y ||) → Does not require regularization N. Papadakis Wasserstein Generative Models for Texture Synthesis 29 / 54
ν = Jj=1 νj δyj assume c is C 1 . Let θ ∈ Θ such that ∂θ g(θ, Z ) exists almost surely and such that g satisfies (H) at θ. J [ Lj (ψ). Let also ψ ∈ RJ such that, almost surely, g(θ, Z ) ∈ j=1 Then h i ∇θ F (ψ, θ) = E (∂θ g(θ, Z ))T ∇ψ c (g(θ, Z )) . S • Fourth assumption: µθ X \ y ∈Y Lψ (y ) = 0 • If µθ (Y) = 0, deal with lipschitz costs (c(x) = ||x − y ||) → Does not require regularization N. Papadakis Wasserstein Generative Models for Texture Synthesis 29 / 54
νj δyj J: size of the dataset j=1 • WGAN problem J h i X min OTλc (gθ ♯ζ, ν) = min max EZ ∼ζ ψ c,λ (gθ (Z )) + νj ψj θ θ with ψ c,λ (x) = −λ log P ψ∈RJ J j=1 exp j=1 ψj −c(x,yj ) λ νj . Alternate optimization - The problem is concave in ψ: averaged stochastic gradient ascent to evaluate {ψj }Jj=1 - ADAM step on θ N. Papadakis Wasserstein Generative Models for Texture Synthesis 31 / 54
n 1X µu = δPi u n i=1 where Pi is the linear operator extracting the i-th patch • Given a target image v search an image u that solves min OTc (µu , µv ) u " No generator here, we just optimize pixel values of u: discrete OT N. Papadakis Wasserstein Generative Models for Texture Synthesis 35 / 54
n 1X µu = δPi u n i=1 where Pi is the linear operator extracting the i-th patch • Given a target image v search an image u that solves min OTc (µu , µv ) u " No generator here, we just optimize pixel values of u: discrete OT N. Papadakis Wasserstein Generative Models for Texture Synthesis 35 / 54
on min OTc (µu , µv ) = min maxm F (ψ, u) u where F (ψ, u) = n1 u ψ∈R Pn 1 Pm c i=1 ψ (Pi u) + m j=1 ψj • At fixed u, maxψ F (ψ, u) is a concave maximization problem with bounded subgradients −→ allows for (stochastic) subgradient ascent. √t) −→ convergence guarantee on ψ in O( log t Alternate Optimization Initialize u 0 . For k = 0, . . . , K − 1 ψ k ≈ argmaxψ F (ψ, u k ) u k +1 = u k − η∇u F (ψ k , u k ) N. Papadakis (subgradient ascent) (gradient descent) Wasserstein Generative Models for Texture Synthesis 36 / 54
Rm . Assume that for all i = 1, . . . , n, we can uniquely define σ(i) = argmin1⩽j⩽m c(Pi u, Pj v ) − ψj . Then F (ψ, ·) is differentiable at u, and n 1X T ∇u F (ψ, u) = Pi ∂x c(Pi u, Pσ(i) v ) n i=1 • If c(x, y ) = 12 ∥x − y ∥22 and η = α sn2 , the image update is u k +1 = (1 − α)u k + αv k n vk = 1 X T Pi Pσk (i) v s2 i=1 k σ (i) = argminj 1 ∥Pi u k − Pj v ∥2 − ψjk 2 • [Kwatra et al. ’05] :Wasserstein ψ = 0 Generative Models for Texture Synthesis N. Papadakis 37 / 54
L, Sℓ u is a down-sampling of u on a grid 2ℓ−1 Z2 min u L X OTc (µSℓ u , µSℓ v ) = min u ℓ=1 L X ℓ=1 max F (ψℓ , Sℓ u) ψℓ Algorithm 1: Multi-resolution Image Optimization Initialize u 0 For k = 0, . . . , K − 1 For ℓ = 1, . . . , L • ψℓk ≈ argmaxψ F (ψ, Sℓ u k ) (subgradient ascent) PL • One step of ADAM algorithm on minu ℓ=1 F (ψℓ , Sℓ u) N. Papadakis Wasserstein Generative Models for Texture Synthesis 39 / 54
et al. ’15]: min ||Gl (u) − Gl (v )||2 , u where Gl are Gram matrices of VGG features at scale l Idea study the following cases Patch distributions and Gram loss → does not work Patch distribution and OT loss → our algorithm VGG feature distribution and Gram loss → [Gatys et al. ’15] VGG feature distribution and OT loss → extension of our method N. Papadakis Wasserstein Generative Models for Texture Synthesis 46 / 54
δPi u discrete → Discrete OT ✗ Optimization for each new image • Generative model: µθ = n1 Pn i=1 (Pi ◦ gθ )♯ζ continuous → Semi-discrete OT ✓ Learn a generator gθ once for all N. Papadakis Wasserstein Generative Models for Texture Synthesis 49 / 54
u by the output gθ (Z ) of a convolutional neural network. • µu by the patch distribution µθ of gθ (Z ). L X max E[F (ψℓ , Sℓ gθ (Z ))]. New loss function min θ ℓ=1 ψℓ Algorithm 2: Multi-resolution Generative Network Optimization Initialize θ For k = 0, . . . , K − 1, For ℓ = 1, . . . , L, • ψℓk ≈ argmaxψ E[F (ψ, Sℓ gθ (Z ))] (ASGA) • Sample z ∼ ζ and take one step of ADAM algorithm on L X min F (ψℓ , Sℓ gθ (z)) θ N. Papadakis ℓ=1 Wasserstein Generative Models for Texture Synthesis 51 / 54
et al. ’16] SinGAN [Shaham et al. ’19] PSGAN [Bergmann et al. ’17] Texto [Rabin et al. ’20] N. Papadakis Wasserstein Generative Models for Texture Synthesis 52 / 54
et al. ’16] SinGAN [Shaham et al. ’19] PSGAN [Bergmann et al. ’17] Texto [Rabin et al. ’20] N. Papadakis Wasserstein Generative Models for Texture Synthesis 52 / 54
et al. ’16] SinGAN [Shaham et al. ’19] PSGAN [Bergmann et al. ’17] Texto [Rabin et al. ’20] N. Papadakis Wasserstein Generative Models for Texture Synthesis 52 / 54
et al. ’16] SinGAN [Shaham et al. ’19] PSGAN [Bergmann et al. ’17] Texto [Rabin et al. ’20] N. Papadakis Wasserstein Generative Models for Texture Synthesis 52 / 54
Ensure existence of gradients in the semi-discrete case • Leads to an alternate optimization framework ✓that can be used for some image synthesis tasks ✗ that cannot scale (yet) to very large target measures P ERSPECTIVES : • Look for regularity results for unregularized framework • Impact of entropic regularization for image synthesis problems • Exploit parameterizations of the dual variable ψ T HANK YOU FOR YOUR ATTENTION N. Papadakis Wasserstein Generative Models for Texture Synthesis 54 / 54
of pages correctl there was some unprocessed data that should have been ad final page this extra page has been added to receive it. If you rerun the document (without altering it) this surplus pag away, because LATEX now knows how many pages to expect f document.
npapadakis
chck
anatolykr
youten622
junkurihara
afroscript
signer
skoyamalab
microbaseinc
reokashiwa
gpeyre
yuukit
stellarcraft
kimpetersen
garrettdimon
tenderlove
coloredviolet
addyosmani
shpigford
mraible
tessaabrams
hatefulcrawdad
dugsong
stephenakadiri
rasagy
![Generative models Variational Auto-encoder [Kingma et al. ’13] • Decoder](https://files.speakerdeck.com/presentations/65e34ed075344cc89af80337031c9fcf/slide_5.jpg){kind=link}
![Formulations Continuous Discrete Semi-discrete [Benamou - Brenier ’00] [Cuturi ’13]](https://files.speakerdeck.com/presentations/65e34ed075344cc89af80337031c9fcf/slide_12.jpg){kind=link}
![Formulations Continuous Discrete Semi-discrete [Benamou - Brenier ’00] [Cuturi ’13]](https://files.speakerdeck.com/presentations/65e34ed075344cc89af80337031c9fcf/slide_13.jpg){kind=link}
![Formulations Continuous Discrete Semi-discrete [Benamou - Brenier ’00] [Cuturi ’13]](https://files.speakerdeck.com/presentations/65e34ed075344cc89af80337031c9fcf/slide_14.jpg){kind=link}
![Formulations Continuous Discrete Semi-discrete [Benamou - Brenier ’00] [Cuturi ’13]](https://files.speakerdeck.com/presentations/65e34ed075344cc89af80337031c9fcf/slide_15.jpg){kind=link}
![Formulations Continuous Discrete Semi-discrete [Benamou - Brenier ’00] [Cuturi ’13]](https://files.speakerdeck.com/presentations/65e34ed075344cc89af80337031c9fcf/slide_16.jpg){kind=link}
![Formulations Continuous Discrete Semi-discrete [Benamou - Brenier ’00] [Cuturi ’13]](https://files.speakerdeck.com/presentations/65e34ed075344cc89af80337031c9fcf/slide_17.jpg){kind=link}
![Formulations Continuous Discrete Semi-discrete [Benamou - Brenier ’00] [Cuturi ’13]](https://files.speakerdeck.com/presentations/65e34ed075344cc89af80337031c9fcf/slide_18.jpg){kind=link}
![Formulations Continuous Discrete Semi-discrete [Benamou - Brenier ’00] [Cuturi ’13]](https://files.speakerdeck.com/presentations/65e34ed075344cc89af80337031c9fcf/slide_19.jpg){kind=link}
![What convex optimization tells us Proposition [Santambrogio’s textbook ’15] Assume](https://files.speakerdeck.com/presentations/65e34ed075344cc89af80337031c9fcf/slide_29.jpg){kind=link}
![What convex optimization tells us Proposition [Santambrogio’s textbook ’15] Assume](https://files.speakerdeck.com/presentations/65e34ed075344cc89af80337031c9fcf/slide_30.jpg){kind=link}
![What convex optimization tells us Proposition [Santambrogio’s textbook ’15] Assume](https://files.speakerdeck.com/presentations/65e34ed075344cc89af80337031c9fcf/slide_31.jpg){kind=link}
![What convex optimization tells us Proposition [Santambrogio’s textbook ’15] Assume](https://files.speakerdeck.com/presentations/65e34ed075344cc89af80337031c9fcf/slide_32.jpg){kind=link}
![Regularized Optimal Transport Definition [Genevay et al. ’19] For λ](https://files.speakerdeck.com/presentations/65e34ed075344cc89af80337031c9fcf/slide_41.jpg){kind=link}
![Visual comparisons Original N. Papadakis Our [Kwatra ’05] [Gatys ’15]](https://files.speakerdeck.com/presentations/65e34ed075344cc89af80337031c9fcf/slide_73.jpg){kind=link}
![Link with [Gatys et al. ’15] Texture synthesis from [Gatys](https://files.speakerdeck.com/presentations/65e34ed075344cc89af80337031c9fcf/slide_74.jpg){kind=link}
![Blur artifacts with patch-based methods [Leclaire and Rabin ’19] -](https://files.speakerdeck.com/presentations/65e34ed075344cc89af80337031c9fcf/slide_101.jpg){kind=link}
![Blur artifacts with patch-based methods [Leclaire and Rabin ’19] -](https://files.speakerdeck.com/presentations/65e34ed075344cc89af80337031c9fcf/slide_102.jpg){kind=link}
