Learning of Wasserstein generative models and patch based texture synthesis

Embed

Start on current slide

Slide 1

Slide 1 text

Learning of Wasserstein generative models and patch based texture synthesis Antoine Houdard, Arthur Leclaire, Nicolas Papadakis, Julien Rabin N. Papadakis Wasserstein Generative Models for Texture Synthesis 1 / 54

Slide 2

Slide 2 text

Outline 1. Introduction 2. Optimal Transport 3. Learning Wasserstein Generative Models 4. Patch-based Texture Synthesis N. Papadakis Wasserstein Generative Models for Texture Synthesis 2 / 54

Slide 3

Slide 3 text

Outline 1. Introduction 2. Optimal Transport 3. Learning Wasserstein Generative Models 4. Patch-based Texture Synthesis N. Papadakis Wasserstein Generative Models for Texture Synthesis 3 / 54

Slide 4

Slide 4 text

Generative models Popular usages: Generating numbers, clothing, bedrooms, faces... N. Papadakis Wasserstein Generative Models for Texture Synthesis 4 / 54

Slide 5

Slide 5 text

Generative models • Data {y1 , . . . , yN } sampled from Y ∼ ν • Synthetic distribution µθ = gθ ♯ζ Goal: find the best θ s.t. µθ is close in some sense to ν N. Papadakis Wasserstein Generative Models for Texture Synthesis 5 / 54

Slide 6

Slide 6 text

Generative models Variational Auto-encoder [Kingma et al. ’13] • Decoder as generative model gθ GAN [Goodfellow et al. ’14] • Discriminator dη between fake gθ (Z ) and true Y samples min max Eν [log(dη (Y ))] + Eζ [log(1 − dη (gθ (Z )))] θ η WGAN [Arjovsky et al. ’17] • Compare fake Z ∼ µθ = gθ ♯ζ and true Y ∼ ν sample distributions min D(µθ , ν) θ • Duality of Wasserstein distance D = W1 yields min max Eν [ψ(Y )] − Eζ [ψ(gθ (Z ))] θ ψ∈Lip1 • Parameterization of the dual variable ψ with dη Questions: Other Wasserstein costs? Training strategies? N. Papadakis Wasserstein Generative Models for Texture Synthesis 6 / 54

Slide 7

Slide 7 text

Patch-based Texture Synthesis → • Copy Paste [Efros and Leung, ’99] → • Iterative refinement with nearest neighbors [Kwatra, ’05] • Impose patch distribution at different scales [Gutierrez et al. ’17, Leclaire and Rabin ’19] ✗ Image composed of patches processed independently ✗ Apply the algorithm for each new synthesis N. Papadakis Wasserstein Generative Models for Texture Synthesis 7 / 54

Slide 8

Slide 8 text

Texture synthesis with Neural Networks • Gram matrix of VGG features at different scales [Gatys et al. ’15] min ||Gu − Gv ||2 u → Prescribe features of an example image v ✓ Process the whole image u and not its patches independently • Train a feedforward generative network gθ [Ulyanov et al, ’16] min Eζ ||Ggθ (Z ) − Gv ||2 θ ✓ Real time synthesis Questions: Wasserstein metric between feature distributions? Dealing with patches? N. Papadakis Wasserstein Generative Models for Texture Synthesis 8 / 54

Slide 9

Slide 9 text

Outline 1. Introduction 2. Optimal Transport 3. Learning Wasserstein Generative Models 4. Patch-based Texture Synthesis N. Papadakis Wasserstein Generative Models for Texture Synthesis 9 / 54

Slide 10

Slide 10 text

Optimal Transport (OT) • OT defines a family of distances between densities of probability • Transport a mass µ(x) onto ν(y) x y Euclidean • Define a cost c(x, y ) of mass transport between locations x and y • OT: application with mimimal global cost that transfers µ onto ν • If c(x, y ) = ||x − y ||p , Lp Wasserstein distance • Interpolation with transport map T N. Papadakis Wasserstein Generative Models for Texture Synthesis 10 / 54

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Optimal Transport (OT) • OT defines a family of distances between densities of probability • Transport a mass µ(x) onto ν(y) x y Wasserstein • Define a cost c(x, y ) of mass transport between locations x and y • OT: application with mimimal global cost that transfers µ onto ν • If c(x, y ) = ||x − y ||p , Lp Wasserstein distance • Interpolation with transport map T N. Papadakis Wasserstein Generative Models for Texture Synthesis 10 / 54

Slide 13

Slide 13 text

Formulations Continuous Discrete Semi-discrete [Benamou - Brenier ’00] [Cuturi ’13] [Mérigot ’11, et al. ’17] N. Papadakis Wasserstein Generative Models for Texture Synthesis 11 / 54

Slide 14

Slide 14 text

Formulations Continuous Discrete Semi-discrete [Benamou - Brenier ’00] [Cuturi ’13] [Mérigot ’11, et al. ’17] Interpolation µt between images N. Papadakis Wasserstein Generative Models for Texture Synthesis 11 / 54

Slide 15

Slide 15 text

Formulations Continuous Discrete Semi-discrete [Benamou - Brenier ’00] [Cuturi ’13] [Mérigot ’11, et al. ’17] Interpolation µt between images µ N. Papadakis ν Wasserstein Generative Models for Texture Synthesis 11 / 54

Slide 16

Slide 16 text

Formulations Continuous Discrete Semi-discrete [Benamou - Brenier ’00] [Cuturi ’13] [Mérigot ’11, et al. ’17] Transfer of colors between images N. Papadakis Wasserstein Generative Models for Texture Synthesis 11 / 54

Slide 17

Slide 17 text

Formulations Continuous Discrete Semi-discrete [Benamou - Brenier ’00] [Cuturi ’13] [Mérigot ’11, et al. ’17] Wasserstein GAN N. Papadakis Wasserstein Generative Models for Texture Synthesis 11 / 54

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Slide 20

Slide 20 text

Formulations Continuous Discrete Semi-discrete [Benamou - Brenier ’00] [Cuturi ’13] [Mérigot ’11, et al. ’17] What’s next • More on the (semi-discrete formulation) of the optimal transport cost OT(µ, ν) • Differentiability and regularization of the cost min OT(µθ , ν) θ • Application to patch-based texture synthesis N. Papadakis Wasserstein Generative Models for Texture Synthesis 11 / 54

Slide 21

Slide 21 text

Optimal Transport cost • Continuous cost function c : Rd × Rd → R • µ, ν probability measures supported on compacts X , Y ⊂ Rd , let Z c(x, y )dπ(x, y ) OTc (µ, ν) = min π∈Π(µ,ν) Π(µ, ν) : set of probability measures on X × Y with marginals µ, ν. Theorem [Villani ’03, Santambrogio ’15] Strong duality holds i.e. Z OTc (µ, ν) = max φ,ψ Z φdµ + X ψdν Y where max is taken on all functions φ ∈ L1 (µ), ψ ∈ L1 (ν) such that φ(x) + ψ(y ) ⩽ c(x, y ) dµ(x) a.e., dν(y ) a.e. N. Papadakis Wasserstein Generative Models for Texture Synthesis 12 / 54

Slide 22

Slide 22 text

c-transforms and semi-dual formulation c-transforms c φ (y ) = min [c(x, y ) − φ(x)] x∈X ψ c (x) = min [c(x, y ) − ψ(y )] y ∈Y Semi-dual Z OTc (µ, ν) = max φ,ψ Z φdµ + X Z ψdν = max Y φ∈C (X ) ZX = max ψ∈C (Y) N. Papadakis Z φc (y )dν(y ) φ(x)dµ(x) + c ZY ψ (x)dµ(x) + X Wasserstein Generative Models for Texture Synthesis ψ(y )dν(y ) Y 13 / 54

Slide 23

Slide 23 text

c-transforms and semi-dual formulation c-transforms c φ (y ) = min [c(x, y ) − φ(x)] x∈X ψ c (x) = min [c(x, y ) − ψ(y )] y ∈Y Semi-dual Z OTc (µ, ν) = max φ,ψ Z φdµ + X Z ψdν = max Y φ∈C (X ) Z φc (y )dν(y ) φ(x)dµ(x) + ZX = max ψ∈C (Y) c ZY ψ (x)dµ(x) + X ψ(y )dν(y ) Y • c-transforms inherit regularity from c N. Papadakis Wasserstein Generative Models for Texture Synthesis 13 / 54

Slide 24

Slide 24 text

c-transforms and semi-dual formulation c-transforms c ψ c (x) = min [c(x, y ) − ψ(y )] φ (y ) = min [c(x, y ) − φ(x)] y ∈Y x∈X Semi-dual Z OTc (µ, ν) = max φ,ψ Z Z φdµ + X ψdν = max φ∈C (X ) Y ZX = max ψ∈C (Y) Z φc (y )dν(y ) φ(x)dµ(x) + ZY c ψ (x)dµ(x) + X ψ(y )dν(y ) Y • If c(x, y ) = ||x − y ||, then ψ c = −ψ and ψ is 1-lipschitz [Kantorovich and Rubinstein, ’58] Z ψ∈Lip1 N. Papadakis Z ψ(y )dν(y ) − OTc (µ, ν) = max Y ψ(x)dµ(x) X Wasserstein Generative Models for Texture Synthesis 13 / 54

Slide 25

Slide 25 text

c-transforms and semi-dual formulation c-transforms c ψ c (x) = min [c(x, y ) − ψ(y )] φ (y ) = min [c(x, y ) − φ(x)] y ∈Y x∈X Semi-dual Z OTc (µ, ν) = max φ,ψ Z φdµ + X Z ψdν = max φ∈C (X ) Y ZX = max ψ∈C (Y) • For a discrete ν = X φc (y )dν(y ) ZY c ψ (x)dµ(x) + X ψ(y )dν(y ) Y J j=1 νj δyj and ψj = ψ(yj ) ψ c (x) = min j∈{1,··· ,J} c(x, yj ) − ψj Z OTc (µ, ν) = max {ψj }Jj=1 N. Papadakis Z φ(x)dµ(x) + X ψ c (x)dµ(x) + J X νj ψj j=1 Wasserstein Generative Models for Texture Synthesis 13 / 54

Slide 26

Slide 26 text

Semi-discrete semi-dual formulation • Semi-discrete cost Z OTc (µ, ν) = max {ψj }Jj=1 c ψ (x)dµ(x) + X J X νj ψj j=1 • Without equally distant indexes j, the argmin in ψ c defines a map Tψ (x) = argmin{yj }J c(x, yj ) − ψj j=1 → “Biased” nearest neighbor matching • Preimages of Tψ are called Laguerre cells: Lj (ψ) = {x | ∀k ̸= j, c(x, yj ) − ψj < c(x, yk ) − ψk }. T0 N. Papadakis Tψ Wasserstein Generative Models for Texture Synthesis 14 / 54

Slide 27

Slide 27 text

Some questions • Design some algorithms to solve Z inf OTc (µθ , ν) = inf max θ θ ψ c ψ dµθ + Z ψdν • Is the loss function OTc (µθ , ν) regular? • If not, what kind of problems happen? • Do these problems appear in discrete/semi-discrete cases? • Does this scale up in order to address image synthesis problems? N. Papadakis Wasserstein Generative Models for Texture Synthesis 15 / 54

Slide 28

Slide 28 text

Outline 1. Introduction 2. Optimal Transport 3. Learning Wasserstein Generative Models 4. Patch-based Texture Synthesis N. Papadakis Wasserstein Generative Models for Texture Synthesis 16 / 54

Slide 29

Slide 29 text

Related GAN works Z inf OTc (µθ , ν) = inf max θ θ ψ ψ c dµθ + Z ψdν [Goodfellow et al. ’14] GAN (Jensen-Shannon divergence) [Arjovsky et al. ’17] Wasserstein GAN (Wasserstein distance with L1 -cost) [Gulrajani et al. ’17] WGAN-GP: Wasserstein GAN with Gradient Penalty [Genevay et al. ’18] Generative models with Sinkhorn divergences [Salimans et al. ’18] Improving GANs using Optimal transport [Liu et al. ’18] WGAN-TS (for Two Steps) [Chen et al. ’19] Semi-discrete Wasserstein generative network training Differential properties of OT [Burger et al. ’12] Wasserstein distance and regularized densitiy [Cuturi and Peyré ’15] Gradient of regularized Wasserstein distance [Cazelles et al. ’19] Proof of differentiability in both previous settings [Degournay et al. ’19] Differentiation w.r.t. the discrete target measure N. Papadakis Wasserstein Generative Models for Texture Synthesis 17 / 54

Slide 30

Slide 30 text

What convex optimization tells us Proposition [Santambrogio’s textbook ’15] Assume c continuous, X , Y ⊂ Rd and fix ν • µ 7→ OTc (µ, ν) is convex • For all subgradient φ ∈ ∂µ OTc (µ, ν) Z Z OTc (µ, ν) = φdµ + φc dν Hence Z OTc (µ + χ, ν) ⩾ OTc (µ, ν) + φdχ • If φ is unique up to additive constants, then one can show Gateaux-differentiability at (µ, ν) NB: Extension to entropy-regularized optimal transport [Feydy et al. ’18] Sufficient condition [Santambrogio ’15] c is C 1 and Supp(µ) (or ν) is the closure of a bounded connected open set Does not include c(x, y ) = ||x − y || N. Papadakis Wasserstein Generative Models for Texture Synthesis 18 / 54

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Slide 33

Slide 33 text

Slide 34

Slide 34 text

WGAN problem Given a generator µθ = gθ ♯ζ, solve Z inf OTc (µθ , ν) = inf max θ For F (ψ, θ) = θ ψ c ψ dµθ + Z ψdν c X ψ (x)dµθ (x) + Y ψ(y )dν(y ) we have R R W (θ) := OTc (µθ , ν) = max F (ψ, θ) ψ Potential ψ acts as a discriminator between µθ and ν N. Papadakis Wasserstein Generative Models for Texture Synthesis 19 / 54

Slide 35

Slide 35 text

WGAN problem Given a generator µθ = gθ ♯ζ, solve Z inf OTc (µθ , ν) = inf max θ For F (ψ, θ) = θ ψ Z c ψ dµθ + ψdν c X ψ (x)dµθ (x) + Y ψ(y )dν(y ) we have R R W (θ) := OTc (µθ , ν) = max F (ψ, θ) ψ Theorem [Arjovsky et al., 2017] Let θ0 and ψ0∗ satisfying W (θ0 ) = F (ψ0∗ , θ0 ). If W and θ 7→ F (ψ0∗ , θ) are both differentiable at θ0 , then ∇W (θ0 ) = ∇θ F (ψ0∗ , θ0 ) (Grad-OT) " There are cases where no such couple (ψ ∗ , θ0 ) exists 0 N. Papadakis Wasserstein Generative Models for Texture Synthesis 19 / 54

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Slide 38

Slide 38 text

Slide 39

Slide 39 text

A telling counter-example Proposition Let µθ = δθ with θ ∈ Rd , and let ν = 21 δy1 + 21 δy2 with y1 , y2 ∈ Rd distinct. Let c(x, y ) = ∥x − y ∥pp , p > 1. Then • θ 7→ W (θ) is differentiable everywhere. 2 and any ψ0∗ ∈ argmaxψ F (ψ, θ0 ), θ 7→ F (ψ0∗ , θ) is not • For θ0 ̸= y1 +y 2 differentiable at θ0 . 2 Hence (Grad-OT) relation does not hold (except for θ0 = y1 +y 2 ). Proof • F (ψ, θ) = ψ c (θ) + P2 ψ1 +ψ2 1 j=1 2 ψj = mini=1,2 [c(θ, yj ) − ψi ] + 2 Fix θ0 and ψ0∗ , then (ψ0∗ )1 − (ψ0∗ )2 = c(θ0 , y1 ) − c(θ0 , y2 ), and ( c(θ, y1 ) + 12 c(θ0 , y2 ) − c(θ0 , y1 ) if θ ∈ L1 (ψ0∗ ) ∗ F (ψ0 , θ) = c(θ, y2 ) + 12 c(θ0 , y1 ) − c(θ0 , y2 ) if θ ∈ L2 (ψ0∗ ) F (ψ0∗ , ·) not differentiable at the boundary between L1 (ψ0∗ ) and L2 (ψ0∗ ) N. Papadakis Wasserstein Generative Models for Texture Synthesis 20 / 54

Slide 40

Slide 40 text

Consequence: Instabilities in training • Iterative algorithms for solving inf W (θ) θ need an estimation of the gradient. • For the L2 -cost ∇W (θ) = θ − y1 + θ − y2 is estimated by ( θ − y1 ∇θ F (ψ, θ) = θ − y2 if θ ∈ L1 (ψ) if θ ∈ L2 (ψ) Solution 1 Regularization of optimal transport 2 Assumption on the generator N. Papadakis Wasserstein Generative Models for Texture Synthesis 21 / 54

Slide 41

Slide 41 text

Slide 42

Slide 42 text

Regularized Optimal Transport Definition [Genevay et al. ’19] For λ > 0, the regularized OT cost is defined by Z λ c(x, y )dπ(x, y ) + λKL(π|µ ⊗ ν) OTc (µ, ν) = inf π∈Π(µ,ν) where KL is the Kullback-Leibler divergence: (R dπ(x,y ) dπ log dµ(x)dν(y ) dπ(x, y ) if dµdν exists KL(π|µ ⊗ ν) = . +∞ otherwise N. Papadakis Wasserstein Generative Models for Texture Synthesis 22 / 54

Slide 43

Slide 43 text

Semi-dual Regularized Problem Proposition Assume that c ∈ L∞ (X × Y), then Z Z OTλc (µ, ν) = max ψ c,λ (x)dµ(x) + ψ(y )dν(y ) ψ∈L∞ (Y) X Y where ψ c,λ (x) = Soft min c(x, yj ) − ψj j∈{1,··· ,J} Z ψ(y ) − c(x, y ) dν(y ) exp = −λ log λ Y Theorem [Genevay ’19, Chizat et al. ’19] For c ∈ L∞ (X × Y), the semi-dual problem admits a solution ψ ∗ ∈ L∞ (ν) which is unique ν − a.e. up to an additive constant NB: Solutions are characterized by the fixed point equation (ψ c,λ )c,λ = ψ N. Papadakis Wasserstein Generative Models for Texture Synthesis 23 / 54

Slide 44

Slide 44 text

Regularity of the full problem • µθ : distribution of g(θ, Z ) • Z : r.v. in Z ⊂ Rp with distribution ζ min OTλc (µθ , ν) = min max E[ψ c,λ (g(θ, Z ))] + θ θ ψ∈L∞ (Y) | N. Papadakis {z :=F λ (ψ,θ) Wasserstein Generative Models for Texture Synthesis Z Y ψdν } 24 / 54

Slide 45

Slide 45 text

Regularity of the full problem • µθ : distribution of g(θ, Z ) • Z : r.v. in Z ⊂ Rp with distribution ζ min OTλc (µθ , ν) = min max E[ψ c,λ (g(θ, Z ))] + θ θ ψ∈L∞ (Y) | Z Y {z :=F λ (ψ,θ) ψdν } Hypothesis (H) There exists L : Θ × Z → R+ such that, for any θ ∈ Θ, there is a neighborhood Vθ of θ such that ∀θ′ ∈ Vθ Z − a.s., ∥g(θ, Z ) − g(θ′ , Z )∥ ⩽ L(θ, Z )∥θ − θ′ ∥ with E[L(θ, Z )] < ∞. Proposition Let λ > 0. Assume that c is C 1 , and g satisfies (H). For any θ0 ∈ Θ and any ψ ∈ L∞ (Y), θ 7→ F λ (ψ, θ) is differentiable at θ0 h i T ∇θ F λ (ψ, θ0 ) = E (∂θ g(θ0 , Z )) ∇ψ c,λ (g(θ0 , Z )) If g is C 1 , then so is F λ (ψ, ·) N. Papadakis Wasserstein Generative Models for Texture Synthesis 24 / 54

Slide 46

Slide 46 text

Slide 47

Slide 47 text

Gradient of the regularized Wasserstein cost Theorem Let λ > 0. Assume that c is C 1 , g is C 1 and satisfies (H). Then W λ : θ 7→ OTλc (µθ , ν) is C 1 , and for any θ ∈ Θ, h i ∇θ W λ (θ) = ∇θ F λ (ψ ∗ , θ) = E (∂θ g(θ, Z ))T ∇ψ ∗,c,λ (g(θ, Z )) where ψ ∗ satisfies W λ (θ) = F λ (ψ ∗ , θ). N. Papadakis Wasserstein Generative Models for Texture Synthesis 25 / 54

Slide 48

Slide 48 text

If the generator g is not C 1 ? Lemma Assume that c is C 1 . Then for any λ ⩾ 0, and any θ, θ′ ∈ Ω, |W λ (θ) − W λ (θ′ )| ⩽ ∥c ′ ∥∞ E[∥g(θ, Z ) − g(θ′ , Z )∥]. Theorem Let λ > 0. Assume that c is C 1 and g satisfies (H). Then W λ is locally Lipschitz and thus differentiable a.e.. For almost any θ, ∇θ W λ (θ) = ∇θ F λ (ψ ∗ , θ) with ψ ∗ such that W λ (θ) = F λ (ψ ∗ , θ) NB: One cannot expect more regularity in W λ than there is in the ground cost c or the generator g N. Papadakis Wasserstein Generative Models for Texture Synthesis 26 / 54

Slide 49

Slide 49 text

Back to the counter-example OTc N. Papadakis OTλc Wasserstein Generative Models for Texture Synthesis 27 / 54

Slide 50

Slide 50 text

In the semi-discrete case N. Papadakis Wasserstein Generative Models for Texture Synthesis 28 / 54

Slide 51

Slide 51 text

In the semi-discrete case N. Papadakis Wasserstein Generative Models for Texture Synthesis 28 / 54

Slide 52

Slide 52 text

In the unregularized semi-discrete case P Theorem For Y finite, ν = Jj=1 νj δyj assume c is C 1 . Let θ ∈ Θ such that ∂θ g(θ, Z ) exists almost surely and such that g satisfies (H) at θ. J [ Lj (ψ). Let also ψ ∈ RJ such that, almost surely, g(θ, Z ) ∈ j=1 Then h i ∇θ F (ψ, θ) = E (∂θ g(θ, Z ))T ∇ψ c (g(θ, Z )) . S • Fourth assumption: µθ X \ y ∈Y Lψ (y ) = 0 • If µθ (Y) = 0, deal with lipschitz costs (c(x) = ||x − y ||) → Does not require regularization N. Papadakis Wasserstein Generative Models for Texture Synthesis 29 / 54

Slide 53

Slide 53 text

Slide 54

Slide 54 text

Slide 55

Slide 55 text

Limitation N. Papadakis Wasserstein Generative Models for Texture Synthesis 30 / 54

Slide 56

Slide 56 text

Limitation N. Papadakis Wasserstein Generative Models for Texture Synthesis 30 / 54

Slide 57

Slide 57 text

Limitation N. Papadakis Wasserstein Generative Models for Texture Synthesis 30 / 54

Slide 58

Slide 58 text

Limitation N. Papadakis Wasserstein Generative Models for Texture Synthesis 30 / 54

Slide 59

Slide 59 text

In practice • Discrete target measure ν = J X νj δyj J: size of the dataset j=1 • WGAN problem J h i X min OTλc (gθ ♯ζ, ν) = min max EZ ∼ζ ψ c,λ (gθ (Z )) + νj ψj θ θ with ψ c,λ (x) = −λ log P ψ∈RJ J j=1 exp j=1 ψj −c(x,yj ) λ νj . Alternate optimization - The problem is concave in ψ: averaged stochastic gradient ascent to evaluate {ψj }Jj=1 - ADAM step on θ N. Papadakis Wasserstein Generative Models for Texture Synthesis 31 / 54

Slide 60

Slide 60 text

Generation of MNIST digits λ = 0.001 λ = 0.01 λ = 0.1 N. Papadakis Wasserstein Generative Models for Texture Synthesis 32 / 54

Slide 61

Slide 61 text

Outline 1. Introduction 2. Optimal Transport 3. Learning Wasserstein Generative Models 4. Patch-based Texture Synthesis N. Papadakis Wasserstein Generative Models for Texture Synthesis 33 / 54

Slide 62

Slide 62 text

Patch-based Texture Synthesis Patches seen as vector of Rs×s N. Papadakis Wasserstein Generative Models for Texture Synthesis 34 / 54

Slide 63

Slide 63 text

Patch-based texture synthesis • Patch distribution of an image u n 1X µu = δPi u n i=1 where Pi is the linear operator extracting the i-th patch • Given a target image v search an image u that solves min OTc (µu , µv ) u " No generator here, we just optimize pixel values of u: discrete OT N. Papadakis Wasserstein Generative Models for Texture Synthesis 35 / 54

Slide 64

Slide 64 text

Slide 65

Slide 65 text

Minimizing OT cost w.r.t. image u • Alternate optimization scheme on min OTc (µu , µv ) = min maxm F (ψ, u) u where F (ψ, u) = n1 u ψ∈R Pn 1 Pm c i=1 ψ (Pi u) + m j=1 ψj • At fixed u, maxψ F (ψ, u) is a concave maximization problem with bounded subgradients −→ allows for (stochastic) subgradient ascent. √t) −→ convergence guarantee on ψ in O( log t Alternate Optimization Initialize u 0 . For k = 0, . . . , K − 1 ψ k ≈ argmaxψ F (ψ, u k ) u k +1 = u k − η∇u F (ψ k , u k ) N. Papadakis (subgradient ascent) (gradient descent) Wasserstein Generative Models for Texture Synthesis 36 / 54

Slide 66

Slide 66 text

Relation with iterated nearest neighbor projections Proposition Let ψ ∈ Rm . Assume that for all i = 1, . . . , n, we can uniquely define σ(i) = argmin1⩽j⩽m c(Pi u, Pj v ) − ψj . Then F (ψ, ·) is differentiable at u, and n 1X T ∇u F (ψ, u) = Pi ∂x c(Pi u, Pσ(i) v ) n i=1 • If c(x, y ) = 12 ∥x − y ∥22 and η = α sn2 , the image update is u k +1 = (1 − α)u k + αv k n vk = 1 X T Pi Pσk (i) v s2 i=1 k σ (i) = argminj 1 ∥Pi u k − Pj v ∥2 − ψjk 2 • [Kwatra et al. ’05] :Wasserstein ψ = 0 Generative Models for Texture Synthesis N. Papadakis 37 / 54

Slide 67

Slide 67 text

Illustration N. Papadakis Wasserstein Generative Models for Texture Synthesis 38 / 54

Slide 68

Slide 68 text

Multi-resolution Algorithm For ℓ = 1, . . . , L, Sℓ u is a down-sampling of u on a grid 2ℓ−1 Z2 min u L X OTc (µSℓ u , µSℓ v ) = min u ℓ=1 L X ℓ=1 max F (ψℓ , Sℓ u) ψℓ Algorithm 1: Multi-resolution Image Optimization Initialize u 0 For k = 0, . . . , K − 1 For ℓ = 1, . . . , L • ψℓk ≈ argmaxψ F (ψ, Sℓ u k ) (subgradient ascent) PL • One step of ADAM algorithm on minu ℓ=1 F (ψℓ , Sℓ u) N. Papadakis Wasserstein Generative Models for Texture Synthesis 39 / 54

Slide 69

Slide 69 text

Results of Image Optimization Exemplar N. Papadakis Initialization Synthesis Loss Wasserstein Generative Models for Texture Synthesis 40 / 54

Slide 70

Slide 70 text

Synthesis with Image Optimization (128 × 128) → (256 × 256) N. Papadakis Wasserstein Generative Models for Texture Synthesis 41 / 54

Slide 71

Slide 71 text

Synthesis with Image Optimization (128 × 128) → (256 × 256) N. Papadakis Wasserstein Generative Models for Texture Synthesis 42 / 54

Slide 72

Slide 72 text

Synthesis with Image Optimization (256 × 256) → (256 × 512) N. Papadakis Wasserstein Generative Models for Texture Synthesis 43 / 54

Slide 73

Slide 73 text

Inpainting N. Papadakis Wasserstein Generative Models for Texture Synthesis 44 / 54

Slide 74

Slide 74 text

Visual comparisons Original N. Papadakis Our [Kwatra ’05] [Gatys ’15] Original Our [Kwatra ’05] Wasserstein Generative Models for Texture Synthesis [Gatys ’15] 45 / 54

Slide 75

Slide 75 text

Link with [Gatys et al. ’15] Texture synthesis from [Gatys et al. ’15]: min ||Gl (u) − Gl (v )||2 , u where Gl are Gram matrices of VGG features at scale l Idea study the following cases Patch distributions and Gram loss → does not work Patch distribution and OT loss → our algorithm VGG feature distribution and Gram loss → [Gatys et al. ’15] VGG feature distribution and OT loss → extension of our method N. Papadakis Wasserstein Generative Models for Texture Synthesis 46 / 54

Slide 76

Slide 76 text

Texture barycenter N. Papadakis Wasserstein Generative Models for Texture Synthesis 47 / 54

Slide 77

Slide 77 text

Style transfer N. Papadakis Wasserstein Generative Models for Texture Synthesis 48 / 54

Slide 78

Slide 78 text

Style transfer N. Papadakis Wasserstein Generative Models for Texture Synthesis 48 / 54

Slide 79

Slide 79 text

Style transfer N. Papadakis Wasserstein Generative Models for Texture Synthesis 48 / 54

Slide 80

Slide 80 text

Style transfer N. Papadakis Wasserstein Generative Models for Texture Synthesis 48 / 54

Slide 81

Slide 81 text

Style transfer N. Papadakis Wasserstein Generative Models for Texture Synthesis 48 / 54

Slide 82

Slide 82 text

Generative Model • Image optimisation: µu = n1 Pn i=1 δPi u discrete → Discrete OT ✗ Optimization for each new image N. Papadakis Wasserstein Generative Models for Texture Synthesis 49 / 54

Slide 83

Slide 83 text

Generative Model • Image optimisation: µu = n1 Pn i=1 δPi u discrete → Discrete OT ✗ Optimization for each new image • Generative model: µθ = n1 Pn i=1 (Pi ◦ gθ )♯ζ continuous → Semi-discrete OT ✓ Learn a generator gθ once for all N. Papadakis Wasserstein Generative Models for Texture Synthesis 49 / 54

Slide 84

Slide 84 text

Generative Networks Texture Networks Generate images of arbitrary size [Ulyanov et al., 2016] N. Papadakis Wasserstein Generative Models for Texture Synthesis 50 / 54

Slide 85

Slide 85 text

Learn a generative network from an exemplar texture Replace • u by the output gθ (Z ) of a convolutional neural network. • µu by the patch distribution µθ of gθ (Z ). L X max E[F (ψℓ , Sℓ gθ (Z ))]. New loss function min θ ℓ=1 ψℓ Algorithm 2: Multi-resolution Generative Network Optimization Initialize θ For k = 0, . . . , K − 1, For ℓ = 1, . . . , L, • ψℓk ≈ argmaxψ E[F (ψ, Sℓ gθ (Z ))] (ASGA) • Sample z ∼ ζ and take one step of ADAM algorithm on L X min F (ψℓ , Sℓ gθ (z)) θ N. Papadakis ℓ=1 Wasserstein Generative Models for Texture Synthesis 51 / 54

Slide 86

Slide 86 text

Synthesis with learned Generative Networks Original Algo 2 TexNet [Ulyanov et al. ’16] SinGAN [Shaham et al. ’19] PSGAN [Bergmann et al. ’17] Texto [Rabin et al. ’20] N. Papadakis Wasserstein Generative Models for Texture Synthesis 52 / 54

Slide 87

Slide 87 text

Slide 88

Slide 88 text

Slide 89

Slide 89 text

Slide 90

Slide 90 text

Quantitative Results Algo 1 Algo 2 T EX N ET S IN G AN P S G AN T EX TO SIFID VGG Gram norm Muti-scale patch OT Avg 0.43 0.02 0.08 0.71 0.31 1.13 0.06 0.18 1.82 0.80 0.11 0.08 0.18 0.17 0.14 0.93 0.10 0.17 0.37 0.39 0.27 0.91 1.14 0.49 0.70 1.22 0.07 0.18 1.67 0.79 Avg 122 6 141 865 283 233 19 151 922 331 218 9 54 190 118 299 8 207 394 227 224 512 753 1366 714 260 24 152 1030 367 Avg 0.45 0.15 0.09 0.69 0.35 0.48 0.16 0.10 0.78 0.38 0.65 0.24 0.17 1.22 0.57 0.54 0.24 0.26 0.79 0.46 0.68 0.43 0.34 1.19 0.66 0.49 0.16 0.11 0.75 0.38 Comparisons based on • SIFID: Single Image Fréchet Inception Distance [Shaham et al. ’19] compares responses to a trained “inception” network • VGG Gram loss [Gatys et al. ’15] compares cross-correlations of responses to the neural network VGG • Our multi-scale patch OT loss. N. Papadakis Wasserstein Generative Models for Texture Synthesis 53 / 54

Slide 91

Slide 91 text

Conclusion • Highlight of differentiability problems in WGAN problems • Ensure existence of gradients in the semi-discrete case • Leads to an alternate optimization framework ✓that can be used for some image synthesis tasks ✗ that cannot scale (yet) to very large target measures P ERSPECTIVES : • Look for regularity results for unregularized framework • Impact of entropic regularization for image synthesis problems • Exploit parameterizations of the dual variable ψ T HANK YOU FOR YOUR ATTENTION N. Papadakis Wasserstein Generative Models for Texture Synthesis 54 / 54

Slide 92

Slide 92 text

Barycenters with c(x, y ) = ||x − y || N. Papadakis Wasserstein Generative Models for Texture Synthesis 54 / 54

Slide 93

Slide 93 text

Barycenters with c(x, y ) = ||x − y || N. Papadakis Wasserstein Generative Models for Texture Synthesis 54 / 54

Slide 94

Slide 94 text

Barycenters with c(x, y ) = ||x − y || N. Papadakis Wasserstein Generative Models for Texture Synthesis 54 / 54

Slide 95

Slide 95 text

Barycenters with c(x, y ) = ||x − y || N. Papadakis Wasserstein Generative Models for Texture Synthesis 54 / 54