Adversarial Learning with Local Coordinate Coding

Generative adversarial networks (GANs) aim to generate realistic data from some prior distribution, (i.e., the input of the generator). However, such prior distribution is often independent of real data and may lose semantic information. In practice, a latent distribution can be learned to represent the semantic information, but it is hard to be used for sampling in GANs for generating data. In this paper, we exploit Local Coordinate Coding (LCC) to improve GANs. Consequently, we are able to employ a new LCC based sampling method with a local coordinate system, rather than sampling from pre-defined prior distribution. More importantly, relying on LCC, we theoretically prove that the generalization ability of GANs depends on the intrinsic dimension of the latent manifold. Moreover, we conduct extensive experiments on real-world datasets to demonstrate the effectiveness of the proposed method.

Transcript

1. Introduction Adversarial Learning with LCC Experiments Conclusions Adversarial Learning with Local Coordinate Coding Jiezhang Cao1, Yong Guo1, Qingyao Wu1, Chunhua Shen2, Junzhou Huang3, Mingkui Tan1 Published in ICML2018 1 School of Software Engineering, South China University of Technology, China 2 School of Computer Science, The University of Adelaide, Australia 3 Tencent AI Lab, China; University of Texas at Arlington, America August 2, 2018
2. Introduction Adversarial Learning with LCC Experiments Conclusions Outline 1 Introduction Generative Adversarial Networks (GANs) Local Coordinate Coding (LCC) 2 Adversarial Learning with LCC Our Motivations LCC-GANs Architecture Objective Function of LCC-GANs Generalization Analysis 3 Experiments Results on Real-world Datasets Comparisons of Representation Methods Demonstration of LCC Sampling 4 Conclusions
3. Introduction Adversarial Learning with LCC Experiments Conclusions Contents 1 Introduction Generative Adversarial Networks (GANs) Local Coordinate Coding (LCC) 2 Adversarial Learning with LCC Our Motivations LCC-GANs Architecture Objective Function of LCC-GANs Generalization Analysis 3 Experiments Results on Real-world Datasets Comparisons of Representation Methods Demonstration of LCC Sampling 4 Conclusions
4. Introduction Adversarial Learning with LCC Experiments Conclusions Introduction Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) have been successfully applied to many applications. Machine Vision ➢ Image/video generation ➢ Image to image translation ➢ Image super-resolution Natural Language Processing ➢ Neural dialogue generation ➢ Text to image translation ➢ Machine translation
5. Introduction Adversarial Learning with LCC Experiments Conclusions Generative Adversarial Networks (GANs) Two-player game: a generator generate data from a prior distribution, and a discriminator distinguishes between the generated data and real data. Objective Function: Given a generator Gu and a discriminator Dv with their parameters u ∈ U and v ∈ V, the objective function is defined as: min u∈U max v∈V E x∼Dreal [log Dv (x)] + E x∼DGu [log(1 − Dv (x))] , where Dreal and DGu are the real and generated distribution, respectively.
6. Introduction Adversarial Learning with LCC Experiments Conclusions Local Coordinate Coding (LCC) Function Approximation: A nonlinear function can be approximated by a linear function w.r.t. the codings,        i i,j j i j f f   x v x where is the coding, and is the basis of the data point .  i,j   j i v x i x , A data point on the manifold can be approximated by a linear combination of a set of local bases, Data Approximation:    i i, j j i j   x v x , i x   i f x
7. Introduction Adversarial Learning with LCC Experiments Conclusions Contents 1 Introduction Generative Adversarial Networks (GANs) Local Coordinate Coding (LCC) 2 Adversarial Learning with LCC Our Motivations LCC-GANs Architecture Objective Function of LCC-GANs Generalization Analysis 3 Experiments Results on Real-world Datasets Comparisons of Representation Methods Demonstration of LCC Sampling 4 Conclusions
8. Introduction Adversarial Learning with LCC Experiments Conclusions Motivations GANs learn from some pre-defined prior latent distribution, e.g., Gaussian distributions or uniform distributions. Such prior distribution is often independent of real data and thus may lose semantic information of data. How to conduct sampling from the latent distribution still remains an open question in GANs. Adversarial AutoEncoder (AAE) (Makhzani et al., 2015) The generalization ability of GANs w.r.t. the dimension of the latent distribution is unknown. The performance of GANs is sensitive to the dimension of the latent distribution in practice. It is hard to study the dimension of latent distribution and its impacts on the generalization ability.
9. Introduction Adversarial Learning with LCC Experiments Conclusions LCC-GANs Architecture Generator LCC Sampling Discriminator LCC Sampling Step 2: Construct a M-dimensional vector with . Step 1: Find -nearest neighbors ℬ = =1 of an arbitrary basis . z~ z Real Data Latent Manifold Constructed LCC Coding AE LCC : Coding Values : LCC Bases : Embeddings = , ∈ ℬ 0 , ∉ ℬ 1 AutoEncoder (AE) learns embeddings on the latent manifold. 2 Local Coordinate Coding (LCC) learns local coordinate systems. 3 The LCC sampling method is conducted on the latent manifold.
10. Introduction Adversarial Learning with LCC Experiments Conclusions LCC-GANs Architecture (1) AutoEncoder Generator LCC Sampling Discriminator LCC Sampling Step 2: Construct a M-dimensional vector with . Step 1: Find -nearest neighbors ℬ = =1 of an arbitrary basis . z~ z Real Data Latent Manifold Constructed LCC Coding AE LCC : Coding Values : LCC Bases : Embeddings = , ∈ ℬ 0 , ∉ ℬ 1 AutoEncoder (AE): Given N training data {xi}N i=1 , we can use AE to extract the embeddings on the latent manifold, i.e., hi = Encoder(xi), i = 1, . . . , N.
11. Introduction Adversarial Learning with LCC Experiments Conclusions LCC-GANs Architecture (2) Local Coordinate Coding Generator LCC Sampling Discriminator LCC Sampling Step 2: Construct a M-dimensional vector with . Step 1: Find -nearest neighbors ℬ = =1 of an arbitrary basis . z~ z Real Data Latent Manifold Constructed LCC Coding AE LCC : Coding Values : LCC Bases : Embeddings = , ∈ ℬ 0 , ∉ ℬ 2 Local Coordinate Coding (LCC): Given N embeddings {hi }N i=1 and the (Lh , LG )-Lipschitz smooth generator, we minimize the objective function of LCC: minγ,C h 2Lh h−r(h) 2 +LG v∈C |γv (h)| v−h 2 2 s.t. v∈C γv (h) = 1, ∀ h, where r(h) = v∈C γv (h)v .
12. Introduction Adversarial Learning with LCC Experiments Conclusions LCC-GANs Architecture (3) LCC Sampling Method Generator LCC Sampling Discriminator LCC Sampling Step 2: Construct a M-dimensional vector with . Step 1: Find -nearest neighbors ℬ = =1 of an arbitrary basis . z~ z Real Data Latent Manifold Constructed LCC Coding AE LCC : Coding Values : LCC Bases : Embeddings = , ∈ ℬ 0 , ∉ ℬ 3 LCC Sampling Method: Step 1: Find d-nearest neighbors B={vi }d j=1 of a basis vk . Step 2: Construct a coding vector γ(h) = [γ1 (h), . . . , γM (h)], γj (h) = zj , vj ∈ B 0 , vj / ∈ B , j = 1, . . . , M, where zj it the j-th element of z from the prior distribution p(z), e.g., standard Gaussian distribution N(0, I).
13. Introduction Adversarial Learning with LCC Experiments Conclusions LCC-GANs Architecture (3) LCC Sampling Method Figure 1: The geometric views on LCC Sampling With the help of LCC, we obtain local coordinate systems for sampling on the latent manifold. Using the local coordinate systems, LCC-GANs always sample some meaningful points to generate new images with different attributes.
14. Introduction Adversarial Learning with LCC Experiments Conclusions Objective Function of LCC-GANs Objective Function: Given a generator Gw and a discriminator Dv with their parameters w ∈ W and v ∈ V, respectively, we minimize the objective function: min w∈W max v∈V E x∼Dreal φ(Dv (x)) + E h∼H φ 1 − Dv (Gw (γ(h))) , where φ(·) is a monotone function and H is the latent distribution. Note that γ(h) is the local coding in the generator. Key question What is the difference between LCC-GANs and other GAN methods?
15. Introduction Adversarial Learning with LCC Experiments Conclusions Difference between GAN methods 1 h Global Coordinate Coding (Vanilla GANs, WGANs, etc) Local Coordinate Coding (LCC-GANs) The input of the generator: LCC-GANs: local coordinate coding. Other GAN methods: global coordinate coding.
16. Introduction Adversarial Learning with LCC Experiments Conclusions Algorithm of LCC-GANs
17. Introduction Adversarial Learning with LCC Experiments Conclusions Generalization Analysis Definition (Generalization) The neural network distance dF,φ (·, ·) between distributions generalizes with N training samples and error , if for a learned distribution DGu , the following holds with high probability, dF,φ DGw , Dreal − infG dF,φ (DGu , Dreal ) ≤ . Theorem (Generalization Bound) Suppose |φ (·)| ≤ Lφ and −∆ ≤ φ(·) ≤ ∆. Given the coordinate coding (γ, C), the latent distribution H and the Lipschitz smooth generator, then the expected generalization error satisfies: EH dF,φ DGw , Dreal − infG EH [dF,φ (DGu , Dreal )] ≤ 2 RX (F) + ∆ 2 N log(1 δ ) + (dM ) , where RX (F) is the Rademacher complexity of F, and the error (dM ) has an upper bound w.r.t. dM .
18. Introduction Adversarial Learning with LCC Experiments Conclusions Contents 1 Introduction Generative Adversarial Networks (GANs) Local Coordinate Coding (LCC) 2 Adversarial Learning with LCC Our Motivations LCC-GANs Architecture Objective Function of LCC-GANs Generalization Analysis 3 Experiments Results on Real-world Datasets Comparisons of Representation Methods Demonstration of LCC Sampling 4 Conclusions
19. Introduction Adversarial Learning with LCC Experiments Conclusions Experimental Settings Methods for comparison: Vanilla GANs, WGANs and Progressive GANs Vanilla GANs or Progressive GANs are used to implement LCC-GANs. Real-world Datasets: MNIST, Oxford-102, LSUN and CelebA Evaluation Metrics: Inception Score (IS) and Multi-Scale Structural Similarity (MS-SSIM)
20. Introduction Adversarial Learning with LCC Experiments Conclusions Experiments Results on MNIST With d = 3, LCC-GANs generate digits with different styles and different orientations compared with Vanilla GANs. With d = 5, LCC-GANs achieve comparable or even better performance than the baselines with d = 100.
21. Introduction Adversarial Learning with LCC Experiments Conclusions Experiments Results on Oxford-102 Flowers In Figure 5, compared with Vanilla GANs, LCC-GANs generate images with clear structure given the small dimensional inputs. In Table 1, LCC-GANs with d = 10 outperform all baseline methods.
22. Introduction Adversarial Learning with LCC Experiments Conclusions Experiments Results on LSUN and CelebA With a small dimension, baseline GANs fail to generate clear images. LCC-GANs with a small dimensional input generate sharper images than baseline GANs with a high dimensional input.
23. Introduction Adversarial Learning with LCC Experiments Conclusions Experiments Comparisons of Different Representation Methods (on Oxford-102) We compare different representation methods and use Inception Score and MS-SSIM to measure the quality and diversity. LCC-GANs consistently outperform other representation methods with various d in both measures.
24. Introduction Adversarial Learning with LCC Experiments Conclusions Experiments Demonstration of LCC Sampling Generated images locate in a local area of the latent manifold and share some common features but with different attributes. LCC sampling method is able to exploit the local information. LCC-GANs generalize well and do not memorize the training samples.
25. Introduction Adversarial Learning with LCC Experiments Conclusions Contents 1 Introduction Generative Adversarial Networks (GANs) Local Coordinate Coding (LCC) 2 Adversarial Learning with LCC Our Motivations LCC-GANs Architecture Objective Function of LCC-GANs Generalization Analysis 3 Experiments Results on Real-world Datasets Comparisons of Representation Methods Demonstration of LCC Sampling 4 Conclusions
26. Introduction Adversarial Learning with LCC Experiments Conclusions Conclusions Our conclusions are as follows: The LCC sampling method is able to sample meaningful points from the latent manifold to generate new data. A small dimensional input is sufficient to achieve good generalization performance. LCC-GANs outperform the baselines by generating sharper images with higher diversity.
27. Introduction Adversarial Learning with LCC Experiments Conclusions Thank you! Q & A