Introduction of GAN

Transcript

1. Generative Adversarial Networks (GAN) Jiezhang Cao South China University of Technology
2. 2012 2013 2014 2015 2016 2017 2018 generative adversarial reinforcement Goodfellow et al., 2014 GAN becomes a very important technology Number of papers whose titles include the keyword
3. All Kinds of GAN … GAN WGAN BGAN DCGAN ACGAN fGAN LCC-GAN (ICML) CGAN ……
4. GAN Zoo ACGAN BGAN CGAN DGAN EGAN fGAN GAGAN HiGAN iGAN JointGAN LCC-GAN KGAN MGAN NetGAN OT-GAN PGAN RCGAN SEGAN ? TGAN UGAN VGAN WGAN XGAN ZipNet-GAN ? https://github.com/hindupuravinash/the-gan-zoo
5. ➢Generative Modeling ➢Generative Adversarial Networks (GANs) ➢Conditional GANs (CGANs) ➢Training Procedure and Evaluation Methods ➢Common Issues and Some Tips ➢Some Applications Contents
6. Generative Modeling • Have training examples ~ () • Want a model that can draw samples: ~ () • Where () ≈ () ~ () ~ ()
7. Why Generative Models? • Generative models — Date generation — Speech synthesis: — Machine translation: Chinese English • Chinese: 我喜欢深度学习。 • English: I love deep learning. • Environment simulator — Reinforcement learning — Planning • Leverage unlabeled data
8. Data Generation Image Generation
9. Data Generation Generator Image Generation 0.3 −0.1 ⋮ −0.7 In a specific range
10. Data Generation Generator Image Generation 0.1 −0.1 ⋮ 0.7 0.3 −0.1 ⋮ −0.7 In a specific range
11. Data Generation Generator Image Generation 0.1 −0.1 ⋮ 0.7 −0.3 0.1 ⋮ 0.9 0.3 −0.1 ⋮ −0.7 In a specific range
12. Data Generation Generator Image Generation Sentence Generation 0.1 −0.1 ⋮ 0.7 −0.3 0.1 ⋮ 0.9 0.3 −0.1 ⋮ −0.7 In a specific range
13. Data Generation Generator Image Generation Sentence Generation Generator 0.1 −0.1 ⋮ 0.7 −0.3 0.1 ⋮ 0.9 How are you? 0.3 −0.1 ⋮ −0.7 0.3 −0.1 ⋮ −0.7 In a specific range
14. Data Generation Generator Image Generation Sentence Generation Generator 0.1 −0.1 ⋮ 0.7 −0.3 0.1 ⋮ 0.9 0.1 −0.1 ⋮ 0.2 Good morning. How are you? 0.3 −0.1 ⋮ −0.7 0.3 −0.1 ⋮ −0.7 In a specific range
15. Data Generation Generator Image Generation Sentence Generation Generator 0.1 −0.1 ⋮ 0.7 −0.3 0.1 ⋮ 0.9 0.1 −0.1 ⋮ 0.2 −0.3 0.1 ⋮ 0.5 Good morning. How are you? 0.3 −0.1 ⋮ −0.7 0.3 −0.1 ⋮ −0.7 Good afternoon. In a specific range
16. ➢Generative modeling ➢Generative adversarial networks (GANs) ➢Conditional GANs (CGANs) ➢Training procedure and evaluation methods ➢Common issues and some tips ➢Some Applications Contents
17. By 2018-12-04
18. Contributors of GAN
20. GAN’s Big Fan: Yann LeCun
21. A Game Theory Perspective ➢Two agents: — Generative model G: generate new samples that are as similar as data and fool the discriminator — Discriminative model D: act as a judge to distinguish the generated samples from the real one adversarial Generator Discriminator Vs
22. General Architecture of GANs (Counterfeiter) (Police) Noise This is real/ fake!
23. Generator Network • G is a neural network with parameters . • is a latent variable drawn from a prior distribution. • Trainable for any size of A generative model tries to learn the distribution of real data from the prior distribution ().
24. Generator Generator It is a neural network (NN), or a function. Generator 0.1 −3 ⋮ 2.4 0.9 image vector Generator 3 −3 ⋮ 2.4 0.9 Generator 0.1 2.1 ⋮ 5.4 0.9 Generator 0.1 −3 ⋮ 2.4 3.5 high dimensional vector Each dimension of input vector represents some characteristic. Long hair blue hair Open mouth low dimensional
25. Generator of DCGAN Image Radford et al. 2015
26. • A discriminative model learns a function that distinguish the generated samples from the real data • (): estimate the probability that is a real data sample — = 1: regards as a real sample — = 0: regards as a fake sample Discriminative Model Binary classification
27. Discriminator Probability value Image Larger value: real smaller value: fake. Discriminator Discriminator Discriminator 1.0 1.0 0.1 Discriminator 0.1 Discriminator It is a neural network (NN), or a function.
28. Discriminator of DCGAN Radford et al. 2015 Real or Fake CONV 1 CONV 2 CONV 3 CONV 4 Image
29. ➢Generative modeling ➢Generative adversarial networks (GANs) ➢Conditional GANs (CGANs) ➢Training procedure and evaluation methods ➢Common issues and some tips ➢Some Applications Contents
30. Conditional GANs condition • Simple modification to the original GAN framework that conditions the model on additional information for better multi-modal learning. • Many practical applications of GANs when we have explicit supervision available
31. Conditional GANs D scalar True text-image pairs: G Gaussian distribution x = G(c,z) c: train Image is realistic or not + c and are matched or not (train , ) (train , ) (cat , ) 1 0 0 Condition
32. Conditional GANs paired data blue eyes red hair short hair red hair, green eyes blue hair, red eyes
33. ➢Generative modeling ➢Generative adversarial networks (GANs) ➢Conditional GANs (CGANs) ➢Training procedure and evaluation methods ➢Common issues and some tips ➢Some Applications Contents
34. Training GANs: Two-player game Generator networks: try to fool the discriminator by generating real-looking images Discriminator networks: try to distinguish between real and fake images
35. Minimax Game • Generator minimizes the log-probability of D being correct • Discriminator maximizes cross-entropy of real and generated data • Equilibrium is a saddle point of the discriminator loss Cross-entropy Loss
36. Minimax Game • Discriminator aims to maximize objective such that D(x) is close to 1 (real) and D(G(z)) is close to 0 (fake) • Generator aims to minimize objective such that D(G(z)) is close to 1 (discriminator is fooled into thinking generated G(z) is real)
37. Training Procedure • Use SGD algorithm on two mini-batches simultaneously: • A mini-batch of training samples • A mini-batch of generated samples • Optional: Run steps of one player for every step of the other player - example: WGAN updates Discriminator for 5 steps and Generator for 1 step.
38. sample generated samples G D Update vector vector vector vector 0 0 0 0 1 1 1 1 randomly sampled Database Step 1: Fix generator G, and update discriminator D Fix Training Algorithm: Discriminator Objective of the discriminator:
39. Training Algorithm: Discriminator
40. Step 2: Fix discriminator D, and update generator G Discriminator Generator vector 0.13 hidden layer update fix large network Generator learns to “fool” the discriminator Training Algorithm: Generator Objective of the discriminator:
41. Training Algorithm: Generator
42. • Sample m examples 1, 2, … , from database • Sample m noise samples 1, 2, … , from a distribution • Obtaining generated data ෤ 1, ෤ 2, … , ෤ , ෤ = • Update discriminator parameters to maximize • ෨ = 1 σ =1 + 1 σ =1 1 − ෤ • ← + ෨ • Sample m noise samples 1, 2, … , from a distribution • Update generator parameters to minimize • ෨ = 1 σ =1 1 − • ← − ෨ Learning D Learning G Training Algorithm of GANs Repeat k times
43. Train GAN in the Browser https://reiinakano.github.io/gan-playground/
44. Training: An Intuitive Explanation Discriminative distribution Data distribution Generative distribution Poorly fit model
45. Training: An Intuitive Explanation Discriminative distribution Data distribution Generative distribution After updating D Poorly fit model
46. Training: An Intuitive Explanation Discriminative distribution Data distribution Generative distribution Poorly fit model After updating D After updating G
47. Training: An Intuitive Explanation Discriminative distribution Data distribution Generative distribution After updating D Poorly fit model After updating G Mixed strategy equilibrium
48. Visualization of GAN https://poloclub.github.io/ganlab/
49. 100 iterations Source of training data: https://zhuanlan.zhihu.com/p/24767059 Anime Face Generation
50. 1000 iterations Anime Face Generation
51. 2000 iterations Anime Face Generation
52. 5000 iterations Anime Face Generation
53. 10000 iterations Anime Face Generation
54. 20000 iterations Anime Face Generation
55. The faces generated by machine.
56. Evaluation: Inception Score = ෍ ෍ | | − ෍ Negative entropy of P(y|x) Entropy of P(y) Inception Score = 1 ෍ | | class 1 class 2 class 3 Formally: Peculiarity Diversity
57. ❖ Multi-scale structural similarity (MS-SSIM): evaluate the diversity of the generated samples ❖ Fré chet Inception Distance (FID) a new metric to evaluate the performance of GANs Other Evaluation Metrics
58. ➢Generative modeling ➢Generative adversarial networks (GANs) ➢Conditional GANs (CGANs) ➢Training procedure and evaluation methods ➢Common issues and some tips ➢Some Applications Contents
59. Common Issues in GANs • Non-convergence: although GAN training is theoretically guaranteed to converge, the model often break down during the training • Oscillation: GAN can be trained for a very long time without clearly generating better samples • Training imbalance: discriminator becomes too strong too quickly and the generator ends up not learning anything • Mode collapse: almost all the samples look the same (very low diversity) • Visual problems: counting, perspective, global structure, etc.
60. Mode Collapse Reed et al. 2016 Reed et al. submitted to ICLR 2017
61. Visualization of Mode Collapse https://poloclub.github.io/ganlab/
62. Problems with Counting Goodfellow et al. 2016
63. Problems with Perspective Goodfellow et al. 2016
64. Problems with Global Structure Goodfellow et al. 2016
65. Cross-entropy (CE) loss: • Default discriminator cost: • One-sided label smoothed cost (Salimans et al. 2016) : label : probability Benefits • Prevents discriminator from giving very large gradient signal to generator • Stabilize the training and prevents extrapolating to encourage extreme samples One-Sided Label Smoothing: Some Solutions
66. D score D D score score Patch GANs: Some Solutions
67. • A modified loss function • DCGAN model design • Track failures early (D loss goes to 0: failure mode) • If you have labels, use them • Add noise to inputs, decay over time Tricks for GANs . . . . . .
68. ➢Generative modeling ➢Generative adversarial networks (GANs) ➢Conditional GANs (CGANs) ➢Training procedure and evaluation methods ➢Common issues and some tips ➢Some Applications Contents
69. Latent Space Interpolation Donahue et al. 2015 https://chrisdonahue.github.io/sdgan/ z1 z2 z0.5 G(z0.5 ) G(z1 ) G(z2 )
70. Latent Space Arithmetic Radford et al. 2015
71. Video Prediction Lee et al. 2018
72. Single Image Super-Resolution Ledig et al. 2018
73. Image Manipulation https://github.com/junyanz/iGAN
74. Image Manipulation http://gandissect.res.ibm.com/ganpaint.html?project=churchoutdoor&layer=layer4
75. Image to Image Translation Isola et al. 2016
76. Image to Image Translation (Cycle GAN) Zhu et al. 2016
77. Multi-Domain Image to Image Translation (StarGAN) Choi et al. 2016
78. Conclusions • GANs are generative models that are implemented using two stochastic neural network modules: Generator and Discriminator. • Generator tries to generate samples from random noise as input. • Discriminator tries to distinguish the samples from Generator and samples from the real data distribution. • Both players are trained adversarially to fool each other, and they become better at their respective tasks.
79. WeChat My website Thank you! My mail: secaojiezhang@mail.scut.edu.cn Questions?