Introduction of GAN

3d3f89506a9a5c12b71a40ecacb474df?s=47 caojiezhang
December 04, 2018
17

Introduction of GAN

3d3f89506a9a5c12b71a40ecacb474df?s=128

caojiezhang

December 04, 2018
Tweet

Transcript

  1. Generative Adversarial Networks (GAN) Jiezhang Cao South China University of

    Technology
  2. 2012 2013 2014 2015 2016 2017 2018 generative adversarial reinforcement

    Goodfellow et al., 2014 GAN becomes a very important technology Number of papers whose titles include the keyword
  3. All Kinds of GAN … GAN WGAN BGAN DCGAN ACGAN

    fGAN LCC-GAN (ICML) CGAN ……
  4. GAN Zoo ACGAN BGAN CGAN DGAN EGAN fGAN GAGAN HiGAN

    iGAN JointGAN LCC-GAN KGAN MGAN NetGAN OT-GAN PGAN RCGAN SEGAN ? TGAN UGAN VGAN WGAN XGAN ZipNet-GAN ? https://github.com/hindupuravinash/the-gan-zoo
  5. ➢Generative Modeling ➢Generative Adversarial Networks (GANs) ➢Conditional GANs (CGANs) ➢Training

    Procedure and Evaluation Methods ➢Common Issues and Some Tips ➢Some Applications Contents
  6. Generative Modeling • Have training examples ~ () • Want

    a model that can draw samples: ~ () • Where () ≈ () ~ () ~ ()
  7. Why Generative Models? • Generative models — Date generation —

    Speech synthesis: — Machine translation: Chinese English • Chinese: 我喜欢深度学习。 • English: I love deep learning. • Environment simulator — Reinforcement learning — Planning • Leverage unlabeled data
  8. Data Generation Image Generation

  9. Data Generation Generator Image Generation 0.3 −0.1 ⋮ −0.7 In

    a specific range
  10. Data Generation Generator Image Generation 0.1 −0.1 ⋮ 0.7 0.3

    −0.1 ⋮ −0.7 In a specific range
  11. Data Generation Generator Image Generation 0.1 −0.1 ⋮ 0.7 −0.3

    0.1 ⋮ 0.9 0.3 −0.1 ⋮ −0.7 In a specific range
  12. Data Generation Generator Image Generation Sentence Generation 0.1 −0.1 ⋮

    0.7 −0.3 0.1 ⋮ 0.9 0.3 −0.1 ⋮ −0.7 In a specific range
  13. Data Generation Generator Image Generation Sentence Generation Generator 0.1 −0.1

    ⋮ 0.7 −0.3 0.1 ⋮ 0.9 How are you? 0.3 −0.1 ⋮ −0.7 0.3 −0.1 ⋮ −0.7 In a specific range
  14. Data Generation Generator Image Generation Sentence Generation Generator 0.1 −0.1

    ⋮ 0.7 −0.3 0.1 ⋮ 0.9 0.1 −0.1 ⋮ 0.2 Good morning. How are you? 0.3 −0.1 ⋮ −0.7 0.3 −0.1 ⋮ −0.7 In a specific range
  15. Data Generation Generator Image Generation Sentence Generation Generator 0.1 −0.1

    ⋮ 0.7 −0.3 0.1 ⋮ 0.9 0.1 −0.1 ⋮ 0.2 −0.3 0.1 ⋮ 0.5 Good morning. How are you? 0.3 −0.1 ⋮ −0.7 0.3 −0.1 ⋮ −0.7 Good afternoon. In a specific range
  16. ➢Generative modeling ➢Generative adversarial networks (GANs) ➢Conditional GANs (CGANs) ➢Training

    procedure and evaluation methods ➢Common issues and some tips ➢Some Applications Contents
  17. By 2018-12-04

  18. Contributors of GAN

  19. None
  20. GAN’s Big Fan: Yann LeCun

  21. A Game Theory Perspective ➢Two agents: — Generative model G:

    generate new samples that are as similar as data and fool the discriminator — Discriminative model D: act as a judge to distinguish the generated samples from the real one adversarial Generator Discriminator Vs
  22. General Architecture of GANs (Counterfeiter) (Police) Noise This is real/

    fake!
  23. Generator Network • G is a neural network with parameters

    . • is a latent variable drawn from a prior distribution. • Trainable for any size of A generative model tries to learn the distribution of real data from the prior distribution ().
  24. Generator Generator It is a neural network (NN), or a

    function. Generator 0.1 −3 ⋮ 2.4 0.9 image vector Generator 3 −3 ⋮ 2.4 0.9 Generator 0.1 2.1 ⋮ 5.4 0.9 Generator 0.1 −3 ⋮ 2.4 3.5 high dimensional vector Each dimension of input vector represents some characteristic. Long hair blue hair Open mouth low dimensional
  25. Generator of DCGAN Image Radford et al. 2015

  26. • A discriminative model learns a function that distinguish the

    generated samples from the real data • (): estimate the probability that is a real data sample — = 1: regards as a real sample — = 0: regards as a fake sample Discriminative Model Binary classification
  27. Discriminator Probability value Image Larger value: real smaller value: fake.

    Discriminator Discriminator Discriminator 1.0 1.0 0.1 Discriminator 0.1 Discriminator It is a neural network (NN), or a function.
  28. Discriminator of DCGAN Radford et al. 2015 Real or Fake

    CONV 1 CONV 2 CONV 3 CONV 4 Image
  29. ➢Generative modeling ➢Generative adversarial networks (GANs) ➢Conditional GANs (CGANs) ➢Training

    procedure and evaluation methods ➢Common issues and some tips ➢Some Applications Contents
  30. Conditional GANs condition • Simple modification to the original GAN

    framework that conditions the model on additional information for better multi-modal learning. • Many practical applications of GANs when we have explicit supervision available
  31. Conditional GANs D scalar True text-image pairs: G Gaussian distribution

    x = G(c,z) c: train Image is realistic or not + c and are matched or not (train , ) (train , ) (cat , ) 1 0 0 Condition
  32. Conditional GANs paired data blue eyes red hair short hair

    red hair, green eyes blue hair, red eyes
  33. ➢Generative modeling ➢Generative adversarial networks (GANs) ➢Conditional GANs (CGANs) ➢Training

    procedure and evaluation methods ➢Common issues and some tips ➢Some Applications Contents
  34. Training GANs: Two-player game Generator networks: try to fool the

    discriminator by generating real-looking images Discriminator networks: try to distinguish between real and fake images
  35. Minimax Game • Generator minimizes the log-probability of D being

    correct • Discriminator maximizes cross-entropy of real and generated data • Equilibrium is a saddle point of the discriminator loss Cross-entropy Loss
  36. Minimax Game • Discriminator aims to maximize objective such that

    D(x) is close to 1 (real) and D(G(z)) is close to 0 (fake) • Generator aims to minimize objective such that D(G(z)) is close to 1 (discriminator is fooled into thinking generated G(z) is real)
  37. Training Procedure • Use SGD algorithm on two mini-batches simultaneously:

    • A mini-batch of training samples • A mini-batch of generated samples • Optional: Run steps of one player for every step of the other player - example: WGAN updates Discriminator for 5 steps and Generator for 1 step.
  38. sample generated samples G D Update vector vector vector vector

    0 0 0 0 1 1 1 1 randomly sampled Database Step 1: Fix generator G, and update discriminator D Fix Training Algorithm: Discriminator Objective of the discriminator:
  39. Training Algorithm: Discriminator

  40. Step 2: Fix discriminator D, and update generator G Discriminator

    Generator vector 0.13 hidden layer update fix large network Generator learns to “fool” the discriminator Training Algorithm: Generator Objective of the discriminator:
  41. Training Algorithm: Generator

  42. • Sample m examples 1, 2, … , from database

    • Sample m noise samples 1, 2, … , from a distribution • Obtaining generated data ෤ 1, ෤ 2, … , ෤ , ෤ = • Update discriminator parameters to maximize • ෨ = 1 σ =1 + 1 σ =1 1 − ෤ • ← + ෨ • Sample m noise samples 1, 2, … , from a distribution • Update generator parameters to minimize • ෨ = 1 σ =1 1 − • ← − ෨ Learning D Learning G Training Algorithm of GANs Repeat k times
  43. Train GAN in the Browser https://reiinakano.github.io/gan-playground/

  44. Training: An Intuitive Explanation Discriminative distribution Data distribution Generative distribution

    Poorly fit model
  45. Training: An Intuitive Explanation Discriminative distribution Data distribution Generative distribution

    After updating D Poorly fit model
  46. Training: An Intuitive Explanation Discriminative distribution Data distribution Generative distribution

    Poorly fit model After updating D After updating G
  47. Training: An Intuitive Explanation Discriminative distribution Data distribution Generative distribution

    After updating D Poorly fit model After updating G Mixed strategy equilibrium
  48. Visualization of GAN https://poloclub.github.io/ganlab/

  49. 100 iterations Source of training data: https://zhuanlan.zhihu.com/p/24767059 Anime Face Generation

  50. 1000 iterations Anime Face Generation

  51. 2000 iterations Anime Face Generation

  52. 5000 iterations Anime Face Generation

  53. 10000 iterations Anime Face Generation

  54. 20000 iterations Anime Face Generation

  55. The faces generated by machine.

  56. Evaluation: Inception Score = ෍ ෍ | | − ෍

    Negative entropy of P(y|x) Entropy of P(y) Inception Score = 1 ෍ | | class 1 class 2 class 3 Formally: Peculiarity Diversity
  57. ❖ Multi-scale structural similarity (MS-SSIM): evaluate the diversity of the

    generated samples ❖ Fré chet Inception Distance (FID) a new metric to evaluate the performance of GANs Other Evaluation Metrics
  58. ➢Generative modeling ➢Generative adversarial networks (GANs) ➢Conditional GANs (CGANs) ➢Training

    procedure and evaluation methods ➢Common issues and some tips ➢Some Applications Contents
  59. Common Issues in GANs • Non-convergence: although GAN training is

    theoretically guaranteed to converge, the model often break down during the training • Oscillation: GAN can be trained for a very long time without clearly generating better samples • Training imbalance: discriminator becomes too strong too quickly and the generator ends up not learning anything • Mode collapse: almost all the samples look the same (very low diversity) • Visual problems: counting, perspective, global structure, etc.
  60. Mode Collapse Reed et al. 2016 Reed et al. submitted

    to ICLR 2017
  61. Visualization of Mode Collapse https://poloclub.github.io/ganlab/

  62. Problems with Counting Goodfellow et al. 2016

  63. Problems with Perspective Goodfellow et al. 2016

  64. Problems with Global Structure Goodfellow et al. 2016

  65. Cross-entropy (CE) loss: • Default discriminator cost: • One-sided label

    smoothed cost (Salimans et al. 2016) : label : probability Benefits • Prevents discriminator from giving very large gradient signal to generator • Stabilize the training and prevents extrapolating to encourage extreme samples One-Sided Label Smoothing: Some Solutions
  66. D score D D score score Patch GANs: Some Solutions

  67. • A modified loss function • DCGAN model design •

    Track failures early (D loss goes to 0: failure mode) • If you have labels, use them • Add noise to inputs, decay over time Tricks for GANs . . . . . .
  68. ➢Generative modeling ➢Generative adversarial networks (GANs) ➢Conditional GANs (CGANs) ➢Training

    procedure and evaluation methods ➢Common issues and some tips ➢Some Applications Contents
  69. Latent Space Interpolation Donahue et al. 2015 https://chrisdonahue.github.io/sdgan/ z1 z2

    z0.5 G(z0.5 ) G(z1 ) G(z2 )
  70. Latent Space Arithmetic Radford et al. 2015

  71. Video Prediction Lee et al. 2018

  72. Single Image Super-Resolution Ledig et al. 2018

  73. Image Manipulation https://github.com/junyanz/iGAN

  74. Image Manipulation http://gandissect.res.ibm.com/ganpaint.html?project=churchoutdoor&layer=layer4

  75. Image to Image Translation Isola et al. 2016

  76. Image to Image Translation (Cycle GAN) Zhu et al. 2016

  77. Multi-Domain Image to Image Translation (StarGAN) Choi et al. 2016

  78. Conclusions • GANs are generative models that are implemented using

    two stochastic neural network modules: Generator and Discriminator. • Generator tries to generate samples from random noise as input. • Discriminator tries to distinguish the samples from Generator and samples from the real data distribution. • Both players are trained adversarially to fool each other, and they become better at their respective tasks.
  79. WeChat My website Thank you! My mail: secaojiezhang@mail.scut.edu.cn Questions?