Intro to Variational AutoEncoder

Presented at "The second Deep Learning summer school at University of Tehran": http://acm.ut.ac.ir/deeplearning/

Transcript

1. Variational AutoEncoder Yasser Souri Uni Bonn souri@iai.uni-bonn.de 1
2. Discriminative vs. Generative • Discriminative p(y|x) !2
3. Discriminative vs. Generative • Generative p(y, x) = p(y|x) p(x) !3
4. Generative Models • Two key tasks • Sampling • Likelihood estimation • And of course you need to learn it from data !4
5. Deep Generative Models • Generative Adversarial Network (GAN) • Tomorrow • Variational AutoEncoder (VAE) • Today !5
6. VAE • Latent Variable Models • ELBO !6
7. Latent Variable Models z x p(x, z) = p(x|z) p(z) !7
8. Latent Variable Models z x pθ (x, z) = pθ (x|z) p(z) !8
9. Latent Variable Models z x pθ (x, z) = pθ (x|z) p(z) !9
10. Latent Variable Models z x pθ (x, z) = pθ (x|z) p(z) Learning with Maximum Likelihood θmax = argmax θ log pθ (x) !10
11. Latent Variable Models z x pθ (x, z) = pθ (x|z) p(z) Learning with Maximum Likelihood θmax = argmax θ log pθ (x) θmax = argmax θ log ∫ pθ (x, z)dz !11
12. Latent Variable Models z x pθ (x, z) = pθ (x|z) p(z) Learning with Maximum Likelihood θmax = argmax θ log ∫ pθ (x, z)dz !12
13. Latent Variable Models z x pθ (x, z) = pθ (x|z) p(z) Posterior probability pθmax (z|x) = pθmax (x, z) ∫ pθmax (x, z)dz !13
14. Latent Variable Models z x pθ (x, z) = pθ (x|z) p(z) Variational distribution pθmax (z|x) = pθmax (x, z) ∫ pθmax (x, z)dz qϕ (z|x) !14
15. Latent Variable Models z x pθ (x, z) = pθ (x|z) p(z) qϕ (z|x) pθ (x|z) Decoder Encoder !15
16. ELBO log pθ (x) !16
17. ELBO log pθ (x) = log ∫ pθ (x, z)dz = log ∫ pθ (x|z) p(z)dz !17
18. ELBO log pθ (x) = log ∫ pθ (x, z)dz = log ∫ pθ (x|z) p(z)dz = log ∫ qϕ (z|x) qϕ (z|x) pθ (x|z) p(z)dz !18
19. ELBO log pθ (x) = log ∫ pθ (x, z)dz = log ∫ pθ (x|z) p(z)dz = log ∫ qϕ (z|x) qϕ (z|x) pθ (x|z) p(z)dz ≥ ∫ qϕ (z|x) log pθ (x|z) p(z) qϕ (z|x) dz !19
20. ELBO log pθ (x) ≥ ∫ qϕ (z|x) log pθ (x|z) p(z) qϕ (z|x) dz !20
21. ELBO log pθ (x) ≥ ∫ qϕ (z|x) log pθ (x|z) p(z) qϕ (z|x) dz = z∼qϕ (z|x) [log pθ (x|z)] − KL[qϕ (z|x) ∥ p(z)] !21
22. Latent Variable Models z x pθ (x, z) = pθ (x|z) p(z) qϕ (z|x) pθ (x|z) Decoder Encoder log pθ (x) ≥ z∼qϕ (z|x) [log pθ (x|z)] − KL[qϕ (z|x) ∥ p(z)] !22
23. VAE !23
24. VAE p(z) = (0,I) !24
25. VAE p(z) = (0,I) pθ (x|z) = (x| f(z; θ), σ2 * I) !25
26. VAE p(z) = (0,I) pθ (x|z) = (x| f(z; θ), σ2 * I) qϕ (z|x) = (z|μ(x; ϕ), Σ(x; ϕ)) !26
27. VAE z∼qϕ (z|x) [log pθ (x|z)] − KL[qϕ (z|x) ∥ p(z)] log pθ (x|z) ∝ − ∥ f(z; θ) − x ∥2 2 KL[qϕ (z|x) ∥ p(z)] = 1 2 (tr(Σ(x; ϕ)) + (μ(x; ϕ)T μ(x; ϕ)) − k − log det(Σ(x; ϕ))) !27
28. VAE Training x μ(x; ϕ) Σ(x; ϕ) qϕ (z|x) KL[qϕ (z|x) ∥ p(z)] z ∼ (μ(x; ϕ), Σ(x; ϕ)) pθ (x|z) f(z; θ) ∥ f(z; θ) − x ∥2 2 !28
29. Reparameterization Trick x μ(x; ϕ) Σ(x; ϕ) qϕ (z|x) KL[qϕ (z|x) ∥ p(z)] ϵ ∼ (0, I) pθ (x|z) f(z; θ) ∥ f(z; θ) − x ∥2 2 * + !29
30. VAE Sampling z ∼ p(z) pθ (x|z) f(z; θ) !30
31. Applications • Conditional Generation
32. Applications • Conditional Generation Walker, Jacob, et al. "An uncertain future: Forecasting from static images using variational autoencoders." European Conference on Computer Vision. Springer, Cham, 2016.
33. Applications • Almost anything GANs can be used for • Tomorrow
34. VAE vs. GAN • VAE: Blurrier and smaller images • VAE + GAN is interesting • VAE gives you a framework to extend and design Eslami, SM Ali, et al. "Attend, infer, repeat: Fast scene understanding with generative models." Advances in Neural Information Processing Systems. 2016.
35. Some Notes 1 - Pyro
36. Some Notes 2 - Distill
37. Some Notes 3 - PhD Positions at Uni Bonn http://cv-uni-bonn.de/
38. Thank you for your attention Doersch, Carl. "Tutorial on variational autoencoders." arXiv preprint arXiv:1606.05908 (2016). Kingma, Diederik P., and Max Welling. "Auto-encoding variational bayes." arXiv preprint arXiv:1312.6114 (2013).