ENHANCE!! - Upscaling Images with Neural Networks

When characters on a TV show “enhance!” a blurry image, you probably laugh and tell your friends that it’s impossible to do that in real life. But over the past year, deep learning research has actually made this kind of possible! How will you explain this to your friends!?

In this talk, you’ll get an intuitive introduction to generative adversarial networks, a new machine learning technique that’s surprisingly good at upscaling images. You’ll learn how these systems are inspired by human art forgers, and how you can use them to do other things like transform a horse into a zebra, convert a sentence into a photo, and so much more!

Transcript

1. ENHANCE!! @geoffreylitt
2. Entertainment Accuracy of portraying computers
4. ENHANCE, PLEASE
7. Bicubic interpolation
8. Bicubic interpolation
9. DO BETTER
10. NEURAL NETWORKS!!!
11. Neural network upscaling
12. Neural network training { } , { } , Training data
13. Neural network training A: Neural network’s guess B: Real answer Per-pixel subtraction Add up all the pixels Total per-pixel difference Minimize this!
14. Neural network training Loss function: “Minimize the total per-pixel difference between the two images”
15. Neural network Code: https://github.com/david-gpu/srez
16. Neural network Bicubic interpolation
17. MORE. ENHANCE.
18. A: Neural network’s guess B: Real answer Per-pixel subtraction
19. A: Neural network’s guess B: Real answer Really high loss
20. A: Neural network’s guess B: Real answer Lower loss
21. A new loss function ??? “Minimize the total per-pixel difference between your guess and the real answer… But also make your guess look sharp and realistic” Loss = Total Per-pixel difference +
22. GENERATIVE ADVERSARIAL NETWORKS
23. Adversarial games I’ll develop better counterfeit detection techniques! I’ll develop better counterfeiting techniques!
24. Generator GAN structure { { Training example B: Real answer A: Neural network’s guess
25. Discriminator Generator GAN structure { { Training example B: Real answer Probability of fake Goal: differentiate fake vs. real with 100% accuracy Goal: Minimize discriminator accuracy A: Neural network’s guess
26. Discriminator Generator GAN structure { { Training example B: Real answer Probability of fake A: Neural network’s guess “Here’s how you could make a better fake next time”
27. low-res bicubic GAN Real answer
28. GAN-trained generator Generator
29. GAN results original GAN Per-pixel diff neural network
30. GOOD JOB
31. GAN Real answer low-res
34. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks Jun-Yan Zhu*, Taesung Park*, Phillip Isola, Alexei A. Efros
36. Image-to-Image Translation with Conditional Adversarial Networks Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros https://affinelayer.com/pixsrv/
37. Image-to-Image Translation with Conditional Adversarial Networks Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros https://affinelayer.com/pixsrv/
38. StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaolei Huang, Xiaogang Wang, Dimitris Metaxas
39. Thanks! @geoffreylitt