ENHANCE!! - Upscaling Images with Neural Networks

When characters on a TV show “enhance!” a blurry image, you probably laugh and tell your friends that it’s impossible to do that in real life. But over the past year, deep learning research has actually made this kind of possible! How will you explain this to your friends!?

In this talk, you’ll get an intuitive introduction to generative adversarial networks, a new machine learning technique that’s surprisingly good at upscaling images. You’ll learn how these systems are inspired by human art forgers, and how you can use them to do other things like transform a horse into a zebra, convert a sentence into a photo, and so much more!

Transcript

1. ENHANCE!!
   @geoffreylitt
   

   View Slide

2. Entertainment
   Accuracy of
   portraying
   computers
   

   View Slide

3. View Slide

4. ENHANCE, PLEASE
   

   View Slide

5. View Slide

6. View Slide

7. Bicubic interpolation
   

   View Slide

8. Bicubic interpolation
   

   View Slide

9. DO BETTER
   

   View Slide

10. NEURAL
    NETWORKS!!!
    

    View Slide

11. Neural network upscaling
    

    View Slide

12. Neural network training
    { }
    , { }
    ,
    Training
    data
    

    View Slide

13. Neural network training
    A: Neural
    network’s
    guess
    B: Real answer
    Per-pixel
    subtraction
    Add up all the
    pixels Total
    per-pixel
    difference
    Minimize this!
    

    View Slide

14. Neural network training
    Loss function:
    “Minimize the total per-pixel
    difference between the two
    images”
    

    View Slide

15. Neural network
    Code: https://github.com/david-gpu/srez
    

    View Slide

16. Neural network
    Bicubic interpolation
    

    View Slide

17. MORE.
    ENHANCE.
    

    View Slide

18. A: Neural
    network’s
    guess
    B: Real answer
    Per-pixel
    subtraction
    

    View Slide

19. A: Neural
    network’s guess
    B: Real answer
    Really high loss
    

    View Slide

20. A: Neural
    network’s guess
    B: Real answer
    Lower loss
    

    View Slide

21. A new loss function
    ???
    “Minimize the total per-pixel difference
    between your guess and the real
    answer…
    But also make your guess look sharp
    and realistic”
    Loss =
    Total Per-pixel difference +
    

    View Slide

22. GENERATIVE
    ADVERSARIAL
    NETWORKS
    

    View Slide

23. Adversarial games
    I’ll develop better
    counterfeit detection
    techniques!
    I’ll develop better
    counterfeiting
    techniques!
    

    View Slide

24. Generator
    GAN structure
    {
    {
    Training example
    B: Real
    answer
    A: Neural
    network’s
    guess
    

    View Slide

25. Discriminator
    Generator
    GAN structure
    {
    {
    Training example
    B: Real
    answer
    Probability
    of fake
    Goal: differentiate fake vs.
    real with 100% accuracy
    Goal:
    Minimize discriminator
    accuracy
    A: Neural
    network’s
    guess
    

    View Slide

26. Discriminator
    Generator
    GAN structure
    {
    {
    Training example
    B: Real
    answer
    Probability
    of fake
    A: Neural
    network’s
    guess
    “Here’s how you could make a
    better fake next time”
    

    View Slide

27. low-res bicubic GAN Real answer
    

    View Slide

28. GAN-trained generator
    Generator
    

    View Slide

29. GAN results
    original
    GAN
    Per-pixel diff
    neural network
    

    View Slide

30. GOOD
    JOB
    

    View Slide

31. GAN Real answer
    low-res
    

    View Slide

32. View Slide

33. View Slide

34. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
    Jun-Yan Zhu*, Taesung Park*, Phillip Isola, Alexei A. Efros
    

    View Slide

35. View Slide

36. Image-to-Image Translation with Conditional Adversarial Networks
    Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros
    https://affinelayer.com/pixsrv/
    

    View Slide

37. Image-to-Image Translation with Conditional Adversarial Networks
    Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros
    https://affinelayer.com/pixsrv/
    

    View Slide

38. StackGAN: Text to Photo-realistic Image Synthesis with
    Stacked Generative Adversarial Networks
    Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang,
    Xiaolei Huang, Xiaogang Wang, Dimitris Metaxas
    

    View Slide

39. Thanks!
    @geoffreylitt
    

    View Slide