Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ENHANCE!! - Upscaling Images with Neural Networks

ENHANCE!! - Upscaling Images with Neural Networks

When characters on a TV show “enhance!” a blurry image, you probably laugh and tell your friends that it’s impossible to do that in real life. But over the past year, deep learning research has actually made this kind of possible! How will you explain this to your friends!?

In this talk, you’ll get an intuitive introduction to generative adversarial networks, a new machine learning technique that’s surprisingly good at upscaling images. You’ll learn how these systems are inspired by human art forgers, and how you can use them to do other things like transform a horse into a zebra, convert a sentence into a photo, and so much more!

Geoffrey Litt

May 07, 2017
Tweet

More Decks by Geoffrey Litt

Other Decks in Technology

Transcript

  1. ENHANCE!!
    @geoffreylitt

    View full-size slide

  2. Entertainment
    Accuracy of
    portraying
    computers

    View full-size slide

  3. ENHANCE, PLEASE

    View full-size slide

  4. Bicubic interpolation

    View full-size slide

  5. Bicubic interpolation

    View full-size slide

  6. NEURAL
    NETWORKS!!!

    View full-size slide

  7. Neural network upscaling

    View full-size slide

  8. Neural network training
    { }
    , { }
    ,
    Training
    data

    View full-size slide

  9. Neural network training
    A: Neural
    network’s
    guess
    B: Real answer
    Per-pixel
    subtraction
    Add up all the
    pixels Total
    per-pixel
    difference
    Minimize this!

    View full-size slide

  10. Neural network training
    Loss function:
    “Minimize the total per-pixel
    difference between the two
    images”

    View full-size slide

  11. Neural network
    Code: https://github.com/david-gpu/srez

    View full-size slide

  12. Neural network
    Bicubic interpolation

    View full-size slide

  13. MORE.
    ENHANCE.

    View full-size slide

  14. A: Neural
    network’s
    guess
    B: Real answer
    Per-pixel
    subtraction

    View full-size slide

  15. A: Neural
    network’s guess
    B: Real answer
    Really high loss

    View full-size slide

  16. A: Neural
    network’s guess
    B: Real answer
    Lower loss

    View full-size slide

  17. A new loss function
    ???
    “Minimize the total per-pixel difference
    between your guess and the real
    answer…
    But also make your guess look sharp
    and realistic”
    Loss =
    Total Per-pixel difference +

    View full-size slide

  18. GENERATIVE
    ADVERSARIAL
    NETWORKS

    View full-size slide

  19. Adversarial games
    I’ll develop better
    counterfeit detection
    techniques!
    I’ll develop better
    counterfeiting
    techniques!

    View full-size slide

  20. Generator
    GAN structure
    {
    {
    Training example
    B: Real
    answer
    A: Neural
    network’s
    guess

    View full-size slide

  21. Discriminator
    Generator
    GAN structure
    {
    {
    Training example
    B: Real
    answer
    Probability
    of fake
    Goal: differentiate fake vs.
    real with 100% accuracy
    Goal:
    Minimize discriminator
    accuracy
    A: Neural
    network’s
    guess

    View full-size slide

  22. Discriminator
    Generator
    GAN structure
    {
    {
    Training example
    B: Real
    answer
    Probability
    of fake
    A: Neural
    network’s
    guess
    “Here’s how you could make a
    better fake next time”

    View full-size slide

  23. low-res bicubic GAN Real answer

    View full-size slide

  24. GAN-trained generator
    Generator

    View full-size slide

  25. GAN results
    original
    GAN
    Per-pixel diff
    neural network

    View full-size slide

  26. GAN Real answer
    low-res

    View full-size slide

  27. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
    Jun-Yan Zhu*, Taesung Park*, Phillip Isola, Alexei A. Efros

    View full-size slide

  28. Image-to-Image Translation with Conditional Adversarial Networks
    Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros
    https://affinelayer.com/pixsrv/

    View full-size slide

  29. Image-to-Image Translation with Conditional Adversarial Networks
    Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros
    https://affinelayer.com/pixsrv/

    View full-size slide

  30. StackGAN: Text to Photo-realistic Image Synthesis with
    Stacked Generative Adversarial Networks
    Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang,
    Xiaolei Huang, Xiaogang Wang, Dimitris Metaxas

    View full-size slide

  31. Thanks!
    @geoffreylitt

    View full-size slide