Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ColorTransfer: Bringing Life to Black & White Photos

ColorTransfer: Bringing Life to Black & White Photos

This is the slideshow from my presentation about using convolutional neural networks for adding color to black-and-white photos.

https://www.meetup.com/Hacking-Machine-Learning/events/249681231/

Preslav Rachev

May 24, 2018
Tweet

More Decks by Preslav Rachev

Other Decks in Programming

Transcript

  1. ColorTransfer
    Bringing Life to Black & White Photos
    @preslavrachev / 24.05.2018

    View full-size slide

  2. Hi there! !
    My name is Preslav
    I am a full-stack backend
    and mobile engineer…
    … and a data scientist by
    heart

    View full-size slide

  3. I have always been
    fascinated by the generative
    nature of simple algorithms

    View full-size slide

  4. All of my sketches shared the
    same idea: start with simple
    rules and apply them over 1000s
    of iterations

    View full-size slide

  5. Add the optimisation
    part, and this starts
    sounding a lot like ML
    Because it is

    View full-size slide

  6. Colorizing BW
    Photos

    View full-size slide

  7. r/Colorization/

    View full-size slide

  8. Source: How obsessive artists colorize old photos | Vox

    View full-size slide

  9. Let’s start with a
    CNN

    View full-size slide

  10. Source: Flickr

    View full-size slide

  11. I mean, a Convolutional
    Neural Network

    View full-size slide

  12. Source: CS231n: Convolutional Neural Networks for Visual Recognition

    View full-size slide

  13. Image Source: Machine Learning Guru

    View full-size slide

  14. CNNs can be used for
    both regression and
    classification

    View full-size slide

  15. Challenge: Make a CNN that takes
    1000s*M*N*256 of input points,
    and generates 1*M*N*2563 outputs
    Without spending a fortune on hardware

    View full-size slide

  16. Lab to the rescue

    View full-size slide

  17. Source: How to colorize black & white photos with just 100 lines of neural network code
    Lab is a 3-dimensional representation of the visible color
    space:
    L (lightness 0-100) / a (green-red -127:127) / b (blue-yellow -127:127)

    View full-size slide

  18. Put simply, by using Lab
    instead of RGB, we can reduce
    our problem space 256 times!
    And get a BW layer for
    free!

    View full-size slide

  19. We start training…

    View full-size slide

  20. …and, bummer!

    View full-size slide

  21. It’s all … brown

    View full-size slide

  22. Source: Wikipedia
    Mean Square Error (MSE)

    View full-size slide

  23. CNNs can be used for
    both regression and
    classification

    View full-size slide

  24. If we think about the problem as
    classification instead, an appropriate
    loss function might have a better
    chance at picking separate colors

    View full-size slide

  25. OK, but choosing across
    2562 == 216 classes? Are
    you out of your mind?

    View full-size slide

  26. Challenge: Turn the a and b
    layers of the source images into
    a single layer of class labels

    View full-size slide

  27. Let’s quantize

    View full-size slide

  28. Source: Wikipedia
    10

    View full-size slide

  29. By quantizing to a grid of 10, we
    lost a fraction of the visual
    quality, but reduced the problem
    space 100 times!

    View full-size slide

  30. By using empirical ML methods,
    such as clustering, we can
    reduce the ab space even
    further.

    View full-size slide

  31. In fact, one of the cornerstone studies in
    the field, has managed to reduce the ab
    space to a mere 313 distinct colors!
    Recall that the non-quantized ab space
    has 65536 possible values

    View full-size slide

  32. Having reduced the color space, we can
    now treat colorization like multi-label
    classification. Each pixel in the input
    image would represent one of 313
    classes:
    0, 0, 0, 0, 0, … 0, 1, 0, 0, 0
    M*N
    0, 0, 1, 0, 0, … 0, 0, 0, 0, 0

    View full-size slide

  33. Results
    (Disclaimer: I used R. Zhang’s pre-trained
    model)

    View full-size slide

  34. Open Source All the
    Things!
    • Stepping on the shoulders of giants in the ML community
    • My models are still far from ideal, but I am trying to open-source every step of
    this journey
    • Repo: https://github.com/preslavrachev/nn-photo-colorization (https://bit.ly/
    2GNCq2W)
    • My main challenges right now are finding more time and computing resources.
    • Using a free Kaggle kernel to do all the training
    • That said, I would love to work on this project full-time
    • Advice: Don’t be scared by ML. Choose a topic of your own liking, and it will be
    your driving force.

    View full-size slide

  35. • Zhang, Richard, Phillip Isola, and Alexei A. Efros. “Colorful image colorization.”
    European Conference on Computer Vision. Springer, Cham, 2016.
    • https://github.com/richzhang/colorization
    • Wallner, E.: How to colorize black & white photos with just 100 lines of neural
    network code Medium.com, Oct 29, 2017
    • Dahl, R.: Automatic Colorization Tinyclouds.org, Jan, 2016
    • Cheng, Z., Yang, Q., Sheng, B.: Deep colorization. In: Proceedings of the IEEE
    International Conference on Computer Vision. (2015) 415–423
    • Zhang, Richard, et al. "Real-time user-guided image colorization with learned deep
    priors." arXiv preprint arXiv:1705.02999 (2017).
    • https://github.com/junyanz/interactive-deep-colorization
    • Karpathy, Andrej. "Cs231n convolutional neural networks for visual
    recognition." Neural networks 1 (2016).

    View full-size slide