Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ColorTransfer: Bringing Life to Black & White Photos

ColorTransfer: Bringing Life to Black & White Photos

This is the slideshow from my presentation about using convolutional neural networks for adding color to black-and-white photos.

https://www.meetup.com/Hacking-Machine-Learning/events/249681231/

Fd47e6bba1f42ecacf2e7af9e4c5fb52?s=128

Preslav Rachev

May 24, 2018
Tweet

Transcript

  1. ColorTransfer Bringing Life to Black & White Photos @preslavrachev /

    24.05.2018
  2. Hi there! ! My name is Preslav I am a

    full-stack backend and mobile engineer… … and a data scientist by heart
  3. I have always been fascinated by the generative nature of

    simple algorithms
  4. None
  5. All of my sketches shared the same idea: start with

    simple rules and apply them over 1000s of iterations
  6. Add the optimisation part, and this starts sounding a lot

    like ML Because it is
  7. Colorizing BW Photos

  8. r/Colorization/

  9. Source: How obsessive artists colorize old photos | Vox

  10. X Y Y′

  11. Let’s start with a CNN

  12. Source: Flickr

  13. I mean, a Convolutional Neural Network

  14. Source: CS231n: Convolutional Neural Networks for Visual Recognition

  15. Image Source: Machine Learning Guru

  16. CNNs can be used for both regression and classification

  17. Challenge: Make a CNN that takes 1000s*M*N*256 of input points,

    and generates 1*M*N*2563 outputs Without spending a fortune on hardware
  18. Lab to the rescue

  19. Source: How to colorize black & white photos with just

    100 lines of neural network code Lab is a 3-dimensional representation of the visible color space: L (lightness 0-100) / a (green-red -127:127) / b (blue-yellow -127:127)
  20. Put simply, by using Lab instead of RGB, we can

    reduce our problem space 256 times! And get a BW layer for free!
  21. We start training…

  22. …and, bummer!

  23. It’s all … brown

  24. None
  25. Source: Wikipedia Mean Square Error (MSE)

  26. CNNs can be used for both regression and classification

  27. If we think about the problem as classification instead, an

    appropriate loss function might have a better chance at picking separate colors
  28. OK, but choosing across 2562 == 216 classes? Are you

    out of your mind?
  29. Challenge: Turn the a and b layers of the source

    images into a single layer of class labels
  30. Let’s quantize

  31. Source: Wikipedia 10

  32. 10 20

  33. By quantizing to a grid of 10, we lost a

    fraction of the visual quality, but reduced the problem space 100 times!
  34. By using empirical ML methods, such as clustering, we can

    reduce the ab space even further.
  35. In fact, one of the cornerstone studies in the field,

    has managed to reduce the ab space to a mere 313 distinct colors! Recall that the non-quantized ab space has 65536 possible values
  36. Having reduced the color space, we can now treat colorization

    like multi-label classification. Each pixel in the input image would represent one of 313 classes: 0, 0, 0, 0, 0, … 0, 1, 0, 0, 0 M*N 0, 0, 1, 0, 0, … 0, 0, 0, 0, 0
  37. Results (Disclaimer: I used R. Zhang’s pre-trained model)

  38. None
  39. None
  40. My Progress

  41. None
  42. None
  43. Open Source All the Things! • Stepping on the shoulders

    of giants in the ML community • My models are still far from ideal, but I am trying to open-source every step of this journey • Repo: https://github.com/preslavrachev/nn-photo-colorization (https://bit.ly/ 2GNCq2W) • My main challenges right now are finding more time and computing resources. • Using a free Kaggle kernel to do all the training • That said, I would love to work on this project full-time • Advice: Don’t be scared by ML. Choose a topic of your own liking, and it will be your driving force.
  44. Thank You!

  45. • Zhang, Richard, Phillip Isola, and Alexei A. Efros. “Colorful

    image colorization.” European Conference on Computer Vision. Springer, Cham, 2016. • https://github.com/richzhang/colorization • Wallner, E.: How to colorize black & white photos with just 100 lines of neural network code Medium.com, Oct 29, 2017 • Dahl, R.: Automatic Colorization Tinyclouds.org, Jan, 2016 • Cheng, Z., Yang, Q., Sheng, B.: Deep colorization. In: Proceedings of the IEEE International Conference on Computer Vision. (2015) 415–423 • Zhang, Richard, et al. "Real-time user-guided image colorization with learned deep priors." arXiv preprint arXiv:1705.02999 (2017). • https://github.com/junyanz/interactive-deep-colorization • Karpathy, Andrej. "Cs231n convolutional neural networks for visual recognition." Neural networks 1 (2016).