ColorTransfer: Bringing Life to Black & White Photos

This is the slideshow from my presentation about using convolutional neural networks for adding color to black-and-white photos.

https://www.meetup.com/Hacking-Machine-Learning/events/249681231/

Transcript

1. ColorTransfer Bringing Life to Black & White Photos @preslavrachev / 24.05.2018
2. Hi there! ! My name is Preslav I am a full-stack backend and mobile engineer… … and a data scientist by heart
3. I have always been fascinated by the generative nature of simple algorithms
5. All of my sketches shared the same idea: start with simple rules and apply them over 1000s of iterations
6. Add the optimisation part, and this starts sounding a lot like ML Because it is
7. Colorizing BW Photos
8. r/Colorization/
9. Source: How obsessive artists colorize old photos | Vox
10. X Y Y′
11. Let’s start with a CNN
12. Source: Flickr
13. I mean, a Convolutional Neural Network
14. Source: CS231n: Convolutional Neural Networks for Visual Recognition
15. Image Source: Machine Learning Guru
16. CNNs can be used for both regression and classification
17. Challenge: Make a CNN that takes 1000s*M*N*256 of input points, and generates 1*M*N*2563 outputs Without spending a fortune on hardware
18. Lab to the rescue
19. Source: How to colorize black & white photos with just 100 lines of neural network code Lab is a 3-dimensional representation of the visible color space: L (lightness 0-100) / a (green-red -127:127) / b (blue-yellow -127:127)
20. Put simply, by using Lab instead of RGB, we can reduce our problem space 256 times! And get a BW layer for free!
21. We start training…
22. …and, bummer!
23. It’s all … brown
25. Source: Wikipedia Mean Square Error (MSE)
26. CNNs can be used for both regression and classification
27. If we think about the problem as classification instead, an appropriate loss function might have a better chance at picking separate colors
28. OK, but choosing across 2562 == 216 classes? Are you out of your mind?
29. Challenge: Turn the a and b layers of the source images into a single layer of class labels
30. Let’s quantize
31. Source: Wikipedia 10
32. 10 20
33. By quantizing to a grid of 10, we lost a fraction of the visual quality, but reduced the problem space 100 times!
34. By using empirical ML methods, such as clustering, we can reduce the ab space even further.
35. In fact, one of the cornerstone studies in the field, has managed to reduce the ab space to a mere 313 distinct colors! Recall that the non-quantized ab space has 65536 possible values
36. Having reduced the color space, we can now treat colorization like multi-label classification. Each pixel in the input image would represent one of 313 classes: 0, 0, 0, 0, 0, … 0, 1, 0, 0, 0 M*N 0, 0, 1, 0, 0, … 0, 0, 0, 0, 0
37. Results (Disclaimer: I used R. Zhang’s pre-trained model)
40. My Progress
43. Open Source All the Things! • Stepping on the shoulders of giants in the ML community • My models are still far from ideal, but I am trying to open-source every step of this journey • Repo: https://github.com/preslavrachev/nn-photo-colorization (https://bit.ly/ 2GNCq2W) • My main challenges right now are finding more time and computing resources. • Using a free Kaggle kernel to do all the training • That said, I would love to work on this project full-time • Advice: Don’t be scared by ML. Choose a topic of your own liking, and it will be your driving force.
44. Thank You!
45. • Zhang, Richard, Phillip Isola, and Alexei A. Efros. “Colorful image colorization.” European Conference on Computer Vision. Springer, Cham, 2016. • https://github.com/richzhang/colorization • Wallner, E.: How to colorize black & white photos with just 100 lines of neural network code Medium.com, Oct 29, 2017 • Dahl, R.: Automatic Colorization Tinyclouds.org, Jan, 2016 • Cheng, Z., Yang, Q., Sheng, B.: Deep colorization. In: Proceedings of the IEEE International Conference on Computer Vision. (2015) 415–423 • Zhang, Richard, et al. "Real-time user-guided image colorization with learned deep priors." arXiv preprint arXiv:1705.02999 (2017). • https://github.com/junyanz/interactive-deep-colorization • Karpathy, Andrej. "Cs231n convolutional neural networks for visual recognition." Neural networks 1 (2016).