ColorTransfer Bringing Life to Black & White Photos @preslavrachev / 24.05.2018 

Hi there! ! My name is Preslav I am a full-stack backend and mobile engineer… … and a data scientist by heart 

I have always been fascinated by the generative nature of simple algorithms 

No content

All of my sketches shared the same idea: start with simple rules and apply them over 1000s of iterations 

Add the optimisation part, and this starts sounding a lot like ML Because it is 

Colorizing BW Photos 

r/Colorization/ 

Source: How obsessive artists colorize old photos | Vox 

X Y Y′ 

Let’s start with a CNN 

Source: Flickr 

I mean, a Convolutional Neural Network 

Source: CS231n: Convolutional Neural Networks for Visual Recognition 

Image Source: Machine Learning Guru 

CNNs can be used for both regression and classification 

Challenge: Make a CNN that takes 1000s*M*N*256 of input points, and generates 1*M*N*2563 outputs Without spending a fortune on hardware 

Lab to the rescue 

Source: How to colorize black & white photos with just 100 lines of neural network code Lab is a 3-dimensional representation of the visible color space: L (lightness 0-100) / a (green-red -127:127) / b (blue-yellow -127:127) 

Put simply, by using Lab instead of RGB, we can reduce our problem space 256 times! And get a BW layer for free! 

We start training… 

…and, bummer! 

It’s all … brown 

No content

Source: Wikipedia Mean Square Error (MSE) 

CNNs can be used for both regression and classification 

If we think about the problem as classification instead, an appropriate loss function might have a better chance at picking separate colors 

OK, but choosing across 2562 == 216 classes? Are you out of your mind? 

Challenge: Turn the a and b layers of the source images into a single layer of class labels 

Let’s quantize 

Source: Wikipedia 10 

10 20 

By quantizing to a grid of 10, we lost a fraction of the visual quality, but reduced the problem space 100 times! 

By using empirical ML methods, such as clustering, we can reduce the ab space even further. 

In fact, one of the cornerstone studies in the field, has managed to reduce the ab space to a mere 313 distinct colors! Recall that the non-quantized ab space has 65536 possible values 

Having reduced the color space, we can now treat colorization like multi-label classification. Each pixel in the input image would represent one of 313 classes: 0, 0, 0, 0, 0, … 0, 1, 0, 0, 0 M*N 0, 0, 1, 0, 0, … 0, 0, 0, 0, 0 

Results (Disclaimer: I used R. Zhang’s pre-trained model) 

No content

No content

My Progress 

No content

No content

Open Source All the Things! • Stepping on the shoulders of giants in the ML community • My models are still far from ideal, but I am trying to open-source every step of this journey • Repo: https://github.com/preslavrachev/nn-photo-colorization (https://bit.ly/ 2GNCq2W) • My main challenges right now are finding more time and computing resources. • Using a free Kaggle kernel to do all the training • That said, I would love to work on this project full-time • Advice: Don’t be scared by ML. Choose a topic of your own liking, and it will be your driving force. 

Thank You! 

• Zhang, Richard, Phillip Isola, and Alexei A. Efros. “Colorful image colorization.” European Conference on Computer Vision. Springer, Cham, 2016. • https://github.com/richzhang/colorization • Wallner, E.: How to colorize black & white photos with just 100 lines of neural network code Medium.com, Oct 29, 2017 • Dahl, R.: Automatic Colorization Tinyclouds.org, Jan, 2016 • Cheng, Z., Yang, Q., Sheng, B.: Deep colorization. In: Proceedings of the IEEE International Conference on Computer Vision. (2015) 415–423 • Zhang, Richard, et al. "Real-time user-guided image colorization with learned deep priors." arXiv preprint arXiv:1705.02999 (2017). • https://github.com/junyanz/interactive-deep-colorization • Karpathy, Andrej. "Cs231n convolutional neural networks for visual recognition." Neural networks 1 (2016).