Preslav Rachev
May 24, 2018
57

# ColorTransfer: Bringing Life to Black & White Photos

This is the slideshow from my presentation about using convolutional neural networks for adding color to black-and-white photos.

https://www.meetup.com/Hacking-Machine-Learning/events/249681231/

May 24, 2018

## Transcript

1. ColorTransfer
Bringing Life to Black & White Photos
@preslavrachev / 24.05.2018

2. Hi there! !
My name is Preslav
I am a full-stack backend
and mobile engineer…
… and a data scientist by
heart

3. I have always been
fascinated by the generative
nature of simple algorithms

4. All of my sketches shared the
rules and apply them over 1000s
of iterations

part, and this starts
sounding a lot like ML
Because it is

6. Colorizing BW
Photos

7. r/Colorization/

8. Source: How obsessive artists colorize old photos | Vox

9. X
Y
Y′

CNN

11. Source: Flickr

12. I mean, a Convolutional
Neural Network

13. Source: CS231n: Convolutional Neural Networks for Visual Recognition

14. Image Source: Machine Learning Guru

15. CNNs can be used for
both regression and
classiﬁcation

16. Challenge: Make a CNN that takes
1000s*M*N*256 of input points,
and generates 1*M*N*2563 outputs
Without spending a fortune on hardware

17. Lab to the rescue

18. Source: How to colorize black & white photos with just 100 lines of neural network code
Lab is a 3-dimensional representation of the visible color
space:
L (lightness 0-100) / a (green-red -127:127) / b (blue-yellow -127:127)

19. Put simply, by using Lab
instead of RGB, we can reduce
our problem space 256 times!
And get a BW layer for
free!

20. We start training…

21. …and, bummer!

22. It’s all … brown

23. Source: Wikipedia
Mean Square Error (MSE)

24. CNNs can be used for
both regression and
classiﬁcation

25. If we think about the problem as
loss function might have a better
chance at picking separate colors

26. OK, but choosing across
2562 == 216 classes? Are

27. Challenge: Turn the a and b
layers of the source images into
a single layer of class labels

28. Let’s quantize

29. Source: Wikipedia
10

30. 10
20

31. By quantizing to a grid of 10, we
lost a fraction of the visual
quality, but reduced the problem
space 100 times!

32. By using empirical ML methods,
such as clustering, we can
reduce the ab space even
further.

33. In fact, one of the cornerstone studies in
the ﬁeld, has managed to reduce the ab
space to a mere 313 distinct colors!
Recall that the non-quantized ab space
has 65536 possible values

34. Having reduced the color space, we can
now treat colorization like multi-label
classiﬁcation. Each pixel in the input
image would represent one of 313
classes:
0, 0, 0, 0, 0, … 0, 1, 0, 0, 0
M*N
0, 0, 1, 0, 0, … 0, 0, 0, 0, 0

35. Results
(Disclaimer: I used R. Zhang’s pre-trained
model)

36. My Progress

37. Open Source All the
Things!
• Stepping on the shoulders of giants in the ML community
• My models are still far from ideal, but I am trying to open-source every step of
this journey
• Repo: https://github.com/preslavrachev/nn-photo-colorization (https://bit.ly/
2GNCq2W)
• My main challenges right now are ﬁnding more time and computing resources.
• Using a free Kaggle kernel to do all the training
• That said, I would love to work on this project full-time
• Advice: Don’t be scared by ML. Choose a topic of your own liking, and it will be

38. Thank You!

39. • Zhang, Richard, Phillip Isola, and Alexei A. Efros. “Colorful image colorization.”
European Conference on Computer Vision. Springer, Cham, 2016.
• https://github.com/richzhang/colorization
• Wallner, E.: How to colorize black & white photos with just 100 lines of neural
network code Medium.com, Oct 29, 2017
• Dahl, R.: Automatic Colorization Tinyclouds.org, Jan, 2016
• Cheng, Z., Yang, Q., Sheng, B.: Deep colorization. In: Proceedings of the IEEE
International Conference on Computer Vision. (2015) 415–423
• Zhang, Richard, et al. "Real-time user-guided image colorization with learned deep
priors." arXiv preprint arXiv:1705.02999 (2017).
• https://github.com/junyanz/interactive-deep-colorization
• Karpathy, Andrej. "Cs231n convolutional neural networks for visual
recognition." Neural networks 1 (2016).