ColorTransfer: Bringing Life to Black & White Photos

Transcript

1. ColorTransfer Bringing Life to Black & White Photos @preslavrachev /

   24.05.2018

2. Hi there! ! My name is Preslav I am a

   full-stack backend and mobile engineer… … and a data scientist by heart

3. I have always been fascinated by the generative nature of

   simple algorithms
4. None

5. All of my sketches shared the same idea: start with

   simple rules and apply them over 1000s of iterations

6. Add the optimisation part, and this starts sounding a lot

   like ML Because it is

7. Colorizing BW Photos

8. r/Colorization/

9. Source: How obsessive artists colorize old photos | Vox

10. X Y Y′

11. Let’s start with a CNN

12. Source: Flickr

13. I mean, a Convolutional Neural Network

14. Source: CS231n: Convolutional Neural Networks for Visual Recognition

15. Image Source: Machine Learning Guru

16. CNNs can be used for both regression and classification

17. Challenge: Make a CNN that takes 1000s*M*N*256 of input points,

    and generates 1*M*N*2563 outputs Without spending a fortune on hardware

18. Lab to the rescue

19. Source: How to colorize black & white photos with just

    100 lines of neural network code Lab is a 3-dimensional representation of the visible color space: L (lightness 0-100) / a (green-red -127:127) / b (blue-yellow -127:127)

20. Put simply, by using Lab instead of RGB, we can

    reduce our problem space 256 times! And get a BW layer for free!

21. We start training…

22. …and, bummer!

23. It’s all … brown

24. None

25. Source: Wikipedia Mean Square Error (MSE)

26. CNNs can be used for both regression and classification

27. If we think about the problem as classification instead, an

    appropriate loss function might have a better chance at picking separate colors

28. OK, but choosing across 2562 == 216 classes? Are you

    out of your mind?

29. Challenge: Turn the a and b layers of the source

    images into a single layer of class labels

30. Let’s quantize

31. Source: Wikipedia 10

32. 10 20

33. By quantizing to a grid of 10, we lost a

    fraction of the visual quality, but reduced the problem space 100 times!

34. By using empirical ML methods, such as clustering, we can

    reduce the ab space even further.

35. In fact, one of the cornerstone studies in the field,

    has managed to reduce the ab space to a mere 313 distinct colors! Recall that the non-quantized ab space has 65536 possible values

36. Having reduced the color space, we can now treat colorization

    like multi-label classification. Each pixel in the input image would represent one of 313 classes: 0, 0, 0, 0, 0, … 0, 1, 0, 0, 0 M*N 0, 0, 1, 0, 0, … 0, 0, 0, 0, 0

37. Results (Disclaimer: I used R. Zhang’s pre-trained model)

38. None
39. None

40. My Progress

41. None
42. None

43. Open Source All the Things! • Stepping on the shoulders

    of giants in the ML community • My models are still far from ideal, but I am trying to open-source every step of this journey • Repo: https://github.com/preslavrachev/nn-photo-colorization (https://bit.ly/ 2GNCq2W) • My main challenges right now are finding more time and computing resources. • Using a free Kaggle kernel to do all the training • That said, I would love to work on this project full-time • Advice: Don’t be scared by ML. Choose a topic of your own liking, and it will be your driving force.

44. Thank You!

45. • Zhang, Richard, Phillip Isola, and Alexei A. Efros. “Colorful

    image colorization.” European Conference on Computer Vision. Springer, Cham, 2016. • https://github.com/richzhang/colorization • Wallner, E.: How to colorize black & white photos with just 100 lines of neural network code Medium.com, Oct 29, 2017 • Dahl, R.: Automatic Colorization Tinyclouds.org, Jan, 2016 • Cheng, Z., Yang, Q., Sheng, B.: Deep colorization. In: Proceedings of the IEEE International Conference on Computer Vision. (2015) 415–423 • Zhang, Richard, et al. "Real-time user-guided image colorization with learned deep priors." arXiv preprint arXiv:1705.02999 (2017). • https://github.com/junyanz/interactive-deep-colorization • Karpathy, Andrej. "Cs231n convolutional neural networks for visual recognition." Neural networks 1 (2016).