A story of comics, neural networks, and Android!

Pedro Gómez - [email protected] - @pedro_g_s - github.com/pedrovgs - Questions
=> sli.do #madgStylesAndComics A story of comics, neural networks & Android! Pedro Gómez Senior Software Engineer at Karumi [email protected] @pedro_g_s github.com/pedrovgs

=> sli.do #madgStylesAndComics

=> sli.do #madgStylesAndComics Questions? https://sli.do #madgStylesAndComics

=> sli.do #madgStylesAndComics This story starts with an awesome app and one of my favorite hobbies

=> sli.do #madgStylesAndComics Unfortunately, we don’t have this metadata and comic pages are not always composed by rectangles

=> sli.do #madgStylesAndComics How does Panels do it?

=> sli.do #madgStylesAndComics Current solutions are based on computer vision

=> sli.do #madgStylesAndComics OpenCV pros and cons • Already implemented solution. • Maintained by smart people. • Works with simple pages. • Really fast. • Obj-c bridging hell. • Doesn’t work with “cool” pages. • Doesn’t work with covers. • Doesn’t work with back covers. • Does not respect artist reading design.

=> sli.do #madgStylesAndComics Can we do it better?

=> sli.do #madgStylesAndComics “I’m sure we can do it better with Artiﬁcial Intelligence” Pedro Gómez, 2018

=> sli.do #madgStylesAndComics Unfortunately, it’s not that simple

=> sli.do #madgStylesAndComics What would we need to solve this using AI? • State of the art review. • Learn Python. • Learn about ML. • Learn about TensorFlow. • Find a model to use in TF. • Create a data set. • Train and export the model. • Consume the model from Android/iOS code. • Integrate the libraries into Panels.

=> sli.do #madgStylesAndComics Let’s start with the hardest part Machine Learning

=> sli.do #madgStylesAndComics Convolutional neural networks to the rescue! Supervised deep learning models frequently used for image classiﬁcation, segmentation and video processing.

=> sli.do #madgStylesAndComics U-Net deep learning model U-Net is a convolutional neural network that was developed for biomedical image segmentation at the Computer Science Department of the University of Freiburg, Germany.

=> sli.do #madgStylesAndComics Machine Learning ➡ ➡

=> sli.do #madgStylesAndComics Using a supervised model means we will need a training dataset ‍♀

=> sli.do #madgStylesAndComics Machine Learning ➡ ➡

=> sli.do #madgStylesAndComics Machine Learning ➡ ➡ Label 0 - Background. Label 1 - Border Label 2 -> Content

=> sli.do #madgStylesAndComics Machine Learning

=> sli.do #madgStylesAndComics Is it that simple?

=> sli.do #madgStylesAndComics Problems • I had no idea about Python. • My ML knowledge was poor. • I had no idea about TF. • We need to create a dataset. • We needed to measure our accuracy. • Our ﬁrst run gave us ~5% acc for the border. • Even if we get the model ready we still need to ﬁnd the panels between the numbers

=> sli.do #madgStylesAndComics Solutions • I learned python. • I studied some ML. • I reviewed some TF tutorials. • I manually created the masks for about than 500 pages. • I started measuring properly the accuracy per label. • We would improve our model/training to get up to 80% accuracy. • We would take care of the panels later.

=> sli.do #madgStylesAndComics ML accuracy improvements • Conﬁgure different metrics. • Increase training dataset size from 50 pages to 500 (Google uses 3k for any sample project). • Increase neural network size from 128 to 224. • Balance our dataset. • Run some random experiments and discard the results.

=> sli.do #madgStylesAndComics Once the model was ready, it was time to think about Android & iOS

=> sli.do #madgStylesAndComics Using the model we created using Python from Android & iOS

=> sli.do #madgStylesAndComics Using the model in mobile ➡ ➡ Label 0 - Background. Label 1 - Border Label 2 -> Content

=> sli.do #madgStylesAndComics Using TensorFlow lite • We optimized and increased the tﬂite size model from 24mb up to 12mb. • We created the code needed to extract the image features from mobile. • Using some Kotlin code we evaluated the model and we got a wonderful matrix of numbers.

=> sli.do #madgStylesAndComics Once we got the output matrix it was time to start working with the computer vision algorithms!

=> sli.do #madgStylesAndComics Using the model in mobile ➡

=> sli.do #madgStylesAndComics Connected Component Labeling aka the slowest algorithm I’ve ever implemented

=> sli.do #madgStylesAndComics The ﬁrst Kotlin version needed 14 seconds to analyze a page

=> sli.do #madgStylesAndComics We ﬁnally ported the algorithm to C++ and we are now able to analyze a page in less than 500ms ⚡

=> sli.do #madgStylesAndComics Android Code?

=> sli.do #madgStylesAndComics Cool facts

=> sli.do #madgStylesAndComics +-500 LOC +-500 comic pagespages

=> sli.do #madgStylesAndComics C++ code is fast, however the main beneﬁt for us is portability!

=> sli.do #madgStylesAndComics Some Android/iOS details took us more time to implement than the ML part ⏳

=> sli.do #madgStylesAndComics Thanks to this solution we expected Panels’ binary size to be lower but it is not

=> sli.do #madgStylesAndComics DeepPanel value is not only the code but the dataset

=> sli.do #madgStylesAndComics Our model doesn’t ﬁnd panels. DeepPanel is able to read a comic!

=> sli.do #madgStylesAndComics The best solution is not based on computer science but asking the artist how we should read the page

=> sli.do #madgStylesAndComics It is a great example of ML bias

=> sli.do #madgStylesAndComics There is room for improvements

=> sli.do #madgStylesAndComics Available in Maven Central and CocoaPods

=> sli.do #madgStylesAndComics References • DeepPanel - https://github.com/pedrovgs/DeepPanel • DeepPanelAndroid - https://github.com/pedrovgs/DeepPanelAndroid • DeepPaneliOS - https://github.com/pedrovgs/DeepPaneliOS • Panels - https://twitter.com/Panels_ink • TensorFlow - https://www.tensorﬂow.org/ • U-Net model - https://en.wikipedia.org/wiki/U-Net • CCL - https://en.wikipedia.org/wiki/Connected-component_labeling • Convolutional Neural Networks - https://en.wikipedia.org/wiki/Convolutional_neural_network • Dot CSV Video - https://www.youtube.com/watch?v=V8j1oENVz00

=> sli.do #madgStylesAndComics Acknowledgements • Asun. • Rafael Muñóz. • Fillito. • Victor Baro.

=> sli.do #madgStylesAndComics Questions?

A story of comics, neural networks, and Android!

A story of comics, neural networks, and Android!

More Decks by Karumi

Other Decks in Programming

Featured

Transcript