Keynote: Scaling Deep Learning Models in Production Using Kubernetes

KubeCon + CloudNativeCon 2018 - Copenhagen

Youtube Video Link - https://www.youtube.com/watch?v=gcij93d9st8

Keynote: Scaling Deep Learning Models in Production Using Kubernetes - Sahil Dua, Software Developer, Booking.com

While there are a lot of machine learning frameworks and libraries available, putting the models in production at large scale is still a challenge. Sahil would like to talk about how they took on the challenge of deploying deep learning models in production: how they chose their tools and developed their internal deep learning infrastructure using Kubernetes. He will cover how they do model training in Docker containers, distributed TensorFlow training in a cluster of containers, automated re-training of models and finally, the deployment of models to serve predictions. At the large scale which they operate on, nothing comes easy. He will also talk about how they optimize their model predictions infrastructure for latency or throughput depending on the use case.

"About Sahil
Sahil is a software developer at Booking.com. He has been involved in leveraging container infrastructure to help Booking.com’s internal teams in taking advantage of deep learning techniques at scale. An open source software enthusiast, Sahil is a core contributor and community leader for DuckDuckGo's open source organization. Besides that he is one of the contributors to the Git project, pandas - open source data analysis library, Linguist project by GitHub and Go-GitHub project by Google. Sahil has been actively speaking at conferences in last one year including FOSDEM, EuroPython and SIGNAL 2017 to name some of them."

Transcript

1. @sahildua2305 @sahildua2305 Scaling Deep Learning Models in Production using Kubernetes Sahil Dua
2. @sahildua2305 @sahildua2305 whoami ➔Software Developer @ Booking.com ➔Previously - Deep Learning Infrastructure ➔Open Source Contributor (Git, Pandas, Kinto, go-github, Hound, etc.) ➔Tech Speaker
3. @sahildua2305 @sahildua2305 ➔Deep Learning at Booking.com ➔Training Models ➔Serving Predictions Agenda
4. @sahildua2305 @sahildua2305 Deep Learning at Booking.com
5. @sahildua2305 @sahildua2305 1.4 million+ active properties in 220+ countries 1,500,000+ room nights booked every 24 hours Scale highlights.
6. @sahildua2305 @sahildua2305 Deep Learning ➔Image tagging ➔Translations ➔Ads bidding ➔...
7. @sahildua2305 @sahildua2305 Image Tagging
8. @sahildua2305 @sahildua2305 Image Tagging
9. @sahildua2305 @sahildua2305 Image Tagging Sea view: 6.38 Balcony/Terrace: 4.82 Photo of the whole room: 4.21 Bed: 3.47 Decorative details: 3.15 Seating area: 2.70
10. @sahildua2305 @sahildua2305
11. @sahildua2305 @sahildua2305 Image Tagging Using the image tag information in the right context Swimming pool, Breakfast Buffet, etc.
12. @sahildua2305 @sahildua2305 What’s so special?
13. @sahildua2305 @sahildua2305 Deep Learning workload ➔Computationally intensive workload ➔Often not highly parallelizable algorithms ➔10 to 100 GBs of data
14. @sahildua2305 @sahildua2305 Why Kubernetes? ➔Isolation ➔Elasticity ➔Flexibility
15. @sahildua2305 @sahildua2305 Training with Kubernetes ➔Base images with ML frameworks ◆ TensorFlow, Torch, VowpalWabbit, etc. ➔Training code is installed at start time ➔Data access - Hadoop (or PVs)
16. @sahildua2305 @sahildua2305 Startup Code Training pod .. start.sh train.py evaluate.py
17. @sahildua2305 @sahildua2305 Startup Data .. start.sh train.py evaluate.py PV Training pod
18. @sahildua2305 @sahildua2305 Streaming logs back Logs .. start.sh train.py evaluate.py Training pod
19. @sahildua2305 @sahildua2305 Exports the model .. start.sh train.py evaluate.py Training pod model
20. @sahildua2305 @sahildua2305 Serving predictions
21. @sahildua2305 @sahildua2305 Serving Predictions Model Client Input Features Prediction
22. @sahildua2305 @sahildua2305 Serving Predictions Model 1 Client Input Features Prediction Model X Client Input Features Prediction
23. @sahildua2305 @sahildua2305 Serving Predictions Model 1 Client Input Features Prediction Model X Client Input Features Prediction
24. @sahildua2305 @sahildua2305 Serving Predictions ➔ Stateless app with common code ➔ Containerized ➔ No model in image ➔ REST API for predictions
25. @sahildua2305 @sahildua2305 Stateless app with common code App
26. @sahildua2305 @sahildua2305 Get trained model from Hadoop App model
27. @sahildua2305 @sahildua2305 Load model in memory App model
28. @sahildua2305 @sahildua2305 Expose HTTP API App model Client Input Features Prediction
29. @sahildua2305 @sahildua2305 App Client Serving Predictions Input Features Prediction model
30. @sahildua2305 @sahildua2305 Serving Predictions Client Input Features Prediction
31. @sahildua2305 @sahildua2305 Serving Predictions Client Input Features Prediction Client Input Features Prediction
32. @sahildua2305 @sahildua2305 Deploying a new model ➔ Create a new Deployment ➔ Create a new Service ➔ Wait for liveness/readiness probe
33. @sahildua2305 @sahildua2305 Summary ➔ Applications of deep learning ➔ Training models ➔ Deploying models to serve predictions
34. @sahildua2305 @sahildua2305 Want to get in touch? LinkedIn / Twitter / GitHub @SahilDua2305
35. @sahildua2305 @sahildua2305 THANK YOU @sahildua2305