Scaling Machine Learning in the Cloud with Kubernetes

Building any production-ready machine learning system is complex. You have to manage services and tools that often don’t play nice with each other. And when they do you have to spend time manually tweaking deployments and hand rolling solutions before a single model can be tested. Worse, these hand-rolled solutions are so tied to your production cluster that it’s impossible to run your code locally making it even harder to spot bugs.

In this workshop, you’ll learn how to leverage Kubernetes to deploying complex workloads in the cloud, on bare metal and locally. You’ll learn how Kubernetes provides a fast iteration cycle, flexible scalability, and a lack of boilerplate which makes it ideal for most of the machine learning experiments.

Transcript

2. scaling machine learning in the ☁ with kubernetes
4. daniele @danielepolencic
5. building — things
6. building — things teach kubernetes
7. building — things teach kubernetes data scientist
8. building — things teach kubernetes ❌ data scientist
9. salman @soulmaniqbal
10. software developer
11. software developer — building on ☁
12. software developer — building things on ☁ machine learning
14. 13 layer policy network
15. 13 layer policy network 30 million positions
16. 13 layer policy network 30 million positions 1,920 cpus & 280 gpus
17. challenges
18. 1. compile once run anywhere
19. 1. compile once run anywhere
20. 1. works on my machine
22. ◻ big data ◻ scale with team and model ◻ cost for money ◻ quick fix
23. ❌ big data ◻ scale with team and model ◻ cost for money ◻ quick fix
24. ❌ big data ❌ scale with team and model ◻ cost for money ◻ quick fix
25. ❌ big data ❌ scale with team and model ❌ cost for money ◻ quick fix
26. ❌ big data ❌ scale with team and model ❌ cost for money ✅ quick fix
27. 2. complexity
28. Training Data
29. Training Data Test Data
30. Training Data Test Data Model
31. Training Data Test Data Model Evaluate
32. Training Data Test Data Model Evaluate
33. Training Data Test Data Model Evaluate Data collection
34. Training Data Test Data Model Evaluate Data collection Feature extraction
35. Training Data Test Data Model Evaluate Data collection Feature extraction Data verification
36. Training Data Test Data Model Evaluate Data collection Configuration Feature extraction Data verification
37. Training Data Test Data Model Evaluate Data collection Configuration Feature extraction Data verification Machine resource management
38. Training Data Test Data Model Evaluate Data collection Configuration Feature extraction Data verification Serving Infrastructure Machine resource management
39. Training Data Test Data Model Evaluate Data collection Configuration Feature extraction Data verification Serving Infrastructure Monitoring Machine resource management
40. Training Data Test Data Model Evaluate Data collection Configuration Feature extraction Data verification Serving Infrastructure Monitoring Analysis tools Machine resource management
41. ◻ learn development ◻ learn ops & infra ◻ learn data science
42. ✅ learn development ◻ learn ops & infra ◻ learn data science
43. ✅ learn development ✅ learn ops & infra ◻ learn data science
44. ✅ learn development ✅ learn ops & infra ✅ learn data science
45. data scientist — expectation
46. 3. training hundreds of models
47. Recommendation Antispam Churn rate
48. Recommendation Antispam Churn rate Recommendation v1
49. Recommendation Antispam Churn rate Recommendation v1 Recommendation v2
50. Recommendation Antispam Churn rate Recommendation v1 Recommendation v2 Recommendation v3
51. Recommendation Antispam Churn rate Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4
52. Recommendation Antispam Churn rate v1 Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4 Antispam Antispam v2 Antispam v3 Antispam v4
53. Churn rate v1 Churn rate v2 Churn rate v3 Churn rate v4 Recommendation Antispam Churn rate v1 Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4 Antispam Antispam v2 Antispam v3 Antispam v4
54. Churn rate v1 Churn rate v2 Churn rate v3 Churn rate v1 Recommendation Antispam Churn rate v1 Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4 Antispam Antispam v2 Antispam v3 Antispam v4 Monday
55. Churn rate v1 Churn rate v2 Churn rate v3 Churn rate v1 Recommendation Antispam Churn rate v1 Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4 Antispam Antispam v2 Antispam v3 Antispam v4 Churn rate v1 Churn rate v2 Churn rate v3 Churn rate v1 Recommendation Antispam Churn rate v1 Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4 Antispam Antispam v2 Antispam v3 Antispam v4 Monday Tuesday
56. Churn rate v1 Churn rate v2 Churn rate v3 Churn rate v1 Recommendation Antispam Churn rate v1 Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4 Antispam Antispam v2 Antispam v3 Antispam v4 Churn rate v1 Churn rate v2 Churn rate v3 Churn rate v1 Recommendation Antispam Churn rate v1 Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4 Antispam Antispam v2 Antispam v3 Antispam v4 Churn rate v1 Churn rate v2 Churn rate v3 Churn rate v1 Recommendation Antispam Churn rate v1 Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4 Antispam Antispam v2 Antispam v3 Antispam v4 Monday Tuesday Wednesday
57. Churn rate v1 Churn rate v2 Churn rate v3 Churn rate v1 Recommendation Antispam Churn rate v1 Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4 Antispam Antispam v2 Antispam v3 Antispam v4 Churn rate v1 Churn rate v2 Churn rate v3 Churn rate v1 Recommendation Antispam Churn rate v1 Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4 Antispam Antispam v2 Antispam v3 Antispam v4 Churn rate v1 Churn rate v2 Churn rate v3 Churn rate v1 Recommendation Antispam Churn rate v1 Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4 Antispam Antispam v2 Antispam v3 Antispam v4 Churn rate v1 Churn rate v2 Churn rate v3 Churn rate v1 Recommendation Antispam Churn rate v1 Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4 Antispam Antispam v2 Antispam v3 Antispam v4 Monday Tuesday Wednesday Thursday
58. Churn rate v1 Churn rate v2 Churn rate v3 Churn rate v1 Recommendation Antispam Churn rate v1 Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4 Antispam Antispam v2 Antispam v3 Antispam v4 Churn rate v1 Churn rate v2 Churn rate v3 Churn rate v1 Recommendation Antispam Churn rate v1 Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4 Antispam Antispam v2 Antispam v3 Antispam v4 Churn rate v1 Churn rate v2 Churn rate v3 Churn rate v1 Recommendation Antispam Churn rate v1 Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4 Antispam Antispam v2 Antispam v3 Antispam v4 Churn rate v1 Churn rate v2 Churn rate v3 Churn rate v1 Recommendation Antispam Churn rate v1 Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4 Antispam Antispam v2 Antispam v3 Antispam v4 Churn rate v1 Churn rate v2 Churn rate v3 Churn rate v1 Recommendation Antispam Churn rate v1 Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4 Antispam Antispam v2 Antispam v3 Antispam v4 Monday Tuesday Wednesday Thursday Friday
59. Churn rate v1 Churn rate v2 Churn rate v3 Churn rate v1 Recommendation Antispam Churn rate v1 Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4 Antispam Antispam v2 Antispam v3 Antispam v4 Churn rate v1 Churn rate v2 Churn rate v3 Churn rate v1 Recommendation Antispam Churn rate v1 Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4 Antispam Antispam v2 Antispam v3 Antispam v4 Churn rate v1 Churn rate v2 Churn rate v3 Churn rate v1 Recommendation Antispam Churn rate v1 Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4 Antispam Antispam v2 Antispam v3 Antispam v4 Churn rate v1 Churn rate v2 Churn rate v3 Churn rate v1 Recommendation Antispam Churn rate v1 Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4 Antispam Antispam v2 Antispam v3 Antispam v4 Churn rate v1 Churn rate v2 Churn rate v3 Churn rate v1 Recommendation Antispam Churn rate v1 Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4 Antispam Antispam v2 Antispam v3 Antispam v4 Churn rate v1 Churn rate v2 Churn rate v3 Churn rate v1 Recommendation Antispam Churn rate v1 Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4 Antispam Antispam v2 Antispam v3 Antispam v4 Monday Tuesday Wednesday Thursday Friday Saturday
60. Churn rate v1 Churn rate v2 Churn rate v3 Churn rate v1 Recommendation Antispam Churn rate v1 Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4 Antispam Antispam v2 Antispam v3 Antispam v4 Churn rate v1 Churn rate v2 Churn rate v3 Churn rate v1 Recommendation Antispam Churn rate v1 Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4 Antispam Antispam v2 Antispam v3 Antispam v4 Churn rate v1 Churn rate v2 Churn rate v3 Churn rate v1 Recommendation Antispam Churn rate v1 Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4 Antispam Antispam v2 Antispam v3 Antispam v4 Churn rate v1 Churn rate v2 Churn rate v3 Churn rate v1 Recommendation Antispam Churn rate v1 Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4 Antispam Antispam v2 Antispam v3 Antispam v4 Churn rate v1 Churn rate v2 Churn rate v3 Churn rate v1 Recommendation Antispam Churn rate v1 Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4 Antispam Antispam v2 Antispam v3 Antispam v4 Churn rate v1 Churn rate v2 Churn rate v3 Churn rate v1 Recommendation Antispam Churn rate v1 Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4 Antispam Antispam v2 Antispam v3 Antispam v4 Churn rate v1 Churn rate v2 Churn rate v3 Churn rate v1 Recommendation Antispam Churn rate v1 Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4 Antispam Antispam v2 Antispam v3 Antispam v4 Monday Tuesday Wednesday Thursday Sunday Friday Saturday
61. ! manual automation !
62. slow!
63. slow! error prone!!
64. slow! error prone!! boring!!!
65. !
66. what can you do?
67. what if…
68. you could use ☁?
69. rent kit by the hour
70. rent kit by the hour bigger machines
71. rent kit by the hour bigger machines scale with the team
72. what if…
73. ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁ ☁
74. rent kit by the hour
75. rent kit by the hour rent fleet of machine by the hour
76. rent fleet of machine by the hour bigger machines
77. rent fleet of machine by the hour bigger machines mixed nodes w/ cpus , gpus , tpus
78. rent fleet of machine by the hour mixed nodes w/ cpus , gpus , tpus scale with the team
79. rent fleet of machine by the hour mixed nodes w/ cpus , gpus , tpus scale with the team
80. rent fleet of machine by the hour mixed nodes w/ cpus , gpus , tpus scale with the company
81. what if…
82. this is ready to be used?
84. what is it?
98. also
99. LOW PRIORITY
100. LOW PRIORITY
101. LOW PRIORITY
102. GPU LOW PRIORITY
103. GPU LOW PRIORITY
104. LOW GPU LOW PRIORITY
105. GPU LOW PRIORITY LOW
106. still single job…
114. how can i ! jobs?
116. Kubernetes
117. Kubernetes Kubeflow
118. Kubernetes Kubeflow
119. Kubernetes Kubeflow
120. Kubernetes Kubeflow
122. architecture
123. MASTER
124. MASTER PARAMETER SERVER PARAMETER SERVER
125. MASTER PARAMETER SERVER PARAMETER SERVER
126. MASTER WORKER WORKER PARAMETER SERVER PARAMETER SERVER
127. MASTER WORKER WORKER WORKER WORKER PARAMETER SERVER PARAMETER SERVER
128. MASTER WORKER WORKER WORKER WORKER PARAMETER SERVER PARAMETER SERVER
129. MASTER WORKER WORKER WORKER WORKER PARAMETER SERVER PARAMETER SERVER
130. meanwhile ☸
139. does k8s run ! python !?
140. nope!
141. container orchestrator
142. ! container !
145. PY JSON +
146. PY JSON + +
147. PY JSON + + CONTAINER
148. language agnostic
150. Java 1.8 Node.js 10 Python 2.7 Python 3.8 .NET Core Lua
158. !
159. if you were to…
160. 1. write a tf model
161. 1. write a tf model 2. package it up
162. 1. write a tf model 2. package it up 3. deploy to kubeflow
163. machine learning in the ☁
164. challenge accepted
165. ready, steady, go!
166. recognising handwritten digits
170. model
179. why stopping here?
180. 1. serving results
182. 2 BACKEND API
183. 2 BACKEND API
184. ✅ tensorflow serving
185. 2. hyper parameters sweeping
189. ✅ parametrised jobs
190. demo
191. ◻ create k8s cluster ◻ install kubeflow ◻ write mnist model
192. ✅ create k8s cluster ◻ install kubeflow ◻ write mnist model
193. ✅ create k8s cluster ✅ install kubeflow ◻ write mnist model
194. ✅ create k8s cluster ✅ install kubeflow ✅ write mnist model
195. https://learnk8s.io/infiniteconf2018 1. test locally 2. create a container 3. deploy to the cluster 4. hyperparameters sweeping 5. serve you model
197. machine learning at scale with k8s
199. 6000 instances of apache solr ~ 1000 servers
203. training ml with 2500 nodes
204. horizontal pod autoscaler
212. cluster autoscaler
216. 256 P100 128000 preemptible vcpus
217. 256 P100 $400/hr 128000 preemptible vcpus $1280/hr
218. 256 P100 $400/hr 128000 preemptible vcpus $1280/hr
221. 320,000 cores
222. 320,000 cores 4,300 projects
223. 320,000 cores 4,300 projects 250 petabytes
224. 3,300 users
225. 3,300 users 10,000 hypervisors
226. 3,300 users 10,000 hypervisors 210 kubernetes clusters
227. ! automation !
228. Recommendation Recommendation v1 Recommendation v2 Recommendation v3 Recommendation v4
231. DEV STAGING PROD
232. DEV STAGING PROD
233. DEV STAGING PROD
234. DEV STAGING PROD
235. DEV STAGING PROD
236. DEV STAGING PROD
237. DEV STAGING PROD
238. DEV STAGING PROD
239. DEV STAGING PROD
240. DEV STAGING PROD
241. DEV STAGING PROD DEV STAGING PROD
242. DEV STAGING PROD DEV STAGING PROD DEV STAGING PROD
243. DEV STAGING PROD DEV STAGING PROD DEV STAGING PROD DEV STAGING PROD
244. ✅ automation
245. what's next?
246. 1. play with ☸ · https://github.com/learnk8s/distributed- tensorflow-on-k8s · https://learnk8s.io/infiniteconf2018 · until monday morning
247. 2. learn k8s · master applications deployment and scaling · ! kubernetes for developers · CKA/CKAD
248. 3. build a poc · use a cloud provider · research small project · move fast, break things™
249. what if I have a question?
250. learnk8s-slack-invite.herokuapp.com
251. do you do any consulting?
253. stay in touch!
254. stay in touch! · Twitter @learnk8s · Newsletter · Daniele Polencic · Salman Iqbal
255. events
256. events · Workshop taster @ Skills Matter, 23rd of July · progNET — Use Kubernetes to Deploy .NET Applications, 13th of Sept · LJC — Autoscaling applications with k8s and Spring Boot, 25th of Sept
257. Thanks! @learnk8s