Lessons learned while crawling the web with Kubernetes

We run a service with 30 microservices all processing data in large amounts, at the same time ... Scaling up and down various parts of the system as needed.

Join me on the journey of scaling a distributed computing system. Follow our story of 4 rewrites (ok not complete but large changes to how we handle processing) to support the scale and manage the system as we grow. You'll see and hear all the bad decisions, what to avoid and where we ended up on the present day.

Starting at badly managed VM's on multiple cloud providers, and ending with a autoscaling system running hundreds of small manageable containers over various machines with Google Compute and Kubernetes.

After this talk you'll hopefully know of various pitfalls and things to think about when looking at running your own distributed service using open source tech.

Transcript

1. Lessons learned while crawling the internet with Kubernetes #devconf
2. @johann_du_toit Johann du Toit johanndutoit.net #devconf GDE for Cloud
4. Overview 1. Backstory
 2. Kubernetes
 3. Lessons learned trying to scale
5. Overview 1. Backstory
 2. Kubernetes
 3. Lessons learned trying to scale
6. Internet = Broken
10. Accelerated Mobile Pages
12. {
13. 74.6
17. Keeps changing
18. Keep checking
21. Overview 1. Backstory
 2. Kubernetes
 3. Lessons learned trying to scale
23. Self-contained image of your app
26. Docker build —tag=<tag> .
27. tag == <name>:<version>
28. == Image * All of the Code
 * Dependencies
 * Can be run on any machine that can run Docker. No more “but it works on my machine”
31. Docker run <tag>
32. Production ?
33. 
 Docker 
 Docker
40. kubectl
41. kubectl run
43. kubectl get <rc/pods>
45. kubectl logs 
 <pod-id>
47. kubectl scale
49. kubectl rolling-update
50. V1 app V1 V1 V1
51. kubectl rolling-update
 app —image=app:v2
52. V1 app V1 V1 V1
53. V1 app V1 V1 V2 app-next
54. V1 app V1 V2 app-next V2
55. V1 app V2 app-next V2 V2
56. V1 app V1 V1 V1 V2 V2 V2 V2
57. == zero downtime
58. kubectl autoscale
59. --max=5 --min=2 --cpu-percent=80
60. Overview 1. Backstory
 2. Kubernetes
 3. Lessons learned trying to scale
63. npm install <package>
64. < 1,000 lines
65. Microservices
66. Small Autonomous services that work together, modelled around a business domain
67. example.com
68. example.com Search Recommendations Users
69. Advantages * Any Stack per Service
 * Clean approach to each Service
 * Small focusable pieces of logic
70. 30+ workers
71. {
72. Network performance checks
74. cache of website contents Database
75. Too many connections are thing
78. “More instances will only hide a bad architecture for so long”
80. Lesson #1:
 Define precisely what is needed to complete a task
81. Core Util Test
83. cache of website contents
84. THE Queue
86. THE Queue User Requests Backend Crawls
88. Lesson #2: Prioritise user requests that demand immediate responses
89. Lesson #1:
 Define precisely what is needed to complete a task
90. 2 Queues User Requests Backend Crawls
91. VM creation is a heavy task
93. Lesson #3: Layers of Scale
94. kubectl autoscale
99. Lessons: 1. Small manageable services that do one thing good 2. Prioritise user work
 3. Layers of scale
100. Resources: gcloud - cloud.google.com/sdk
 Kubernetes - kubernetes.io
 Docker - docker.com
 GDE - developers.google.com/experts
101. @johann_du_toit Thanks johanndutoit.net #devconf www.devconf.co.za/Rate