Lessons learned while crawling the web with Kubernetes

Lessons learned while crawling the web with Kubernetes

We run a service with 30 microservices all processing data in large amounts, at the same time ... Scaling up and down various parts of the system as needed.

Join me on the journey of scaling a distributed computing system. Follow our story of 4 rewrites (ok not complete but large changes to how we handle processing) to support the scale and manage the system as we grow. You'll see and hear all the bad decisions, what to avoid and where we ended up on the present day.

Starting at badly managed VM's on multiple cloud providers, and ending with a autoscaling system running hundreds of small manageable containers over various machines with Google Compute and Kubernetes.

After this talk you'll hopefully know of various pitfalls and things to think about when looking at running your own distributed service using open source tech.


Johann du Toit

March 08, 2016