Kubernetes

Hootsuite's journey to a production grade cluster.

We will go through an example setup for a Kubernetes cluster that scales for ~100s of nodes and then extend that to a cluster for ~1000s of nodes.

We will explore the challenges that Hootsuite has had with running and maintaining Kubernetes in AWS.

Transcript

1. Anubhav Mishra Senior Developer - Production Operations and Delivery @anubhavm Hootsuite’s Journey to a Production Grade Cluster Kubernetes
3. Agenda • Fundamentals • Architecture • AWS deployment • Other options for deployment
4. Fundamentals
5. Fundamentals • Pods • Nodes
6. Fundamentals - Pods
7. Fundamentals - Nodes
8. Architecture
10. API
11. API kubelet
12. scheduler API kubelet
13. scheduler API kubelet controller Manager
14. scheduler API kubelet controller Manager kube proxy
15. scheduler API kubelet controller Manager kube proxy kube DNS
16. scheduler API kubelet controller Manager kube proxy kube DNS etcd
17. scheduler API kubelet controller Manager kube proxy kube DNS etcd
18. Computer
19. Computer The journey begins….
20. “Kubernetes The hard Way” Kelsey Hightower Developer Advocate - Google
23. pod1 pod2
29. “Source/Destination Checks” whaaaaat?
33. Advantages • Using layer 3 (AWS Routing Table) • Simpler implementation Limitations • Routing table (50 entries max) = 50 nodes • Complex orchestration*
34. Our Production Criteria • ~1000s of nodes / cluster • Operationally friendly • Highly available • Easily reproducible ◦ Multi region ◦ Multiple clusters / environment • Secure
35. Our Production Criteria • ~1000s of nodes / cluster • Operationally friendly • Highly available • Easily reproducible ◦ Multi region ◦ Multiple clusters / environment • Secure
36. Lets not use the AWS Routing Table? But how? pod1 pod2
37. Overlay Network (Flannel, Calico)
38. Overlay Network (Flannel, Calico)
39. Overlay Network (Flannel, Calico)
40. Our Production Criteria • ~1000s of nodes / cluster • Operationally friendly • Highly available • Easily reproducible ◦ Multi region ◦ Multiple clusters / environment • Secure
41. Our Production Criteria • ~1000s of nodes / cluster • Operationally friendly • Highly available • Easily reproducible ◦ Multi region ◦ Multiple clusters / environment • Secure
42. AWS Deployment
46. Terraform
47. Vault
48. 4 months x
51. But wait isn’t there something called “kops”?
52. kops (Kubernetes Operations) • Easy to get a production grade cluster • Create/Upgrade/Delete a cluster • Reconfigure the cluster components • Manage multiple clusters • Export terraform
53. Kelsey Hightower
54. Yay! We made it through. Yes! We are hiring :D Senior Developer - Production Operations and Delivery @anubhavm Anubhav Mishra