ChaDev: Deploying Self Healing Services with Kubernetes

Downtime can be both expensive and frustrating. In this talk Rob will explain how Kubernetes automatically redistributes systems, including some real world examples. Diving deeper, he'll cover some of the best practices for deploying self healing services with Kubernetes, including readiness probes, liveness probes, and affinity configuration.

Related to this Presentation:

YouTube (https://www.youtube.com/watch?v=bsrXifq3Pjc)

GitHub (https://github.com/robscott/chadev-2017-05-04)

Further Reading:

The Children's Illustrated Guide to Kubernetes (https://deis.com/blog/2016/kubernetes-illustrated-guide/)

Quickstart for Google Container Engine (https://cloud.google.com/container-engine/docs/quickstart)

Setting up an HA Kubernetes Cluster in AWS with private topology with Kops 1.5.1 (https://www.nivenly.com/kops-1-5-1/)

KubeCon Videos (https://www.youtube.com/playlist?list=PLj6h78yzYM2PAavlbv0iZkod4IVh_iGqV)

Transcript

1. Deploying Self Healing Services with Kubernetes Rob Scott ChaDev, May 4, 2017
2. @robertjscott robertjscott.ca Rob Scott
3. @spire spire.me
4. Remember This? February 28, 2017
5. All Spire systems were still up
6. Not so fast though…
7. An EC2 instance failed
8. EC2 instances could not be created or replaced during this outage
9. How our old system handled instance failure
10. How Kubernetes handles instance failure
11. All Spire systems were still up
12. So what happened?
13. The core services powering Spire Website API Scheduler HTTP Services Background Services Background Processing Notifications Management Portal
14. Node 1 Notifications Background Processing API Management Portal What it all looks like in Kubernetes API Website STAGING DEMO QA QA Scheduler Website QA STAGING STAGING Node 2 Notifications Background Processing API Management Portal API Website DEMO QA QA Website STAGING Node 3 Notifications Background Processing Management Portal API STAGING DEMO DEMO Scheduler Website DEMO DEMO Notifications Notifications DEMO DEMO Background ProcessingQA Node 4 Notifications API Management Portal Website QA QA Scheduler Background Processing DEMO Background ProcessingQA Management Portal STAGING DEMO STAGING STAGING QA Management Portal STAGING STAGING STAGING DEMO QA
15. Node 1 Notifications Background Processing API Management Portal What if a Node dies? API Website STAGING DEMO QA QA Scheduler Website QA STAGING STAGING Node 2 Notifications Background Processing API Management Portal API Website DEMO QA QA Website STAGING Node 3 Notifications Background Processing Management Portal API STAGING DEMO DEMO Scheduler Website DEMO DEMO Notifications Notifications DEMO DEMO Background ProcessingQA Node 4 Notifications API Management Portal Website QA QA Scheduler Background Processing DEMO Background ProcessingQA Management Portal STAGING DEMO STAGING STAGING QA Management Portal STAGING STAGING STAGING DEMO QA
16. Node 1 Notifications Background Processing API Management Portal After redistribution API Website STAGING DEMO QA QA Scheduler Website QA STAGING STAGING Node 2 Notifications Background Processing API Management Portal API Website DEMO QA QA Website STAGING Node 3 Notifications Background Processing Management Portal API STAGING DEMO DEMO Scheduler Website DEMO DEMO Notifications Notifications DEMO DEMO Background ProcessingQA Notifications API Management Portal Website QA QA Scheduler Background Processing DEMO Background ProcessingQA Management Portal STAGING DEMO STAGING STAGING QA Management Portal STAGING STAGING STAGING DEMO QA
17. An Analogy
18. Container Ship == Orchestration Tool
19. Container Orchestration Tools SWARM
20. Container Orchestration Trends
21. Container Orchestration Trends
22. Shipyard == Provisioning
23. Provisioning Tools
24. Provisioning Tools Kops Kargo
25. Container Yard == Container Registry
26. Container Registries HUB
27. Container == Container
28. Container Implementations
29. Kops Kubernetes What we use Provisioning Orchestration Container Implementation Container Storage
30. Kubernetes We’re moving toward Provisioning Orchestration Container Implementation Container Storage
31. Key Kubernetes Components
32. Pod The smallest Kubernetes component. Contains one or more containers.
33. Deployment Used to ensure multiple replicas of a pod are maintained. Enables rolling updates and self healing functionality.
34. Service Provides an endpoint to communicate with pods. This can be internal with a cluster IP or external with a load balancer.
35. Namespace A logical way to separate environments in your cluster. Examples might include staging and QA namespaces.
36. Demo
37. Recap
38. Liveness Probes When these probes fail, Kubernetes attempts to restart the container.
39. Readiness Probes Kubernetes does not send traffic to the container until these probes succeed.
40. Resource Limits Without enforcing proper resource limits, a single rogue container can take down a node.
41. Bonus
42. Affinity and Anti-Affinity Proper configuration can ensure your pods are deployed across availability zones or regions.
43. affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: failure-domain.beta.kubernetes.io/zone operator: In values: - us-east-1c - us-east-1d preferredDuringSchedulingIgnoredDuringExecution: - weight: 1 preference: matchExpressions: - key: beta.kubernetes.io/instance-type operator: In values: - m4.large
44. Where to go from here • The Children's Illustrated Guide to Kubernetes • Quickstart for Google Container Engine • Setting up an HA Kubernetes Cluster in AWS with private topology with Kops 1.5.1 • KubeCon Videos
45. Kubernetes is great, you should try it out @robertjscott | robertjscott.ca