ChaDev: Deploying Self Healing Services with Kubernetes

ChaDev: Deploying Self Healing Services with Kubernetes

Downtime can be both expensive and frustrating. In this talk Rob will explain how Kubernetes automatically redistributes systems, including some real world examples. Diving deeper, he'll cover some of the best practices for deploying self healing services with Kubernetes, including readiness probes, liveness probes, and affinity configuration.

Related to this Presentation:

YouTube (https://www.youtube.com/watch?v=bsrXifq3Pjc)

GitHub (https://github.com/robscott/chadev-2017-05-04)

Further Reading:

The Children's Illustrated Guide to Kubernetes (https://deis.com/blog/2016/kubernetes-illustrated-guide/)

Quickstart for Google Container Engine (https://cloud.google.com/container-engine/docs/quickstart)

Setting up an HA Kubernetes Cluster in AWS with private topology with Kops 1.5.1 (https://www.nivenly.com/kops-1-5-1/)

KubeCon Videos (https://www.youtube.com/playlist?list=PLj6h78yzYM2PAavlbv0iZkod4IVh_iGqV)

8f4b23887287d886cde6dee7b54a83e5?s=128

Rob Scott

May 04, 2017
Tweet

Transcript

  1. Deploying Self Healing Services with Kubernetes Rob Scott ChaDev, May

    4, 2017
  2. @robertjscott robertjscott.ca Rob Scott

  3. @spire spire.me

  4. Remember This? February 28, 2017

  5. All Spire systems were still up

  6. Not so fast though…

  7. An EC2 instance failed

  8. EC2 instances could not be created or replaced during this

    outage
  9. How our old system handled instance failure

  10. How Kubernetes handles instance failure

  11. All Spire systems were still up

  12. So what happened?

  13. The core services powering Spire Website API Scheduler HTTP Services

    Background Services Background Processing Notifications Management Portal
  14. Node 1 Notifications Background Processing API Management Portal What it

    all looks like in Kubernetes API Website STAGING DEMO QA QA Scheduler Website QA STAGING STAGING Node 2 Notifications Background Processing API Management Portal API Website DEMO QA QA Website STAGING Node 3 Notifications Background Processing Management Portal API STAGING DEMO DEMO Scheduler Website DEMO DEMO Notifications Notifications DEMO DEMO Background ProcessingQA Node 4 Notifications API Management Portal Website QA QA Scheduler Background Processing DEMO Background ProcessingQA Management Portal STAGING DEMO STAGING STAGING QA Management Portal STAGING STAGING STAGING DEMO QA
  15. Node 1 Notifications Background Processing API Management Portal What if

    a Node dies? API Website STAGING DEMO QA QA Scheduler Website QA STAGING STAGING Node 2 Notifications Background Processing API Management Portal API Website DEMO QA QA Website STAGING Node 3 Notifications Background Processing Management Portal API STAGING DEMO DEMO Scheduler Website DEMO DEMO Notifications Notifications DEMO DEMO Background ProcessingQA Node 4 Notifications API Management Portal Website QA QA Scheduler Background Processing DEMO Background ProcessingQA Management Portal STAGING DEMO STAGING STAGING QA Management Portal STAGING STAGING STAGING DEMO QA
  16. Node 1 Notifications Background Processing API Management Portal After redistribution

    API Website STAGING DEMO QA QA Scheduler Website QA STAGING STAGING Node 2 Notifications Background Processing API Management Portal API Website DEMO QA QA Website STAGING Node 3 Notifications Background Processing Management Portal API STAGING DEMO DEMO Scheduler Website DEMO DEMO Notifications Notifications DEMO DEMO Background ProcessingQA Notifications API Management Portal Website QA QA Scheduler Background Processing DEMO Background ProcessingQA Management Portal STAGING DEMO STAGING STAGING QA Management Portal STAGING STAGING STAGING DEMO QA
  17. An Analogy

  18. Container Ship == Orchestration Tool

  19. Container Orchestration Tools SWARM

  20. Container Orchestration Trends

  21. Container Orchestration Trends

  22. Shipyard == Provisioning

  23. Provisioning Tools

  24. Provisioning Tools Kops Kargo

  25. Container Yard == Container Registry

  26. Container Registries HUB

  27. Container == Container

  28. Container Implementations

  29. Kops Kubernetes What we use Provisioning Orchestration Container Implementation Container

    Storage
  30. Kubernetes We’re moving toward Provisioning Orchestration Container Implementation Container Storage

  31. Key Kubernetes Components

  32. Pod The smallest Kubernetes component. Contains one or more containers.

  33. Deployment Used to ensure multiple replicas of a pod are

    maintained. Enables rolling updates and self healing functionality.
  34. Service Provides an endpoint to communicate with pods. This can

    be internal with a cluster IP or external with a load balancer.
  35. Namespace A logical way to separate environments in your cluster.

    Examples might include staging and QA namespaces.
  36. Demo

  37. Recap

  38. Liveness Probes When these probes fail, Kubernetes attempts to restart

    the container.
  39. Readiness Probes Kubernetes does not send traffic to the container

    until these probes succeed.
  40. Resource Limits Without enforcing proper resource limits, a single rogue

    container can take down a node.
  41. Bonus

  42. Affinity and Anti-Affinity Proper configuration can ensure your pods are

    deployed across availability zones or regions.
  43. affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: failure-domain.beta.kubernetes.io/zone operator:

    In values: - us-east-1c - us-east-1d preferredDuringSchedulingIgnoredDuringExecution: - weight: 1 preference: matchExpressions: - key: beta.kubernetes.io/instance-type operator: In values: - m4.large
  44. Where to go from here • The Children's Illustrated Guide

    to Kubernetes • Quickstart for Google Container Engine • Setting up an HA Kubernetes Cluster in AWS with private topology with Kops 1.5.1 • KubeCon Videos
  45. Kubernetes is great, you should try it out @robertjscott |

    robertjscott.ca