Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Magic of Kubernetes Self-Healing Capabilities

The Magic of Kubernetes Self-Healing Capabilities

Kubecon EU 2019 Barcelona Talk.
Abstract: https://sched.co/MPcA
Video: https://youtu.be/91dgNqma7-Q

515740717a9ba5d00fb79158c9071097?s=128

Saad Ali

May 22, 2019
Tweet

Transcript

  1. The Magic of Kubernetes Self-Healing Capabilities Saad Ali Senior Software

    Engineer, Google May 22, 2019 github.com/saad-ali twitter.com/the_saad_ali
  2. • Kubernetes manages clusters with a single node up to

    1000s of node • Failure is inevitable • Humans can’t keep up! Problem
  3. Kubernetes Self Healing This is where Kubernetes really shines!

  4. Agenda • How Kubernetes Self Healing Works • Examples of

    Self Healing in Kubernetes • Areas for Improvement
  5. How Kubernetes Self Healing Works Observe and rectify. Declare intended

    state. Controllers Declarative API
  6. Imperative APIs Node A Node B

  7. Imperative APIs Node A Node B Container A

  8. How Kubernetes Self Healing Works Kubernetes APIs are declarative rather

    then imperative.
  9. Imperative API - Manual • You: provide exact set of

    instructions to drive to desired state • System: executes instructions • You: monitor system, and provide further instructions if it deviates. Declarative API - Automatic • You: define desired state • System: works to drive towards that state Declarative APIs
  10. Declarative APIs - Creating a pod Master: API Server The

    Kubernetes way! • You: create API object that is persisted on kube API server until deletion • System: all components work in parallel to drive to that state Node A Node B kubectl create -f replica.yaml
  11. Declarative APIs - Creating a pod The Kubernetes way! •

    You: create API object that is persisted on kube API server until deletion • System: all components work in parallel to drive to that state Master: API Server Node A Node B kubectl create -f replica.yaml apiVersion: apps/v1 kind: ReplicaSet metadata: name: frontend spec: replicas: 1 template: metadata: ... spec: ... containers: - name: nginx image: internal.mycorp.com:5000/mycontainer:1.7.9
  12. Declarative APIs - Creating a pod The Kubernetes way! •

    You: create API object that is persisted on kube API server until deletion • System: all components work in parallel to drive to that state Master: API Server Node A Node B Pod A definition
  13. Declarative APIs - Creating a pod The Kubernetes way! •

    You: create API object that is persisted on kube API server until deletion • System: all components work in parallel to drive to that state Master: API Server Node A Node B Pod A definition Pod A
  14. Declarative APIs - Creating a pod All components watch the

    Kubernetes API, and figure out what they need to do. Master: API Server Node A Node B Master: Scheduler
  15. Declarative APIs - Creating a pod All components watch the

    Kubernetes API, and figure out what they need to do. Master: API Server Node A Node B Master: Scheduler kubectl create pod
  16. Declarative APIs - Creating a pod All components watch the

    Kubernetes API, and figure out what they need to do. Master: API Server Node A Node B Master: Scheduler Pod A
  17. Declarative APIs - Creating a pod All components watch the

    Kubernetes API, and figure out what they need to do. Master: API Server Node A Node B Master: Scheduler Pod A Node: B
  18. Declarative APIs - Creating a pod All components watch the

    Kubernetes API, and figure out what they need to do. Master: API Server Node A Node B Master: Scheduler Pod A Node: B Pod A
  19. Declarative APIs - Creating a pod All components watch the

    Kubernetes API, and figure out what they need to do. Master: API Server Node A Node B Master: Scheduler Pod A Node: B Pod A kubectl delete pod A
  20. Declarative APIs - Creating a pod All components watch the

    Kubernetes API, and figure out what they need to do. Master: API Server Node A Node B Master: Scheduler Pod A
  21. Declarative APIs - Creating a pod All components watch the

    Kubernetes API, and figure out what they need to do. Master: API Server Node A Node B Master: Scheduler
  22. Level triggered instead of edge triggered -- no “missing events”

    issues. No single point of failure. Simple master components. Automatic recovery! Resulting in a Simpler, more robust system that can easily recover from failure of components. Benefits of Declarative API
  23. • In memory cache ◦ Desired State ◦ Actual State

    • Reconciler loop • Populator -- adds and removes from desired state. Controllers
  24. Example of Automatic Recovery Master: API Server Node A Node

    B Pod A definition Pod A
  25. Example of Automatic Recovery Master: API Server Node A Node

    B Pod A definition Pod A
  26. Example of Automatic Recovery Master: API Server Node A Node

    B Master: Scheduler Master: Node Controller Master: Replica Controller
  27. Example of Automatic Recovery Master: API Server Node A Node

    B Pod A definition Pod A Pod A
  28. Challenges What if actual state cache drifts from real world?

  29. • Should have a way to observe and rectify Actual

    State Cache • Not always easy to implement ◦ Example: Orphaned volume mounts. ◦ Room for improvement. Actual State Drift
  30. Challenges Eventually consistent can take a long time.

  31. • Detection ◦ 5 minutes • Force detach ◦ 6

    minutes • Attach volume ◦ Seconds to minutes • Starting new pod ◦ Seconds to minutes 10+ minutes to detect a shutdown node and move it. Node Shutdown
  32. Node Shutdown Kubernetes gives you automatic recovery NOT high availability!!

  33. Thank you! Questions? github.com/saad-ali twitter.com/the_saad_ali