The Magic of Kubernetes Self-Healing Capabilities Saad Ali Senior Software Engineer, Google May 22, 2019 github.com/saad-ali twitter.com/the_saad_ali

● Kubernetes manages clusters with a single node up to 1000s of node ● Failure is inevitable ● Humans can’t keep up! Problem

Kubernetes Self Healing This is where Kubernetes really shines!

Agenda ● How Kubernetes Self Healing Works ● Examples of Self Healing in Kubernetes ● Areas for Improvement

How Kubernetes Self Healing Works Observe and rectify. Declare intended state. Controllers Declarative API

Imperative APIs Node A Node B

Imperative APIs Node A Node B Container A

How Kubernetes Self Healing Works Kubernetes APIs are declarative rather then imperative.

Imperative API - Manual ● You: provide exact set of instructions to drive to desired state ● System: executes instructions ● You: monitor system, and provide further instructions if it deviates. Declarative API - Automatic ● You: define desired state ● System: works to drive towards that state Declarative APIs

Declarative APIs - Creating a pod Master: API Server The Kubernetes way! ● You: create API object that is persisted on kube API server until deletion ● System: all components work in parallel to drive to that state Node A Node B kubectl create -f replica.yaml

Declarative APIs - Creating a pod The Kubernetes way! ● You: create API object that is persisted on kube API server until deletion ● System: all components work in parallel to drive to that state Master: API Server Node A Node B kubectl create -f replica.yaml apiVersion: apps/v1 kind: ReplicaSet metadata: name: frontend spec: replicas: 1 template: metadata: ... spec: ... containers: - name: nginx image: internal.mycorp.com:5000/mycontainer:1.7.9

Declarative APIs - Creating a pod The Kubernetes way! ● You: create API object that is persisted on kube API server until deletion ● System: all components work in parallel to drive to that state Master: API Server Node A Node B Pod A definition

Declarative APIs - Creating a pod The Kubernetes way! ● You: create API object that is persisted on kube API server until deletion ● System: all components work in parallel to drive to that state Master: API Server Node A Node B Pod A definition Pod A

Declarative APIs - Creating a pod All components watch the Kubernetes API, and figure out what they need to do. Master: API Server Node A Node B Master: Scheduler

Declarative APIs - Creating a pod All components watch the Kubernetes API, and figure out what they need to do. Master: API Server Node A Node B Master: Scheduler kubectl create pod

Declarative APIs - Creating a pod All components watch the Kubernetes API, and figure out what they need to do. Master: API Server Node A Node B Master: Scheduler Pod A

Declarative APIs - Creating a pod All components watch the Kubernetes API, and figure out what they need to do. Master: API Server Node A Node B Master: Scheduler Pod A Node: B

Declarative APIs - Creating a pod All components watch the Kubernetes API, and figure out what they need to do. Master: API Server Node A Node B Master: Scheduler Pod A Node: B Pod A

Declarative APIs - Creating a pod All components watch the Kubernetes API, and figure out what they need to do. Master: API Server Node A Node B Master: Scheduler Pod A Node: B Pod A kubectl delete pod A

Declarative APIs - Creating a pod All components watch the Kubernetes API, and figure out what they need to do. Master: API Server Node A Node B Master: Scheduler Pod A

Declarative APIs - Creating a pod All components watch the Kubernetes API, and figure out what they need to do. Master: API Server Node A Node B Master: Scheduler

Level triggered instead of edge triggered -- no “missing events” issues. No single point of failure. Simple master components. Automatic recovery! Resulting in a Simpler, more robust system that can easily recover from failure of components. Benefits of Declarative API

● In memory cache ○ Desired State ○ Actual State ● Reconciler loop ● Populator -- adds and removes from desired state. Controllers

Example of Automatic Recovery Master: API Server Node A Node B Pod A definition Pod A

Example of Automatic Recovery Master: API Server Node A Node B Pod A definition Pod A

Example of Automatic Recovery Master: API Server Node A Node B Master: Scheduler Master: Node Controller Master: Replica Controller

Example of Automatic Recovery Master: API Server Node A Node B Pod A definition Pod A Pod A

Challenges What if actual state cache drifts from real world?

● Should have a way to observe and rectify Actual State Cache ● Not always easy to implement ○ Example: Orphaned volume mounts. ○ Room for improvement. Actual State Drift

Challenges Eventually consistent can take a long time.

● Detection ○ 5 minutes ● Force detach ○ 6 minutes ● Attach volume ○ Seconds to minutes ● Starting new pod ○ Seconds to minutes 10+ minutes to detect a shutdown node and move it. Node Shutdown

Node Shutdown Kubernetes gives you automatic recovery NOT high availability!!

Thank you! Questions? github.com/saad-ali twitter.com/the_saad_ali