Slide 1

Slide 1 text

Rolling Update Kubernetes Deployment with Zero Downtime Takeshi Kondo / @chaspy mkmk #12

Slide 2

Slide 2 text

We are deploying application on Kubernetes Deployment Service 35 replicas of pods Reverse Proxy Service Router

Slide 3

Slide 3 text

Kubernetes provides Rolling Update Performing a Rolling Update https://kubernetes.io/docs/tutorials/kubernetes-basics/update/update-intro/

Slide 4

Slide 4 text

Kubernetes provides Rolling Update Performing a Rolling Update https://kubernetes.io/docs/tutorials/kubernetes-basics/update/update-intro/

Slide 5

Slide 5 text

Kubernetes manifest apiVersion: extensions/v1beta1 kind: Deployment metadata: name: "${SERVICE_NAME}" spec: replicas: 35 strategy: rollingUpdate: maxSurge: 21 maxUnavailable: 7 type: RollingUpdate

Slide 6

Slide 6 text

Means replacing 35 pods while creating to 21 pods and deleting 7 pods

Slide 7

Slide 7 text

With Zero Downtime

Slide 8

Slide 8 text

Really?

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

Let’s check the termination of pods mechanism

Slide 13

Slide 13 text

Termination of Pods https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods

Slide 14

Slide 14 text

Termination of Pods • 3. Pod shows up as “Terminating” when listed in client commands • 4. (simultaneous with 3) When the Kubelet sees that a Pod has been marked as terminating because the time in 2 has been set, it begins the Pod shutdown process. • 1. If one of the Pod’s containers has defined a preStop hook, it is invoked inside of the container. If the prestop hook is still running after the grace period expires, step 2 is then invoked with a small (2 second) extended grace period. • 2. The container is sent the TERM signal. Note that not all containers in the Pod will receive the TERM signal at the same time and may each require a prestop hook if the order in which they shut down matters. • 5. (simultaneous with 3) Pod is removed from endpoints list for service, and are no longer considered part of the set of running Pods for replication controllers. Pods that shutdown slowly cannot continue to serve traffic as load balancers (like the service proxy) remove them from their rotations. https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods

Slide 15

Slide 15 text

Termination of Pods • 3. Pod shows up as “Terminating” • 4. (simultaneous with 3) When the Kubelet sees that a Pod has been marked as terminating • Do PreStop hook • Sent the TERM signal • 5. Pod is removed from endpoints list for service, If preStop is not set, it means that Sent the TERM signal and Pod is removed from endpoints list for service are performed asynchronously

Slide 16

Slide 16 text

Happens asynchronously Deployment Service 35 replicas of pods Reverse Proxy Service Router Pod Sent the TERM signal Pod is removed from endpoints list for service

Slide 17

Slide 17 text

Happens asynchronously Deployment Service 35 replicas of pods Reverse Proxy Service Router Pod Sent the TERM signal Pod is removed from endpoints list for service *G4FOUUIF5&3.TJHOBMJTEPOFFBSMJFS UIBO1PEJTSFNPWFEGSPNFOEQPJOUTMJTU GPSTFSWJDF SFRVFTUGBJMT

Slide 18

Slide 18 text

By the way, if unicorn received SIGTERM…

Slide 19

Slide 19 text

Solution: Sleep by preStop • 3. Pod shows up as “Terminating” • 4. (simultaneous with 3) When the Kubelet sees that a Pod has been marked as terminating • Do PreStop hook • Sent the TERM signal • 5. Pod is removed from endpoints list for service,

Slide 20

Slide 20 text

Set preStop

Slide 21

Slide 21 text

Result of release this week…?

Slide 22

Slide 22 text

Before

Slide 23

Slide 23 text

After

Slide 24

Slide 24 text

To be continued…

Slide 25

Slide 25 text

Conclusion • When a pod is deleted, the following happens asynchronously • Do PreStop and Sent the TERM signal • Pod is removed from endpoints list for service • If you do not wait synchronously with prestop, traffic will be sent after sending SIGTERM

Slide 26

Slide 26 text

Thank You! chaspy chaspy_ / chaspy_en Site Reliability Engineer at Quipper Takeshi Kondo SRE Lounge Terraform-jp