Rolling Update Kubernetes Deployment with Zero Downtime Takeshi Kondo / @chaspy mkmk #12

We are deploying application on Kubernetes Deployment Service 35 replicas of pods Reverse Proxy Service Router

Kubernetes provides Rolling Update Performing a Rolling Update https://kubernetes.io/docs/tutorials/kubernetes-basics/update/update-intro/

Kubernetes provides Rolling Update Performing a Rolling Update https://kubernetes.io/docs/tutorials/kubernetes-basics/update/update-intro/

Kubernetes manifest apiVersion: extensions/v1beta1 kind: Deployment metadata: name: "${SERVICE_NAME}" spec: replicas: 35 strategy: rollingUpdate: maxSurge: 21 maxUnavailable: 7 type: RollingUpdate

Means replacing 35 pods while creating to 21 pods and deleting 7 pods

With Zero Downtime

Really?

No content

No content

No content

Let’s check the termination of pods mechanism

Termination of Pods https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods

Termination of Pods • 3. Pod shows up as “Terminating” when listed in client commands • 4. (simultaneous with 3) When the Kubelet sees that a Pod has been marked as terminating because the time in 2 has been set, it begins the Pod shutdown process. • 1. If one of the Pod’s containers has defined a preStop hook, it is invoked inside of the container. If the prestop hook is still running after the grace period expires, step 2 is then invoked with a small (2 second) extended grace period. • 2. The container is sent the TERM signal. Note that not all containers in the Pod will receive the TERM signal at the same time and may each require a prestop hook if the order in which they shut down matters. • 5. (simultaneous with 3) Pod is removed from endpoints list for service, and are no longer considered part of the set of running Pods for replication controllers. Pods that shutdown slowly cannot continue to serve traffic as load balancers (like the service proxy) remove them from their rotations. https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods

Termination of Pods • 3. Pod shows up as “Terminating” • 4. (simultaneous with 3) When the Kubelet sees that a Pod has been marked as terminating • Do PreStop hook • Sent the TERM signal • 5. Pod is removed from endpoints list for service, If preStop is not set, it means that Sent the TERM signal and Pod is removed from endpoints list for service are performed asynchronously

Happens asynchronously Deployment Service 35 replicas of pods Reverse Proxy Service Router Pod Sent the TERM signal Pod is removed from endpoints list for service

Happens asynchronously Deployment Service 35 replicas of pods Reverse Proxy Service Router Pod Sent the TERM signal Pod is removed from endpoints list for service *G4FOUUIF5&3.TJHOBMJTEPOFFBSMJFS UIBO1PEJTSFNPWFEGSPNFOEQPJOUTMJTU GPSTFSWJDF SFRVFTUGBJMT

By the way, if unicorn received SIGTERM…

Solution: Sleep by preStop • 3. Pod shows up as “Terminating” • 4. (simultaneous with 3) When the Kubelet sees that a Pod has been marked as terminating • Do PreStop hook • Sent the TERM signal • 5. Pod is removed from endpoints list for service,

Set preStop

Result of release this week…?

Before

After

To be continued…

Conclusion • When a pod is deleted, the following happens asynchronously • Do PreStop and Sent the TERM signal • Pod is removed from endpoints list for service • If you do not wait synchronously with prestop, traffic will be sent after sending SIGTERM

Thank You! chaspy chaspy_ / chaspy_en Site Reliability Engineer at Quipper Takeshi Kondo SRE Lounge Terraform-jp