Rolling Update Kubernetes Deployment with Zero Downtime

mkmk#12

Transcript

1. Rolling Update Kubernetes Deployment with Zero Downtime Takeshi Kondo / @chaspy mkmk #12
2. We are deploying application on Kubernetes Deployment Service 35 replicas of pods Reverse Proxy Service Router
3. Kubernetes provides Rolling Update Performing a Rolling Update https://kubernetes.io/docs/tutorials/kubernetes-basics/update/update-intro/
4. Kubernetes provides Rolling Update Performing a Rolling Update https://kubernetes.io/docs/tutorials/kubernetes-basics/update/update-intro/
5. Kubernetes manifest apiVersion: extensions/v1beta1 kind: Deployment metadata: name: "${SERVICE_NAME}" spec: replicas: 35 strategy: rollingUpdate: maxSurge: 21 maxUnavailable: 7 type: RollingUpdate
6. Means replacing 35 pods while creating to 21 pods and deleting 7 pods
7. With Zero Downtime
8. Really?
12. Let’s check the termination of pods mechanism
13. Termination of Pods https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods
14. Termination of Pods • 3. Pod shows up as “Terminating” when listed in client commands • 4. (simultaneous with 3) When the Kubelet sees that a Pod has been marked as terminating because the time in 2 has been set, it begins the Pod shutdown process. • 1. If one of the Pod’s containers has defined a preStop hook, it is invoked inside of the container. If the prestop hook is still running after the grace period expires, step 2 is then invoked with a small (2 second) extended grace period. • 2. The container is sent the TERM signal. Note that not all containers in the Pod will receive the TERM signal at the same time and may each require a prestop hook if the order in which they shut down matters. • 5. (simultaneous with 3) Pod is removed from endpoints list for service, and are no longer considered part of the set of running Pods for replication controllers. Pods that shutdown slowly cannot continue to serve traffic as load balancers (like the service proxy) remove them from their rotations. https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods
15. Termination of Pods • 3. Pod shows up as “Terminating” • 4. (simultaneous with 3) When the Kubelet sees that a Pod has been marked as terminating • Do PreStop hook • Sent the TERM signal • 5. Pod is removed from endpoints list for service, If preStop is not set, it means that Sent the TERM signal and Pod is removed from endpoints list for service are performed asynchronously
16. Happens asynchronously Deployment Service 35 replicas of pods Reverse Proxy Service Router Pod Sent the TERM signal Pod is removed from endpoints list for service
17. Happens asynchronously Deployment Service 35 replicas of pods Reverse Proxy Service Router Pod Sent the TERM signal Pod is removed from endpoints list for service *G
    4FOU
    UIF
    5&3.
    TJHOBM
    JT
    EPOF
    FBSMJFS
    UIBO
    1PE
    JT
    SFNPWFE
    GSPN
    FOEQPJOUT
    MJTU
    GPS
    TFSWJDF
    SFRVFTU
    GBJMT
18. By the way, if unicorn received SIGTERM…
19. Solution: Sleep by preStop • 3. Pod shows up as “Terminating” • 4. (simultaneous with 3) When the Kubelet sees that a Pod has been marked as terminating • Do PreStop hook • Sent the TERM signal • 5. Pod is removed from endpoints list for service,
20. Set preStop
21. Result of release this week…?
22. Before
23. After
24. To be continued…
25. Conclusion • When a pod is deleted, the following happens asynchronously • Do PreStop and Sent the TERM signal • Pod is removed from endpoints list for service • If you do not wait synchronously with prestop, traffic will be sent after sending SIGTERM
26. Thank You! chaspy chaspy_ / chaspy_en Site Reliability Engineer at Quipper Takeshi Kondo SRE Lounge Terraform-jp