Rolling Update Kubernetes Deployment with Zero Downtime

Transcript

1. Rolling Update Kubernetes Deployment with Zero Downtime Takeshi Kondo /

   @chaspy mkmk #12

2. We are deploying application on Kubernetes Deployment Service 35 replicas

   of pods Reverse Proxy Service Router

3. Kubernetes provides Rolling Update Performing a Rolling Update https://kubernetes.io/docs/tutorials/kubernetes-basics/update/update-intro/

4. Kubernetes provides Rolling Update Performing a Rolling Update https://kubernetes.io/docs/tutorials/kubernetes-basics/update/update-intro/

5. Kubernetes manifest apiVersion: extensions/v1beta1 kind: Deployment metadata: name: "${SERVICE_NAME}" spec:

   replicas: 35 strategy: rollingUpdate: maxSurge: 21 maxUnavailable: 7 type: RollingUpdate

6. Means replacing 35 pods while creating to 21 pods and

   deleting 7 pods

7. With Zero Downtime

8. Really?

9. None
10. None
11. None

12. Let’s check the termination of pods mechanism

13. Termination of Pods https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods

14. Termination of Pods • 3. Pod shows up as “Terminating”

    when listed in client commands • 4. (simultaneous with 3) When the Kubelet sees that a Pod has been marked as terminating because the time in 2 has been set, it begins the Pod shutdown process. • 1. If one of the Pod’s containers has defined a preStop hook, it is invoked inside of the container. If the prestop hook is still running after the grace period expires, step 2 is then invoked with a small (2 second) extended grace period. • 2. The container is sent the TERM signal. Note that not all containers in the Pod will receive the TERM signal at the same time and may each require a prestop hook if the order in which they shut down matters. • 5. (simultaneous with 3) Pod is removed from endpoints list for service, and are no longer considered part of the set of running Pods for replication controllers. Pods that shutdown slowly cannot continue to serve traffic as load balancers (like the service proxy) remove them from their rotations. https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods

15. Termination of Pods • 3. Pod shows up as “Terminating”

    • 4. (simultaneous with 3) When the Kubelet sees that a Pod has been marked as terminating • Do PreStop hook • Sent the TERM signal • 5. Pod is removed from endpoints list for service, If preStop is not set, it means that Sent the TERM signal and Pod is removed from endpoints list for service are performed asynchronously

16. Happens asynchronously Deployment Service 35 replicas of pods Reverse Proxy

    Service Router Pod Sent the TERM signal Pod is removed from endpoints list for service

17. Happens asynchronously Deployment Service 35 replicas of pods Reverse Proxy

    Service Router Pod Sent the TERM signal Pod is removed from endpoints list for service *G
    4FOU
    UIF
    5&3.
    TJHOBM
    JT
    EPOF
    FBSMJFS
    UIBO
    1PE
    JT
    SFNPWFE
    GSPN
    FOEQPJOUT
    MJTU
    GPS
    TFSWJDF
    SFRVFTU
    GBJMT

18. By the way, if unicorn received SIGTERM…

19. Solution: Sleep by preStop • 3. Pod shows up as

    “Terminating” • 4. (simultaneous with 3) When the Kubelet sees that a Pod has been marked as terminating • Do PreStop hook • Sent the TERM signal • 5. Pod is removed from endpoints list for service,

20. Set preStop

21. Result of release this week…?

22. Before

23. After

24. To be continued…

25. Conclusion • When a pod is deleted, the following happens

    asynchronously • Do PreStop and Sent the TERM signal • Pod is removed from endpoints list for service • If you do not wait synchronously with prestop, traffic will be sent after sending SIGTERM

26. Thank You! chaspy chaspy_ / chaspy_en Site Reliability Engineer at

    Quipper Takeshi Kondo SRE Lounge Terraform-jp