Rolling Update Kubernetes Deployment with Zero Downtime

Rolling Update Kubernetes Deployment with Zero Downtime

mkmk#12

93c80c388fe9d8f9df7d030549a0ff0b?s=128

Takeshi Kondo

August 22, 2019
Tweet

Transcript

  1. Rolling Update Kubernetes Deployment with Zero Downtime Takeshi Kondo /

    @chaspy mkmk #12
  2. We are deploying application on Kubernetes Deployment Service 35 replicas

    of pods Reverse Proxy Service Router
  3. Kubernetes provides Rolling Update Performing a Rolling Update https://kubernetes.io/docs/tutorials/kubernetes-basics/update/update-intro/

  4. Kubernetes provides Rolling Update Performing a Rolling Update https://kubernetes.io/docs/tutorials/kubernetes-basics/update/update-intro/

  5. Kubernetes manifest apiVersion: extensions/v1beta1 kind: Deployment metadata: name: "${SERVICE_NAME}" spec:

    replicas: 35 strategy: rollingUpdate: maxSurge: 21 maxUnavailable: 7 type: RollingUpdate
  6. Means replacing 35 pods while creating to 21 pods and

    deleting 7 pods
  7. With Zero Downtime

  8. Really?

  9. None
  10. None
  11. None
  12. Let’s check the termination of pods mechanism

  13. Termination of Pods https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods

  14. Termination of Pods • 3. Pod shows up as “Terminating”

    when listed in client commands • 4. (simultaneous with 3) When the Kubelet sees that a Pod has been marked as terminating because the time in 2 has been set, it begins the Pod shutdown process. • 1. If one of the Pod’s containers has defined a preStop hook, it is invoked inside of the container. If the prestop hook is still running after the grace period expires, step 2 is then invoked with a small (2 second) extended grace period. • 2. The container is sent the TERM signal. Note that not all containers in the Pod will receive the TERM signal at the same time and may each require a prestop hook if the order in which they shut down matters. • 5. (simultaneous with 3) Pod is removed from endpoints list for service, and are no longer considered part of the set of running Pods for replication controllers. Pods that shutdown slowly cannot continue to serve traffic as load balancers (like the service proxy) remove them from their rotations. https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods
  15. Termination of Pods • 3. Pod shows up as “Terminating”

    • 4. (simultaneous with 3) When the Kubelet sees that a Pod has been marked as terminating • Do PreStop hook • Sent the TERM signal • 5. Pod is removed from endpoints list for service, If preStop is not set, it means that Sent the TERM signal and Pod is removed from endpoints list for service are performed asynchronously
  16. Happens asynchronously Deployment Service 35 replicas of pods Reverse Proxy

    Service Router Pod Sent the TERM signal Pod is removed from endpoints list for service
  17. Happens asynchronously Deployment Service 35 replicas of pods Reverse Proxy

    Service Router Pod Sent the TERM signal Pod is removed from endpoints list for service *G4FOUUIF5&3.TJHOBMJTEPOFFBSMJFS UIBO1PEJTSFNPWFEGSPNFOEQPJOUTMJTU GPSTFSWJDF SFRVFTUGBJMT
  18. By the way, if unicorn received SIGTERM…

  19. Solution: Sleep by preStop • 3. Pod shows up as

    “Terminating” • 4. (simultaneous with 3) When the Kubelet sees that a Pod has been marked as terminating • Do PreStop hook • Sent the TERM signal • 5. Pod is removed from endpoints list for service,
  20. Set preStop

  21. Result of release this week…?

  22. Before

  23. After

  24. To be continued…

  25. Conclusion • When a pod is deleted, the following happens

    asynchronously • Do PreStop and Sent the TERM signal • Pod is removed from endpoints list for service • If you do not wait synchronously with prestop, traffic will be sent after sending SIGTERM
  26. Thank You! chaspy chaspy_ / chaspy_en Site Reliability Engineer at

    Quipper Takeshi Kondo SRE Lounge Terraform-jp