Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Best Practices for Applications on Kubernetes to Achieve Both Frequent Updates and Stability

hhiroshell
December 07, 2022

Best Practices for Applications on Kubernetes to Achieve Both Frequent Updates and Stability

Presentation at the #kubeday japan 2022.

hhiroshell

December 07, 2022
Tweet

More Decks by hhiroshell

Other Decks in Technology

Transcript

  1. #KubeDay Best Practices for Applications on Kubernetes to Achieve Both

    Frequent Updates and Stability Hiroshi Hayakawa | @hhiroshell Cloud Platform Developer & 12th Black Belt Engineer of CloudNative Stack, Yahoo Japan Corporation
  2. #KubeDay Requirements for modern applications

  3. #KubeDay Requirements for modern applications • Applications would be restarted

    for a variety of reasons – Frequent updates – Automatic / Manual scaling – Maintenance of infrastructure • We should consider restarts as something that can happen at any time • Need to be stable even with frequent restarts 3
  4. #KubeDay Further challenges • Want to achieve frequent restarts, stability,

    and… – sufficient service level. – running a large number of pods. – sleep well at night. 4
  5. #KubeDay So, What should we do? • Various events will

    happen in the Kubernetes pods 5 First deployment v1 Pods v2 Pods v3 Pods Rolling Update Restart for some reason Scale out under high loads
  6. #KubeDay So, What should we do? • To simplify the

    issue, let's break it down into three perspectives 6 v1 Pods v2 Pods v3 Pods Ensure that the pod starts safely with sufficient preparation Shut it down truly Graceful Eliminate unexpected crashes
  7. #KubeDay Ensure that pods start safely with sufficient preparation

  8. #KubeDay Ensure that pods start with sufficient preparation • “Pod

    is Ready” means user requests will come into the pod • Containers in the pod must be ready to receive actual requests before the pod’s status become “Ready.” 8
  9. #KubeDay What should it be in our applications? • It

    depends on the application 😅 • Examples: – Prepare connections of relational database – Allocate sufficient initial heap memory – Warm up the app and optimize it with JIT compilation 9
  10. #KubeDay 10 What happens in a pod at a startup

    time InitContainers run sequentially Containers starts and runs ENTRYPOINT command startup probe readiness probe liveness probe … … … It also runs postStart lifecycle hook • On control plane nodes • A new pod is persisted to the API Server • The scheduler decides which Node to place the pod • On worker nodes • The kubelet on the Node starts to make an environment for containers in the pod Pre-start process Containers start up (Our Concern)
  11. #KubeDay 11 What happens in a pod at a startup

    time InitContainers run sequentially Containers starts and runs ENTRYPOINT command startup probe readiness probe liveness probe … … … It also runs postStart lifecycle hook • On control plane nodes • A new pod is persisted to the API Server • The scheduler decides which Node to place the pod • On worker nodes • The kubelet on the Node starts to make an environment for containers in the pod Pre-start process Containers start up (Our Concern) Service In Pod becomes “READY.” Requests come into the pod. Make containers ready here,
  12. #KubeDay How can we prepare before the pod becomes ready?

    • initContainers – Runs as separate containers from main containers – Performs pre-processing before main containers start (e.g., dynamically create configuration files for the main containers) • postStart lifecycle hook – Runs different commands in a main container than the startup command – Pod will not be ready until postStart lifecycle hook is finished – Can be used to prepare main containers before these accept requests (e.g., send warm up requests to the main container with curl) 12
  13. #KubeDay 13 Example - Warm up an application with postStart

    hook apiVersion: v1 kind: Pod metadata: name: java-app spec: containers: - name: java-app image: java-app-image:v1.0.0 ...(snip)... ports: - containerPort: 8080 name: http readinessProbe: failureThreshold: 3 httpGet: path: /health/readiness port: http periodSeconds: 1 lifecycle: postStart: exec: command: - sh - -c - |- sleep 10 for i in $(seq 10000); do curl -s http://localhost:8080/ ∖ > /dev/null done preStop: ...(snip)... ✓Wait for the java-app to start. ✓Send requests to the app in the same pod.
  14. #KubeDay Eliminate unexpected crashes of pods

  15. #KubeDay Eliminate unexpected crashes of pods • Pods unexpectedly stop

    for various reasons • If pods don’t stop safely, incoming requests will end with a failure • When a pod stops due to heavy load, other pods are overloaded and stop one after another 15
  16. #KubeDay 16 Crash scenarios of pods High Traffic High Memory

    Usage High CPU Usage Bottlenecks outside the pod Full GC Can‘t respond to liveness probe The kubelet terminates the pod OOM Kill * If the liveness probe depends on outside the pod
  17. #KubeDay 17 Crash scenarios of pods Bottlenecks outside the pod

    Full GC * If the liveness probe depends on outside the pod High Traffic High Memory Usage High CPU Usage Can‘t respond to liveness probe The kubelet terminates the pod OOM Kill Enough Load testing and Tuning!
  18. #KubeDay A quick review of Kubernetes resource control • We

    can set “requests” and “limits” for each container in a Pod – .spec.containers[].resources.requests.[cpu|memory]: • quantity of resources guaranteed to be available to the container – .spec.containers[].resources.limits.[cpu|memory]: • limits for resource consumption beyond reqeusts 18 resources.limits resources.requests Actual resource consumption of the container
  19. #KubeDay A quick review of Kubernetes resource control • Behavior

    when a container is going to consume more than resource.limits – CPU: Throttling – Memory: OOM Kill • If an OOM Kill occurs, the application cannot handle OS signals and cannot shutdown gracefully 19
  20. #KubeDay JVM memory under Kubernetes resource control • JVM memory

    consumption consists of multiple areas such as heap memory, thread stack, etc. • Tune JVM parameters to keep memory usage within resource limit 20 resource limit Total memory consumption of JVM = Heap + Thread Stack + Code Cache + Metaspace + Direct Memory + Native Memory resource request 🤯 Heap other areas
  21. #KubeDay Tips of JVM heap memory tuning • By default,

    in the JVM, the maximum heap memory is 20-30% of the resource limit. It often means that the resource requests/limits are left unused. Max size of heap memory 21 • You can write, for example, “-XX:MaxRAMPercentage=50.0” in JVM flags to specify heap memory as a percentage of the resource limit resources.limits resources.requests Heap other areas
  22. #KubeDay 22 Crash scenarios of pods High CPU Usage Bottlenecks

    outside the pod OOM Kill * If the liveness probe depends on outside the pod High Traffic High Memory Usage Full GC Can‘t respond to liveness probe The kubelet terminates the pod Select an appropriate GC algorithm
  23. #KubeDay JVM garbage collection of containerized Java applications • In

    the container world, we often use Serial GC due to the small quantity of resources allocated • Serial GC takes a long time for Full GC, During Full GC, the application cannot respond to livenessProbe, and it increases the risk that pods restart unnecessarily 23 Java app container http livenessProbe request fails TERMINATE kubelet Stop the world (Full GC)
  24. #KubeDay JVM garbage collection of containerized Java applications • Consider

    using concurrent GC algorithms (e.g., G1GC) with two or more CPU cores • More heap memory will increase the time of full GC. Even just increasing the memory should not be done without testing 24
  25. #KubeDay 25 Crash scenarios of pods Bottlenecks outside the pod

    OOM Kill * If the liveness probe depends on outside the pod High Traffic High Memory Usage High CPU Usage Full GC Can‘t respond to liveness probe The kubelet terminates the pod Make livenessProbe stable
  26. #KubeDay livenessProbe • But few situations require a reboot 🤔

    26 The kubelet uses liveness probes to know when to restart a container. For example, liveness probes could catch a deadlock, where an application is running, but unable to make progress. “ “ -- Kubernetes documentation
  27. #KubeDay livenessProbe • Setting up a livenessProbe requires careful consideration

    – A livenesspProbe that becomes unresponsive under high load will cause a pod to restart when you want it to be running – Sometimes default (it judges success if the application process is running) is still better 27 Java app container http livenessProbe request fails too many requests (and some of them fails) TERMINATE kubelet users
  28. #KubeDay 28 Crash scenarios of pods High Memory Usage High

    CPU Usage Full GC OOM Kill High Traffic Bottlenecks outside the pod Can‘t respond to liveness probe The kubelet terminates the pod * If the liveness probe depends on outside the pod Make the livenessProbe depends only on inside the pod
  29. #KubeDay livenessProbe • Results of livenessProbe should not depends on

    outside of the pod – Situations outside of the Pod cause the Pod to be terminated – If the cause is not in the Pod, restarting the Pod will not make the situation better 29
  30. #KubeDay Shutdown pods truly graceful

  31. #KubeDay Shutdown pods truly graceful • Two things happen in

    the shutdown process of a pod – kubelet terminates containers – kube-proxy stops traffic to the pod • If containers start to be terminated before traffic stops, requests will result in errors 31
  32. #KubeDay 32 What happens in a pod at a shutdown

    preStop lifecycle hook starts kube-proxy stops traffic to the pod deletionGracePeriodSeconds (default: 30s) kubelet sends SIGKILL to the container Some kube-apiserver client adds a deletionTimestamp annotation to the pod kubelet sends SIGTERM to the container Service Out There is no incoming traffic to the pod from here.
  33. #KubeDay 33 What happens in a pod at a shutdown

    preStop lifecycle hook starts kube-proxy stops traffic to the pod deletionGracePeriodSeconds (default: 30s) kubelet sends SIGKILL to the container Some kube-apiserver client adds a deletionTimestamp annotation to the pod kubelet sends SIGTERM to the container Service Out There is no incoming traffic to the pod from here. Handle SIGTERM and shutdown gracefully Wait for service out
  34. #KubeDay How can we make a pod achieve a truly

    graceful shutdown? • Wait for service out – kubelet sends a SIGTERM OS signal to the container after preStop lifecycle hook exits – Wait until kube-proxy stops traffic to the pod by running the ”sleep xx” command in the preStop hook • Handle SIGTERM signal in the application and shutdown gracefully – The application returns all responses to in-flight requests before it exits – In many languages, SDKs or frameworks supports it 34
  35. #KubeDay 35 Example - Wait for kube-proxy to stop requests

    apiVersion: v1 kind: Pod metadata: name: java-app spec: containers: - name: java-app image: java-app-image:v1.0.0 ...(snip)... ports: - containerPort: 8080 name: http readinessProbe: failureThreshold: 3 httpGet: path: /health/readiness port: http periodSeconds: 1 lifecycle: postStart: ...(snip)... preStop: exec: command: - sh - -c - sleep 10 ✓Wait for kube-proxy to stop requests
  36. #KubeDay Example - SIGTERM handling with Spring Boot • You

    can enable SIGTERM Handling by simply adding some properties to the application.properties 36 server.shutdown=graceful spring.lifecycle.timeout-per-shutdown-phase=15s
  37. #KubeDay Conclusion • Three perspectives for application stability 37 v1

    Pods v2 Pods v3 Pods Ensure that the pod starts safely with sufficient preparation Shut it down truly Graceful Eliminate unexpected crashes
  38. #KubeDay Thank you • I hope you sleep well 😴

    38