Containers starts and runs ENTRYPOINT command startup probe readiness probe liveness probe … … … It also runs postStart lifecycle hook • On control plane nodes • A new pod is persisted to the API Server • The scheduler decides which Node to place the pod • On worker nodes • The kubelet on the Node starts to make an environment for containers in the pod Pre-start process Containers start up (Our Concern)
Containers starts and runs ENTRYPOINT command startup probe readiness probe liveness probe … … … It also runs postStart lifecycle hook • On control plane nodes • A new pod is persisted to the API Server • The scheduler decides which Node to place the pod • On worker nodes • The kubelet on the Node starts to make an environment for containers in the pod Pre-start process Containers start up (Our Concern) サービスイン Podが「Ready」になり、 トラフィックが流入する ここで十分な準備を行っておく
apiVersion: v1 kind: Pod metadata: name: java-app spec: containers: - name: java-app image: java-app-image:v1.0.0 ...(snip)... ports: - containerPort: 8080 name: http readinessProbe: failureThreshold: 3 httpGet: path: /health/readiness port: http periodSeconds: 1 lifecycle: postStart: exec: command: - sh - -c - |- sleep 10 for i in $(seq 10000); do curl -s http://localhost:8080/ ∖ > /dev/null done preStop: ...(snip)... ✓Wait for the java-app to start. ✓Send requests to the app in the same pod.
Usage High CPU Usage Bottlenecks outside the pod Full GC Can‘t respond to liveness probe The kubelet terminates the pod OOM Kill * livenessProbeの成否がPod外に依存している場合
the pod Full GC High Traffic High Memory Usage High CPU Usage Can‘t respond to liveness probe The kubelet terminates the pod OOM Kill 負荷試験とチューニングを十分に 行いましょう
Usage Bottlenecks outside the pod OOM Kill High Traffic High Memory Usage Full GC Can‘t respond to liveness probe The kubelet terminates the pod GCの特性を知って適切な 設定を行いましょう
the pod OOM Kill High Traffic High Memory Usage High CPU Usage Full GC Can‘t respond to liveness probe The kubelet terminates the pod 不安定にならないように livenessProbeを設定しよう
「コンテナは動作しているが再起動が必要な状況」とは… 29 The kubelet uses liveness probes to know when to restart a container. For example, liveness probes could catch a deadlock, where an application is running, but unable to make progress. “ “ -- Kubernetes documentation
高負荷時によってlivenessProbeに応答できなくなると、トラフィック が多い状況にも関わらずコンテナが再起動されてしまう • デフォルトならプロセスが生きていれば成功となるので、この方が 良い場合も 30 Java app container http livenessProbe request fails too many requests (and some of them fails) TERMINATE kubelet users
CPU Usage Full GC OOM Kill High Traffic Bottlenecks outside the pod Can‘t respond to liveness probe The kubelet terminates the pod * livenessProbeの成否がPod外に依存している場合 livenessProbeの成否がPod内にだけ 依存するようにしましょう
kube-proxy stops traffic to the pod deletionGracePeriodSeconds (default: 30s) kubelet sends SIGKILL to the container Some kube-apiserver client adds a deletionTimestamp annotation to the pod kubelet sends SIGTERM to the container サービスアウト この時点からPodにリクエストが届かなくなる
kube-proxy stops traffic to the pod deletionGracePeriodSeconds (default: 30s) kubelet sends SIGKILL to the container Some kube-apiserver client adds a deletionTimestamp annotation to the pod kubelet sends SIGTERM to the container アプリでSIGTERMをハンドリング してGracefulに終了する ここでトラフィックの停止を待つ サービスアウト この時点からPodにリクエストが届かなくなる
Spring BootでのGracefulシャットダウン • Spring Bootの設定ファイルにプロパティを追加することで、Graceful シャットダウンを有効化できる 41 server.shutdown=graceful spring.lifecycle.timeout-per-shutdown-phase=15s
hhiroshell
tarotaro0129
nenonaninu
duelist2020jp
ytaka23
ritou
os1ma
faithandbrave
goataka
shingo_kawahara
hakubishin3
bengo4com
twada
holman
honnibal
productmarketing
nonsquared
sergeychernyshev
andyhume
hatefulcrawdad
lara
jnunemaker
3n
scottboms
paigeccino
