k8sの可用性とScalabilityを担保するための大事な観点 / Best practices for ensuring availability and scalability for k8s

k8sの可用性とScalabilityを担保するための大事な観点 / Best practices for ensuring availability and scalability for k8s

5c29b6a44d2463092e8bd5e93eedad8a?s=128

Hiroki Sakamoto

September 29, 2020
Tweet

Transcript

  1. k8sͷAvailabilityͱScalabilityΛ୲อ͢ΔͨΊͷେࣄͳ؍఺ @taisho6339

  2. ࣗݾ঺հ ࡔຊେক (Hiroki Sakamoto) Twitter: taisho6339 Github: taisho6339 ΩϟϦΞ Ϡϑʔ

    → ϦΫϧʔτςΫϊϩδʔζ → ϑϦʔϥϯε ݱࡏͷ࢓ࣄ k8sʹΑΔϚΠΫϩαʔϏεͷͨΊͷج൫ͮ͘Γͱӡ༻ ࠓޙͷํ਑ ΑΓࡋྔΛ΋ͬͯಇͨ͘Ίɺਖ਼ࣾһݕ౼தɻ
  3. ຊ೔ͷςʔϚ k8sΛ҆શʹӡ༻͢Δʹ͋ͨͬͯ୲อͨ͠؍఺Λ2఺ʹߜͬͯ੔ཧʂ Scalability Availability

  4. ࢿྉͷ໨త ະདྷͷPJͰৼΓฦΔͨΊͷόΠϒϧΛ໨ࢦ͢

  5. 5ͭͷେࣄͳ؍఺ 1. ϨΠςϯγΛ୲อ͢Δ 2. εϧʔϓοτΛ୲อ͢Δ 3. εύΠΫʹඋ͑Δ 4. ϊʔυͷμ΢ϯʹඋ͑Δ 5.

    ϚϧνΫϥελʹΑΔϝϯςφϯεઓུ
  6. 1. LatencyΛ୲อ͠Α͏ Throughput Latency

  7. ϨΠςϯγͷ୲อ Pod୯ମͰ૝ఆϨΠςϯγ಺ͰϨεϙϯεΛฦͤΔ͔ΛνΣοΫʂ ͜͜ͷ୲อ͕͓Ζ͔ͦͩͱਫฏεέʔϧͤͯ͞΋ޮՌ͕ബ͍ Pod of Service A locust cluster How

    fast?
  8. ϨΠςϯγͷ୲อ ໰୊ ϨΠςϯγ͕૝ఆΑΓߴ͍ ରࡦ 1. PodΛεέʔϧΞοϓ 2. ࠷దͳNodeʹ഑ஔ͢Δ 3. ΞϓϦέʔγϣϯΛνϡʔχϯά

  9. ϨΠςϯγͷ୲อ ~PodͷεέʔϧΞοϓ~ containers: ... resources: limits: cpu: 1.0 memory: 512Mi

    requests: cpu: 0.2 memory: 512Mi ղܾࡦ1. PodͷεέʔϧΞοϓ • CPUɺϝϞϦͳͲͷϦιʔεΛࢦఆՄೳ • PodͷఆٛʹrequestͱlimitͰઃఆ
  10. ϨΠςϯγͷ୲อ ~PodͷεέʔϧΞοϓ~ requestʹΑΔࢦఆ • PodʹׂΓ౰ͯΔϝϞϦͱCPUΛࢦఆ • Podʹrequest͞ΕͨϦιʔεྔΛݩʹ഑ஔ͞ΕΔϊʔυΛܾఆ

  11. ϨΠςϯγͷ୲อ ~PodͷεέʔϧΞοϓ~ limitʹΑΔࢦఆ • Pod͕࣮ࡍʹ࢖༻Ͱ͖ΔݶքͷϦιʔεྔ ◦ requestͷׂ౰ϦιʔεΛ௒͑Δ͜ͱ͕Ͱ͖Δ ◦ ࢦఆ͠ͳ͍ͱrequestͱಉ͡ʹͳΔ ◦

    ීஈ͸গͳ͍͍͕ͯ͘ɺҰ࣌తʹόʔετ͢ΔՄೳੑͷ͋ΔϫʔΫϩʔυʹ༗ޮ • limitΛӽ͑Α͏ͱ͢ΔͱεϩοτϦϯά͞Εɺ࢖༻཰Λ཈͑ΒΕΔ
  12. ϨΠςϯγͷ୲อ ~PodͷεέʔϧΞοϓ~ requestͱlimitͷ஫ҙ఺ • limitͱrequest ͷ͕ࠩେ͖͍৔߹ ◦ limit·Ͱ࢖༻཰্͕͕ͬͨͱ͖ʹϊʔυϦιʔε͕ރׇ͢ΔՄೳੑ • requestࢦఆ͕ͳ͍৔߹

    ◦ Scheduler͕Ϧιʔε࢖༻ྔΛ൑ఆͰ͖ͳ͍ͷͰಛఆϊʔυʹूத͢ΔՄೳੑ
  13. ϨΠςϯγͷ୲อ ~࠷దͳϊʔυ΁ͷ഑ஔ~ ղܾࡦ2. ࠷దͳϊʔυ΁ͷ഑ஔ • GPUɺSSDͳͲͷϦιʔελΠϓΛબ୒ͯ͠഑ஔ HDD SSD Node1 Node2

    ߴ଎ʹIOॲཧΛͯ͠ ΄͍͠ͷͰ SSDͷϊʔυ΁
  14. ϨΠςϯγͷ୲อ ~࠷దͳϊʔυ΁ͷ഑ஔ~ ϊʔυΛࢦఆ͢Δํ๏ • nodeSelector ◦ ಛఆͷϥϕϧΛ࣋ͭNodeʹ഑ஔ • NodeAffinity ◦

    ಛఆͷϥϕϧΛ࣋ͭNodeʹ഑ஔɻͪ͜Βͷ΄͏͕ΑΓॊೈ • Taint + Toleration ◦ Nodeʹ഑ஔ੍ݶΛՃ͑ɺ഑ஔද໌Λ͍ͯ͠ΔPodͷΈ഑ஔ ࢀߟϦϯΫ: Node্΁ͷPodͷεέδϡʔϦϯά
  15. ϨΠςϯγͷ୲อ ~ΞϓϦέʔγϣϯΛνϡʔχϯά~ ղܾࡦ3. ΞϓϦέʔγϣϯΛνϡʔχϯά • APMͳͲΛ׆༻ͯ͠ϘτϧωοΫΛಛఆ͠ɺ࣮૷Λมߋͯ͠ ύϑΥʔϚϯεվળΛߦ͏

  16. 2. ThroughputΛ୲อ͠Α͏ Throughput Latency

  17. ૝ఆΞʔΩςΫνϟ Ingress Gateway Service A Service B Service C LB

    ϚΠΫϩαʔϏε + API Gatewayͳύλʔϯ
  18. εϧʔϓοτͷ୲อ جຊํ਑ • LBɺAPI GatewayɺServiceͷॱʹ୲อ͍ͯ͘͠ • ϘτϧωοΫΛݟ͑΍͘͢͢ΔͨΊʹHPA͸ແޮʹ͠ɺखಈͰεέʔϧ

  19. εϧʔϓοτͷ୲อ 6000RPSΛ ୲อ͍ͨ͠ʂ جຊํ਑ Ingress Gateway Service A Service B

    Service C LB ϚΠΫϩαʔϏε + API Gatewayͳύλʔϯ
  20. εϧʔϓοτͷ୲อ Ingress Gateway Service A Service B Service C LB

    ϚΠΫϩαʔϏε + API Gatewayͳύλʔϯ LB͸6000RPS ग़Δʁ Nginx (੩తίϯςϯπΛฦ٫) جຊํ਑
  21. εϧʔϓοτͷ୲อ Ingress Gateway Service A Service B Service C LB

    ϚΠΫϩαʔϏε + API Gatewayͳύλʔϯ Ingress Gateway ͸6000RPSग़Δʁ جຊํ਑
  22. εϧʔϓοτͷ୲อ جຊํ਑ Ingress Gateway Service A Service B Service C

    LB ϚΠΫϩαʔϏε + API Gatewayͳύλʔϯ αʔϏε͸ 6000RPSग़Δʁ
  23. εϧʔϓοτͷ୲อ εϧʔϓοτ͕৳ͼͳ͘ͳͬͨΒ.... • Ͳ͜ʹϘτϧωοΫ͕དྷ͍ͯΔ͔Λ֬ೝ͢Δ ◦ CPU࢖༻཰ɺϝϞϦ࢖༻཰ɺIOPSɺϩʔυΞϕϨʔδɺJVMͷώʔϓɺ ίωΫγϣϯϓʔϧɺΩϟογϡώοτ཰... • ϘτϧωοΫΛҠಈͤ͞Α͏ʂ ◦

    ਫฏεέʔϧɺਨ௚εέʔϧɺΞϓϦέʔγϣϯνϡʔχϯάɺ࠷దͳ ϊʔυ΁ͷ഑ஔɺϧʔςΟϯάͷ࢓ํͷݟ௚͠Λ࣮ࢪ
  24. 3. εύΠΫʹඋ͑Α͏ ٸܹͳ Traffic૿ʂ

  25. εύΠΫͰ͸ͳ͘؇΍͔ͳτϥϑΟοΫ૿ͳΒɾɾɾ • Horizontal Pod Autoscaler + Cluster Autoscaler ͰରԠͰ͖Δ εύΠΫʹඋ͑Α͏

  26. εύΠΫʹඋ͑Α͏ ~Horizontal Pod Autoscaler~ HPAͷ࢓૊Έ • 30ඵ͝ͱʹCPU࢖༻཰ɺϦΫΤετ਺ͳͲͷϝ τϦΫεΛνΣοΫ͠ɺPod਺Λܭࢉ͢Δ • ࠷େͰ3෼ʹ1ճεέʔϧΞ΢τɺ5෼ʹҰճε

    έʔϧΠϯ ࢀߟ: Horizontal Pod Autoscaler
  27. εύΠΫʹඋ͑Α͏ ~Horizontal Pod Autoscaler~ HPAͷܭࢉࣜ desiredReplicas = ceil[currentReplicas * (

    currentMetricValue / desiredMetricValue )] εέʔϧޙPod਺ = ceil [4 * (90 / 60)] = 6 ܭࢉྫ ܭࢉର৅ϝτϦΫε: CPU࢖༻཰ Target CPU࢖༻཰: 60% ݱࡏͷReplica਺ = 4 ݱࡏͷPodͷฏۉCPU࢖༻཰: 90%
  28. εύΠΫʹඋ͑Α͏ ~Cluster Autoscaler~ Cluster Autoscalerͷ࢓૊Έ request͞ΕͨϦιʔε͕଍Γͳ͘ͳΓɺ Pod͕഑ஔͰ͖ͳ͘ͳͬͨλΠϛϯάͰϊʔυ͕εέʔϧ͢Δ Node1 Node2 Full!

    Full! εέδϡʔϧ ͢Δ ͱ͜Ζ͕ͳ͍...
  29. εύΠΫʹඋ͑Α͏ ~Cluster Autoscaler~ Cluster Autoscalerͷ࢓૊Έ request͞ΕͨϦιʔε͕଍Γͳ͘ͳΓɺ Pod͕഑ஔͰ͖ͳ͘ͳͬͨλΠϛϯάͰϊʔυ͕εέʔϧ͢Δ Node1 Node2 Node3

    ϊʔυ௥Ճ Scheduled!
  30. εύΠΫʹඋ͑Α͏ HPA + Cluster Autoscalerͷ໰୊఺ ϊʔυɺPod͕εέʔϧ͢Δ·ͰͷҰఆͷϦʔυλΠϜ͕͔͔ΔͷͰ εύΠΫతͳ૿Ճͩͱؒʹ߹Θͳ͍

  31. εύΠΫʹඋ͑Α͏ ରԠࡦ • ͕࣌ؒ༧ଌͰ͖Δ৔߹ (CMޮՌɺYahoo๒ͳͲ) ◦ CronJobͳͲͰಛఆ࣌ؒʹHPAͷminReplica਺Λ্͛Δ • ͕࣌ؒ༧ଌͰ͖ͳ͍৔߹ ◦

    ྫ͑͹CPU࢖༻཰ͷ໨ඪ஋ΛΏΔ͓ͯ͘͘͠ ◦ ༧ΊminReplica਺ΛੵΜͰ͓͘ ◦ CDNͳͲͷΩϟογϡઓུΛݟ௚͢
  32. 4. ϊʔυͷμ΢ϯʹඋ͑Α͏ Node1 Node2 Node3 Evict!

  33. ϊʔυͷμ΢ϯʹඋ͑Α͏ ϊʔυ͕μ΢ϯ͢Δཧ༝ • ϋʔυ΢ΣΞো֐ • κʔϯɺϦʔδϣϯো֐ • ΫϥελΞοϓάϨʔυ • Մ༻ੑͷอূ͞Εͳ͍ϊʔυ

    (PreemptibleϊʔυɺSpotΠϯελϯε) • ΦϖϨʔγϣϯϛε
  34. ϊʔυͷμ΢ϯʹඋ͑Α͏ ϊʔυͷμ΢ϯʹର͢Δରࡦ 1. Podͷ৑௕Խͱκʔϯ෼ࢄ 2. ҆શͳPodͷఀࢭ ◦ Graceful Shutdownͷઃఆ ◦

    Podͷద੾ͳୀආઓུ 3. Ϋϥελͷਖ਼͍͠ઃఆ ◦ ϝϯςφϯε΢Οϯυ΢ͱSurge Upgradeͷઃఆ ◦ Մ༻ੑඇอূϊʔυͷਖ਼͍͠ӡ༻
  35. 4.1 ϊʔυͷμ΢ϯʹඋ͑Α͏ Node1 Node2 Node3 Evict! - Podͷ৑௕Խͱκʔϯ෼ࢄ -

  36. ϊʔυͷμ΢ϯʹඋ͑Α͏ ~Podͷ৑௕Խͱκʔϯ෼ࢄ~ Podͷ෼ࢄઓུ ~഑ஔϊʔυͷ෼ࢄ~ Node1 Node2 serviceA serviceA Pod Anti

    AffinityΛ׆༻ͯ͠ɺ ಉαʔϏεͷPod͕ ಉ͡ϊʔυʹ ͳΔ΂͘഑ஔ͞Εͳ͍Α͏ʹ͢Δ
  37. ϊʔυͷμ΢ϯʹඋ͑Α͏ ~Podͷ৑௕Խͱκʔϯ෼ࢄ~ Podͷ෼ࢄઓུ ~κʔϯͷ෼ࢄ~ Node1 asia-northeast1-a serviceA serviceA Node2 asia-northeast1-b

    Pod Anti AffinityΛ׆༻ͯ͠ɺ ಉαʔϏεͷPod͕ ಉ͡κʔϯʹ ͳΔ΂͘഑ஔ͞Εͳ͍Α͏ʹ͢Δ 1.18Ҏ্͸ Topology Spread Constraints ͕Φεεϝʂ ϦʔδϣφϧΫϥελʂ
  38. 4.2 ϊʔυͷμ΢ϯʹඋ͑Α͏ Node1 Node2 Node3 Evict! - ҆શͳPodͷఀࢭ -

  39. ϊʔυͷμ΢ϯʹඋ͑Α͏ ~҆શͳఀࢭ~ ϊʔυ͕ఀࢭ͢Δͱ͖ Node1 Node2 Node3 Schedule͞Ε͍ͯΔ Pod͕ऴྃ͞ΕΔ Terminate!

  40. ϊʔυͷμ΢ϯʹඋ͑Α͏ ~҆શͳఀࢭ~ ϊʔυ͕ఀࢭ͢Δͱ͖ Node1 Node2 Node3 ผͷϊʔυͰ৽ͨʹCreate

  41. ϊʔυͷμ΢ϯʹඋ͑Α͏ ~҆શͳఀࢭ~ ҆શͳӡ༻ͷͨΊʹ͸ PodΛ҆શʹఀࢭ͢Δॲཧ͕ඞཁෆՄܽ

  42. Graceful Shutdownͷઃఆ ৽ͨͳϦΫΤετ͸ࢭΊͭͭɺॲཧதͷϦΫΤετͷ׬ྃΛ଴͔ͬͯΒϓϩηε Λམͱͨ͢Ίͷઃఆ ϊʔυͷμ΢ϯʹඋ͑Α͏ ~҆શͳఀࢭ~ ϦΫΤετͷ ॲཧΛ׬ྃͤ͞Δ

  43. Podͷऴྃ࣌ͷڍಈ ϊʔυͷμ΢ϯʹඋ͑Α͏ ~҆શͳఀࢭ~ Podͷऴྃ PreStopॲཧ SIGTERM SIGTERM ॲཧ SIGKILL Endpoint͔ΒPodΛ࡟আ

  44. Podͷऴྃ࣌ͷڍಈ ϊʔυͷμ΢ϯʹඋ͑Α͏ ~҆શͳఀࢭ~ Podͷऴྃ PreStopॲཧ SIGTERM SIGTERM ॲཧ SIGKILL Endpoint͔ΒPodΛ࡟আ

    ̎ͭͷϥΠϯ͸ಉ࣌ʹॲཧ͕૸Δ
  45. Podͷऴྃ࣌ͷڍಈ ϊʔυͷμ΢ϯʹඋ͑Α͏ ~҆શͳఀࢭ~ Podͷऴྃ PreStopॲཧ SIGTERM SIGTERM ॲཧ SIGKILL Endpoint͔ΒPodΛ࡟আ

    Service Endpoint ϧʔςΟϯά͕ࢭ·Δʂ
  46. Podͷऴྃ࣌ͷڍಈ ϊʔυͷμ΢ϯʹඋ͑Α͏ ~҆શͳఀࢭ~ Podͷऴྃ PreStopॲཧ SIGTERM SIGTERM ॲཧ SIGKILL Endpoint͔ΒPodΛ࡟আ

    ೚ҙͷίϚϯυ ॲཧ ΞϓϦέʔγϣϯ ଆͰϋϯυϧ
  47. Podͷऴྃ࣌ͷڍಈ ϊʔυͷμ΢ϯʹඋ͑Α͏ ~҆શͳఀࢭ~ Podͷऴྃ PreStopॲཧ SIGTERM SIGTERM ॲཧ SIGKILL Endpoint͔ΒPodΛ࡟আ

    ͜͏ͳΔͱࢮΜͩίϯςφʹ ϧʔςΟϯά͞ΕΔ
  48. Pod΁ͷϧʔςΟϯάఀࢭΛ଴ͭ ϊʔυͷμ΢ϯʹඋ͑Α͏ ~҆શͳఀࢭ~ PreStopॲཧ SIGTERM ॲཧ Endpoint͔ΒPodΛ࡟আ lifecycle: preStop: exec:

    command: ["/bin/sh", "-c", "sleep 10"]
  49. ॲཧதͷϦΫΤετ͸ॲཧ׬ྃΛ଴ͬͯϓϩηεΛऴྃ͢Δ ϊʔυͷμ΢ϯʹඋ͑Α͏ ~҆શͳఀࢭ~ PreStopॲཧ SIGTERM ॲཧ Endpoint͔ΒPodΛ࡟আ ΞϓϦέʔγϣϯShutdownॲཧ

  50. Pod Disruption Budget ϊʔυ͕PodΛഉग़͢Δͱ͖ʹಉ࣌ʹఀࢭ͢ΔPodͷ਺Λ੍ޚ͢ΔͨΊͷϦιʔε ϊʔυͷμ΢ϯʹඋ͑Α͏ ~҆શͳఀࢭ~ ࢀߟ: Disruption Node ࢦఆͷ਺ͣͭ

    ഉग़ apiVersion: policy/v1beta1 kind: PodDisruptionBudget metadata: name: sample spec: maxUnavailable: "25%" selector: matchLabels: app: sample
  51. 4.3 ϊʔυͷμ΢ϯʹඋ͑Α͏ Node1 Node2 Node3 Evict! - Ϋϥελͷઃఆ -

  52. ΫϥελͷAuto Upgradeʹඋ͑Δ • ϝϯςφϯε΢Οϯυ΢͸τϥϑΟοΫ͕গͳ͍࣌ؒʹઃఆ͢Δ • Surge UpgradeΛઃఆͯ͠ɺॱ൪ʹUpgrade͞ΕΔΑ͏ʹ͢Δ ϊʔυͷμ΢ϯʹඋ͑Α͏ ~ΫϥελͷΞοϓάϨʔυ~

  53. Preemptibleϊʔυͷӡ༻ ϊʔυͷμ΢ϯʹඋ͑Α͏ ~ඇՄ༻ੑอূͷϊʔυͷӡ༻~ Node1 ௨ৗͷNode Pool Node2 Node1 Preemptible Node

    Pool Node2 • ྆ํͷNode Poolʹ ഑ஔ͢Δ • ॏཁͳPod͸഑ஔ͠ͳ ͍ • શମͷϊʔυ਺ͷҰ෦ ʹݶఆ͢Δ ServiceA ServiceB ServiceA ServiceB
  54. 5. Ϋϥελͷϝϯςφϯεઓུ Ingress Gateway Service A Service B Service C

    LB Ingress Gateway Service A Service B Service C
  55. ΫϥελϝϯςφϯεʹవΔ໰୊఺ ΫϥελͷϝϯςφϯεʹవΔϦεΫ • k8sͷόʔδϣϯΞοϓͰյΕΔΞϓϦέʔγϣϯ͸ͳ͍͔ʁ • Istioͷߋ৽ͳͲͰαʔϏεͷૄ௨͕ࢮ͵Մೳੑ • Ϧʔδϣϯো֐ͰҰ੪ʹࢮ͵Մೳੑ

  56. ΫϥελϝϯςφϯεʹవΔ໰୊఺ Ϋϥελϝϯςφϯε΁ͷରॲ ΫϥελΛ৑௕Խͯ͠ϚϧνΫϥελԽ͢Δ

  57. ϚϧνΫϥελͰՄ༻ੑΛ޲্ Ingress Gateway Service A Service B Service C LB

    Ingress Gateway Service A Service B Service C ΫϥελΛ৑௕Խ ୯ҰͷVIPΛఏڙ͢ΔLB
  58. ϚϧνΫϥελͰՄ༻ੑΛ޲্ Ingress Gateway Service A Service B Service C LB

    Ingress Gateway Service A Service B Service C Ϋϥελͷߋ৽࣌
  59. ϚϧνΫϥελͰՄ༻ੑΛ޲্ Ingress Gateway Service A Service B Service C LB

    Ingress Gateway Service A Service B Service C LB͔Β੾Γ཭ͯ͠ Ϋϥελߋ৽࡞ۀ
  60. ϚϧνΫϥελͰՄ༻ੑΛ޲্ Ingress Gateway Service A Service B Service C LB

    Ingress Gateway Service A Service B Service C ࠶ϧʔςΟϯά
  61. ϚϧνΫϥελͰՄ༻ੑΛ޲্ Ingress Gateway Service A Service B Service C LB

    Ingress Gateway Service A Service B Service C ໰୊ͳ͚Ε͹ ͪ͜Β΋ߋ৽
  62. ϚϧνΫϥελͰՄ༻ੑΛ޲্ Ingress Gateway Service A Service B Service C LB

    Ingress Gateway Service A Service B Service C ࠶ϧʔςΟϯά
  63. ϚϧνΫϥελͰՄ༻ੑΛ޲্ Ingress Gateway Service A Service B Service C LB

    Ingress Gateway Service A Service B Service C Ϋϥελͷ ϩʔϦϯάΞοϓσʔτΛ࣮ݱ
  64. ϚϧνΫϥελͰՄ༻ੑΛ޲্ Ingress Gateway Service A Service B Service C LB

    Ingress Gateway Service A Service B Service C Ϧʔδϣϯো֐Ͱ΋ ϑΣΠϧΦʔόʔ
  65. ϚϧνΫϥελͷ࣮ݱํ๏ • GCLB + NEGΛ࢖ͬͨϧʔςΟϯάΛࣗಈ Ͱߏங • ManagedͳService MeshΛఏڙ •

    Observabilityͷ୲อͱSLO/SLIϞχλϦϯ ά • ΫϥελϦιʔεͷΫϥελؒಉظ GCP Anthos
  66. ϚϧνΫϥελʹΑΔ෇ՃՁ஋ Մ༻ੑ޲্͚ͩͰͳ͘ɺ༷ʑͳϝϦοτ΋ • κʔϯΫϥελΛϚϧνΫϥελԽ͢Δ͜ͱͰϦʔδϣφϧΑΓ ҆͘ࡁΉՄೳੑ • VIP + IP AnycastʹΑΔ௿ϨΠςϯγԽ

  67. ·ͱΊ • ϨΠςϯγΛ୲อ͢Δ ◦ requestͱlimitΛద੾ʹઃఆ͠Α͏ʂ ◦ ࠷దͳϦιʔεΛ࣋ͭϊʔυʹ഑ஔ͠Α͏ʂ ◦ ΞϓϦέʔγϣϯΛνϡʔχϯά͠Α͏

  68. ·ͱΊ • εϧʔϓοτΛ୲อ͢Δ ◦ Ͳ͜ʹϘτϧωοΫ͕དྷ͍ͯΔͷ͔ΛՄࢹԽͯ͠νϡʔχϯά͍ͯ͜͠͏ ◦ PodͷਫฏεέʔϧɺrequestͱlimitʹΑΔਨ௚εέʔϧΛ͠Α͏

  69. ·ͱΊ • εύΠΫʹඋ͑Α͏ ◦ HPAͱCluster Autoscalerͷ࢓૊ΈΛཧղͯ͠࢖͓͏ ◦ ෆे෼ͳ৔߹͸CronJobͳͲͰReplica਺Λௐ੔ͨ͠ΓɺεέʔϧͷϝτϦΫε໨ඪ஋Λ ΏΔͨ͘͠Γ͠Α͏

  70. ·ͱΊ • ϊʔυͷμ΢ϯʹඋ͑Α͏ ◦ κʔϯɺϊʔυΛ෼ࢄͯ͠PodΛ഑ஔ͠Α͏ ◦ Graceful ShutdownɺPDBΛద੾ʹઃఆͯ҆͠શʹఀࢭ͠Α͏ ◦ Ϋϥελͷϝϯςφϯε࣌ؒɺϝϯςφϯεํ਑ΛͪΌΜͱઃఆ͠Α͏

    ◦ Preemptible͸৻ॏʹʂ
  71. ·ͱΊ • Ϋϥελͷϝϯςφϯεઓུ ◦ ϚϧνΫϥελԽ͢Δͱ҆શʹΫϥελΛΧφϦϦϦʔεɺϩʔϦϯάΞοϓσʔτͰ͖ Δ ◦ ௿ϨΠςϯγԽͳͲͷ෇ՃՁ஋΋ଘࡏ ◦ GCP

    AnthosΛ࢖͏ͱ୯ҰVIPʹෳ਺ΫϥελΛͿΒԼ͛Δߏ੒ΛͱΔ͜ͱ͕Ͱ͖Δ
  72. Follow Me!! @taisho6339

  73. Thank you for listening!