Upgrade to Pro — share decks privately, control downloads, hide ads and more …

kube-system落としてみました

657fb6673f7551440601417843ec5152?s=47 tarosaiba
March 18, 2019

 kube-system落としてみました

kube-system 意識していますか? (私は意識してませんでした)
kube-dns が死んだりすると、サービス停止しちゃいます (経験談)
せっかくなのでkube-systemをそれぞれ落として、影響を調べてみました。 (反省を兼ねて)
今回はGKEで試しています

657fb6673f7551440601417843ec5152?s=128

tarosaiba

March 18, 2019
Tweet

Transcript

  1. Kubernetes Meetup Tokyo #16 LT 1  4IVOUBSP4BJCB kube-systemམͱͯ͠Έ·ͨ͠

  2. ຊ೔͓఻͍͑ͨ͜͠ͱ ˔  LVCFTZTUFNҙ͍ࣝͯ͠·͔͢ʁ ࢲ͸ҙࣝͯ͠·ͤΜͰͨ͠  ˔  LVCFEOT͕ࢮΜͩΓ͢ΔͱɺαʔϏεఀࢭͪ͠Ό͍·͢ ܦݧஊ  ˔ 

    ͔ͤͬ͘ͳͷͰLVCFTZTUFNΛͦΕͧΕམͱͯ͠ɺ ӨڹΛௐ΂ͯΈ·ͨ͠ɻ ൓লΛ݉Ͷͯ  ˔  ࠓճ͸(,&Ͱࢼ͍ͯ͠·͢  2
  3. @tarosaiba ˔  'VUVSF"SDIJUFDU *OD ˔  5FDIOPMPHZ*OOPWBUJPO(SPVQ ˔  *OGSB.8$*पΓ΍͖ͬͯ·ͨ͠ ˔  ΤϯϓϥͰ"84࠷ۙ͸($1

    4IVOUBSP4BJCB 3
  4. 4 ࢖ͬͨ؀ڥ Microservices Kubernetes Engine Nodes ˔  (,&Λ/PEFͰཱͯ·ͨ͠ɻ EFGBVMUͷBEEPOΛ͢΂ͯ༗ޮ 

    ˓  WFSTJPO(,&HLF 
  5. 5 ࢖ͬͨ؀ڥ ˔  ,VCFSOFUFTͷαϯϓϧΞϓϦlHVFTUCPPLzΛσϓϩΠ  https://cloud.google.com/kubernetes-engine/docs/tutorials/guestbook

  6. ɹɹɹɹࠓճݟΔ΋ͷ 6

  7. 7 # Get kube-system pods > kubectl get pods --namespace=kube-system

    NAME READY STATUS RESTARTS AGE event-exporter-v0.2.3-85644fcdf-bzddn 2/2 Running 0 3h fluentd-gcp-scaler-8b674f786-9tp2z 1/1 Running 0 3h fluentd-gcp-v3.2.0-6gq6r 2/2 Running 0 3h fluentd-gcp-v3.2.0-8h5hp 2/2 Running 0 3h fluentd-gcp-v3.2.0-pb44f 2/2 Running 0 2h heapster-v1.6.0-beta.1-74f4bbc55c-52khj 3/3 Running 0 3h kube-dns-7df4cb66cb-9ft98 4/4 Running 0 3h kube-dns-7df4cb66cb-wvhpz 4/4 Running 0 2m kube-dns-autoscaler-67c97c87fb-xgrx6 1/1 Running 0 3h kube-proxy-gke-demo-default-pool-0dacde9e-dvbd 1/1 Running 0 3h kube-proxy-gke-demo-default-pool-0dacde9e-gc4m 1/1 Running 0 3h kube-proxy-gke-demo-default-pool-0dacde9e-lr6s 1/1 Running 0 2h l7-default-backend-7ff48cffd7-vfmxz 1/1 Running 0 3h metrics-server-v0.2.1-fd596d746-xz8s2 2/2 Running 0 3h OBNFTQBDFʹzLVCFTZTUFNzΛࢦఆ
  8. k8sΛࢧ͑Δొ৔ਓ෺ ͓͞Β͍  8 kube-apiserver etcd kube-controller manager kube-scheduler kubelet

    kube-proxy container runtime ... ... Master Node GCP Managed Our Project kubelet kube-dns
  9. k8sΛࢧ͑Δొ৔ਓ෺ ͓͞Β͍  9 kube-apiserver etcd kube-controller manager kube-scheduler kubelet

    kube-proxy container runtime ... ... Master Node GCP Managed kubelet kube-dns
  10. k8sΛࢧ͑Δొ৔ਓ෺ ͓͞Β͍  10 kube-apiserver etcd kube-controller manager kube-scheduler kubelet

    kube-proxy container runtime ... ... Master Node GCP Managed Our Project kubelet kube-dns
  11. k8sΛࢧ͑Δొ৔ਓ෺ ͓͞Β͍  11 kube-apiserver etcd kube-controller manager kube-scheduler kubelet

    kube-proxy container runtime ... ... Master Node GCP Managed Our Project kubelet kube-dns ͪΌΜͱҙࣝͰ͖ͯͳ͔ͬͨ ൓ল 
  12. 12 Name ໾ׂ kube-apiserver ˔  LTͷϦιʔεͷૢ࡞Λߦ͏ͨΊͷ ,VCFSOFUFT"1*Λఏڙ etcd ˔  ෼ࢄ,FZ7BMVF4UPSF

    ˔  LTΫϥελͷ৘ใΛอଘ kube-scheduler ˔  ֤/PEFͷঢ়ଶΛݟͯɺ1PEͷׂ౰Λߦ͏εέδϡʔϥ kube-controller-manager ˔  ֤ίϯτϩʔϥʔΛ࣮ߦ͢Δ ˔  ྫ3FQMJDB4FUͷ1PE਺Λ؂ࢹͯ͠ɺ1PEͷొ࿥ΛߦͬͨΓ  ɹɹɹɹɾɾɾ ɹଞ GCP Managed ($1ϚωʔδυͰΑΖ͘͠ʹ
  13. Name Type ਺ ໾ׂ kubelet systemd 3 •  kube-apiserverͱ࿈ܞɺίϯςφىಈΛ͢Δ kube-proxy-*

    ୯ମ Pod 3 •  NWτϥϑΟοΫΛ؅ཧ kube-dns Deployment 2 •  Ϋϥελ಺ͷDNS kube-dns-autoscaler Deployment 1 •  kube-dnsͷεέʔϦϯάΛߦ͏ fluentd-gcp-v* DaemonSet 3 •  Stackdriver LoggingʹContainerͷϩάΛૹ͍ͬͯΔ fluentd-gcp-scaler Deployment 1 •  fluentdͷεέʔϦϯάΛߦ͏ event-exporter-v* Deployment 1 •  Stackdriver Loggingʹkube-apiserverͷϩάΛૹ͍ͬͯΔ metrics-server-v** Deployment 1 •  HPA༻ʹϝτϦΫεΛఏڙ heapster-v** Deployment 1 •  Stackdriver MonitoringʹϝτϦΫεΛૹ৴ 13 Our Project ˞(,&HLFͷ/PEF୆ͷ৔߹ ˞MEFGBVMUCBDLFOE͸ࠓճলུ
  14. Our Project ˞(,&HLFͷ/PEF୆ͷ৔߹ ˞MEFGBVMUCBDLFOE͸ࠓճলུ Name Type ਺ ໾ׂ kubelet systemd

    3 •  kube-apiserverͱ࿈ܞɺίϯςφىಈΛ͢Δ kube-proxy-* ୯ମ Pod 3 •  NWτϥϑΟοΫΛ؅ཧ kube-dns Deployment 2 •  Ϋϥελ಺ͷDNS kube-dns-autoscaler Deployment 1 •  kube-dnsͷεέʔϦϯάΛߦ͏ fluentd-gcp-v* DaemonSet 3 •  Stackdriver LoggingʹContainerͷϩάΛૹ͍ͬͯΔ fluentd-gcp-scaler Deployment 1 •  fluentdͷεέʔϦϯάΛߦ͏ event-exporter-v* Deployment 1 •  Stackdriver Loggingʹkube-apiserverͷϩάΛૹ͍ͬͯΔ metrics-server-v** Deployment 1 •  HPA༻ʹϝτϦΫεΛఏڙ heapster-v** Deployment 1 •  Stackdriver MonitoringʹϝτϦΫεΛૹ৴ 14 ݁ߏ͍Ζ͍Ζಈ͍ͯͨΜͩʂ ஌Βͳ͔ͬͨɻ ൓ল 
  15. ࢖͍Α͏ʹΑͬͯ͸ɺ མͪͯ͠·͏͜ͱ΋͋Δ͔΋  15

  16. ͳͷͰɺͻͱͭͣͭམͱͯ͠ ӨڹΛ֬ೝ͠·ͨ͠ 16

  17. 17 Our Project ˞(,&HLFͷ/PEF୆ͷ৔߹ ˞ ͸ͣ ະ֬ೝ ো֐࠶ݱଌఆ͕೉͔ͬͨ͠  Name

    Type ਺ ॏཁ౓ ࢭΊͨͱ͖ͷڍಈ kubelet systemd 3 •  αʔϏε͸Քಇ͢Δ͕ɺPodͷ࡟আ/ىಈ͕ෆՄʹͳͬͨ kube-proxy-* Pod 3 •  ଈ࣌ʹ෮چͨ͠ͷͰ֬ೝͰ͖ͳ͔͕ͬͨɺ ίϯςφؒ௨৴͕ෆՄʹͳΔ?ͨΊαʔϏεো֐ʹͳΔ(͸ͣ) kube-dns Deployment 2 •  Serviceܦ༝ͷAPP → DB ͷૄ௨ෆՄɻαʔϏεো֐ൃੜ kube-dns-autoscaler Deployment 1 •  αʔϏεʹӨڹͳ͠ɻεέʔϧૢ࡞͸ෆՄʹͳΔ(͸ͣ) fluentd-gcp-v* DaemonSet 3 •  αʔϏεʹӨڹͳ͠ɻϩά͕సૹͰ͖ͳ͘ͳΔ fluentd-gcp-scaler Deployment 1 •  αʔϏεʹӨڹͳ͠ɻεέʔϧૢ࡞͸ෆՄʹͳΔ(͸ͣ) event-exporter-v* Deployment 1 •  αʔϏεʹӨڹͳ͠ɻΠϕϯτϩά͕సૹͰ͖ͳ͘ͳΔ(͸ͣ) metrics-server-v** Deployment 1 •  αʔϏεʹӨڹͳ͠ɻHPA͕ػೳ͠ͳ͍ (͸ͣ) heapster-v** Deployment 1 •  αʔϏεʹӨڹͳ͠ɻϝτϦΫε͕సૹͰ͖ͳ͘ͳΔ(͸ͣ)
  18. 18 Our Project ˞(,&HLFͷ/PEF୆ͷ৔߹ ˞ ͸ͣ ະ֬ೝ ো֐࠶ݱଌఆ͕೉͔ͬͨ͠  Name

    Type ਺ ॏཁ౓ ࢭΊͨͱ͖ͷڍಈ kubelet systemd 3 •  αʔϏε͸Քಇ͢Δ͕ɺPodͷ࡟আ/ىಈ͕ෆՄʹͳͬͨ kube-proxy-* Pod 3 •  ଈ࣌ʹ෮چͨ͠ͷͰ֬ೝͰ͖ͳ͔͕ͬͨɺ ίϯςφؒ௨৴͕ෆՄʹͳΔ?ͨΊαʔϏεো֐ʹͳΔ(͸ͣ) kube-dns Deployment 2 •  Serviceܦ༝ͷAPP → DB ͷૄ௨ෆՄɻαʔϏεো֐ൃੜ kube-dns-autoscaler Deployment 1 •  αʔϏεʹӨڹͳ͠ɻεέʔϧૢ࡞͸ෆՄʹͳΔ(͸ͣ) fluentd-gcp-v* DaemonSet 3 •  αʔϏεʹӨڹͳ͠ɻϩά͕సૹͰ͖ͳ͘ͳΔ fluentd-gcp-scaler Deployment 1 •  αʔϏεʹӨڹͳ͠ɻεέʔϧૢ࡞͸ෆՄʹͳΔ(͸ͣ) event-exporter-v* Deployment 1 •  αʔϏεʹӨڹͳ͠ɻΠϕϯτϩά͕సૹͰ͖ͳ͘ͳΔ(͸ͣ) metrics-server-v** Deployment 1 •  αʔϏεʹӨڹͳ͠ɻHPA͕ػೳ͠ͳ͍ (͸ͣ) heapster-v** Deployment 1 •  αʔϏεʹӨڹͳ͠ɻϝτϦΫε͕సૹͰ͖ͳ͘ͳΔ(͸ͣ) ݸਓతɺؾΛ෇͚Α͏ϙΠϯτ %/4αʔό͕ಈ͍͍ͯΔͷ ҙࣝͯ͠ͳ͔ͬͨɻ ൓ল 
  19. kube-system Node Ͱಈ͍͍ͯΔ΍ͭ 19 kubelet kube-proxy container runtime Node kube-apiserver

    default APP-B APP-A GCP Service kubernetes Master Namespaces heapstar event exporter metrics server Monitoring API Logging API HPA མͪͯ΋αʔϏεӨڹͳ͠ མͪΔͱαʔϏεӨڹ͋Γ Stackdriver
  20. kube-system Node Ͱಈ͍͍ͯΔ΍ͭ 20 kubelet kube-proxy container runtime Node kube-dns

    kube-apiserver autoscaler fluentd-gcp autoscaler default APP-B APP-A GCP Service kubernetes Master Namespaces Monitoring API Logging API Stackdriver མͪͯ΋αʔϏεӨڹͳ͠ མͪΔͱαʔϏεӨڹ͋Γ
  21. kube-system Node Ͱಈ͍͍ͯΔ΍ͭ 21 kubelet kube-proxy container runtime Node kube-dns

    kube-apiserver autoscaler fluentd-gcp autoscaler default APP-B APP-A GCP Service kubernetes Master Namespaces Monitoring API Logging API Stackdriver ໊લղܾ͕Ͱ͖ͳ͘ͳΔ མͪͯ΋αʔϏεӨڹͳ͠ མͪΔͱαʔϏεӨڹ͋Γ
  22. ˔  LVCFTZTUFNͷ%FQMPZNFOU %BFNPOTFUΛEFMFUFͯ͠΋ɺ BEEPONBOBHFS͕Ϧιʔε͝ͱউखʹ෮׆ͤ͞ΔΜͰ͢Ͷɻ ˓  LVCFDUMEFMFUFEFQMPZNFOUΛ࿈ଧͯ͠ɺো֐Λ࠶ݱ͠·ͨ͠׼   ͓·͚ؾ͕͍ͭͨ͜ͱ 22

  23. ˔  LVCFTZTUFNͷো֐Ίͬͨʹى͖ͳ͍ͩΖ͏͕ɺҙࣝ͸ඞཁͱ࣮ײ ˓  ಛʹLVCFEOT ˓  ͪͳΈʹLVCFEOT͸୆Ҏ্Ͱ৑௕Խ͞Ε͍ͯΔ͕Ͱ΋  ˔  ࢓૊Έ͸ͪΌΜͱཧղ͓͔ͯ͠ͳ͍ͱ͍͚ͳ͍ͳͱ࣮ײ ˓ 

    ͳΜʹͰ΋ݴ͑Δ͜ͱͰ͕͢ ˓  LT൒೥ۙ͘࢖͍͕ͬͯͨɺࠓճௐ΂ͯॳΊͯΘ͔ͬͨ͜ͱ͕ͨ͘͞Μ ˓  ·ͩ·ͩษڧෆ଍ͳͷͰɺ͜Ε͔ΒͪΌΜͱௐ΂·͢ɻ  ·ͱΊ 23
  24. ͋Γ͕ͱ͏͍͟͝·ͨ͠ 24 ($1 (,& YΤϯϓϥ ΤϯδχΞઈࢍืूதͰ͢