Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Is Kubernetes On-premises Hardway?

Is Kubernetes On-premises Hardway?

CloudNativeDays Tokyo 2021にて登壇した資料です。

1b838da2065660793d5b26f2cdc32de7?s=128

Kazuhiko Yamashita

November 05, 2021
Tweet

More Decks by Kazuhiko Yamashita

Other Decks in Technology

Transcript

  1. Is Kubernetes On-premises Hardway? ʙ݁ࠗɺͦΕ͸ର࿩Ͱ͋Δʙ

  2. ࢁԼ࿨඙!QZBNB (.0ϖύϘٕज़ج൫νʔϜ γχΞɾϓϦϯγύϧ UFOTOBQPODPN QZBNBGVO TUOTKQ

  3. 45/4 -JOVY/444FSWFS TUOTKQ

  4. 45/4

  5. https://github.com/pyama86/github-replacer

  6. ϗεςΟϯάࣄۀ &$ࢧԉࣄۀ ϋϯυϝΠυɾͦͷଞࣄۀ

  7. Is Kubernetes On-premises Hardway?

  8. ϚωʔδυαʔϏεͷϝϦοτ • Control Plane/Data Planeͷ؅ཧ • όʔδϣϯΞοϓͷ༰қ͞ • Ϋϥ΢υࣄۀऀ͕ఏڙ͢ΔͦͷଞͷϚωʔδυαʔϏεͱͷ࿈ܞͷ༰қ͞ •

    ແݶεέʔϦϯά(If you have much money)
  9. ΦϯϓϨϛεͷϝϦοτ • ͢΂ͯΛ΍ΒͶ͹ͳΒͳ͍͕ނʹࣗ༝ • ਓ݅අΛআ͘ϥϯχϯάίετͷ҆͞

  10. ࠓ೔࿩͢͜ͱ • ϖύϘͷKubernetesΫϥελͷ֓ཁ • Hardwayͩͬͨ͜ͱ • ࠓޙ΍Γ͍ͨ͜ͱ

  11. KubernetesΫϥελ • OpenStack (Nyah) • Nyah Kubernetes Engine(NKE)

  12. KubernetesΫϥελ ن໛ײ • ঎ࡐ͝ͱʹΫϥελΛ෼཭͓ͯ͠Γɺ23Ϋϥελ(ൃද࣌఺) • ঎ࡐʹΑͬͯ͸NKE / GKE / EKSͰͷϋΠϒϦουΫϥ΢υͰར༻

    • AWS Direct ConnectͰઐ༻ઢར༻
  13. KubernetesΫϥελ ٕज़ج൫νʔϜ Embedded SRE • NKEίϚϯυͷ։ൃ • ϓϦηοτϚχϑΣετͷߋ৽ • Ϋϥελ্Ͱಈ͘ιϑτ΢ΣΞͷಋೖ

    • όʔδϣϯΞοϓͳͲͷϝϯςφϯε ։ൃͱར༻ऀ͕ҟͳΔ
  14. NKE • ΫϥελͷߏஙɺόʔδϣϯΞοϓ • Ϋϥελ؅ཧϚχϑΣετͷద༻ • Data Planeͷ௥Ճɺ࡟আ • AnsibleΛ༻͍ͨϓϩϏδϣχϯά

    Ϋϥελ؅ཧΛίʔυԽ͠CLIΠϯλʔϑΣʔεʹͨ͠΋ͷ
  15. NKE ઃఆϑΝΠϧɺൿಗ৘ใετΞʹج͖ͮɺ ΫϥελΛߏஙɺӡ༻ VM VM VM NKE tenant- con fi

    g.toml Hashicorp Vault conta iner conta iner conta iner
  16. NKE • Golang • Hashicorp Vault • Consul • Packer

    ओཁίϯϙʔωϯτ
  17. Kubernetesͷόʔδϣϯ؅ཧ

  18. Kubernetesͷόʔδϣϯ؅ཧ • NKEͷϒϥϯν͝ͱʹόʔδϣϯ؅ཧ • trunk: ։ൃ༻ϒϥϯν • 1.20,1.21 ϦϦʔεϒϥϯν

  19. Kubernetesͷόʔδϣϯ؅ཧ trunk 1.20 1.21 Unit Test E2E Test Unit Test

    E2E Test Unit Test E2E Test merge merge
  20. Kubernetesͷόʔδϣϯ؅ཧ • CIΛར༻ͨ͠ςετΛύεͨ͠৔߹͸ɺ։ൃ༻Ϋϥελɺࣾ಺πʔϧ༻Ϋ ϥελͷόʔδϣϯΞοϓίϚϯυΛ࣮ߦ • ֤Ϋϥελͷ؅ཧऀ͕όʔδϣϯΞοϓίϚϯυΛ࣮ߦ • ΫϥελʹΑͬͯ͸2ܥ࣋ͭΑ͏ʹͯ͠ɺόʔδϣϯΞοϓ࡞ۀͳͲͷ
 μ΢ϯλΠϜΛආ͚Δ޻෉Λ͍ͯ͠Δ

  21. Kubernetesͷόʔδϣϯ؅ཧ • Control Plane,Data Planeͱ΋ʹPodΛ௥͍ग़ͭͭ͠ɺ
 ϩʔϦϯάΞοϓσʔτ • Control PlaneɺEtcdʹ͍ͭͯ͸1୆ೖΕସ͑͝ͱʹϔϧενΣοΫΛ
 ͍Εͯμ΢ϯλΠϜΛආ͚͍ͯΔ

  22. Kubernetesͷӡ༻؅ཧ

  23. Kubernetesͷӡ༻؅ཧ • ؂ࢹ • ηΩϡϦςΟ؂ࠪ • CI/CD • ϩά؅ཧ

  24. Kubernetesͷ؂ࢹ Prometheus Alert Manager Grafana mackerel-agent ࣌ܥྻσʔλͷอଘ ڞ௨ϧʔϧʹै͍ɺSlack௨஌ PrometheusͷσʔλͷϏδϡΞϥΠθʔγϣϯ Prometheus+AlertManagerͷ؂ࢹ

  25. KubernetesͷηΩϡϦςΟ؂ࠪ • Wazuh • Falco • GateKeeper

  26. Wazuh https://atmarkit.itmedia.co.jp/ait/articles/1902/18/news012.html OSͷઃఆ؂ࠪ ෆਖ਼ΞΫηεݕ஌ ੬ऑੑ؂ࠪ

  27. Falco ίϯςφͷৼΔ෣͍؂ࠪɾݕ஌

  28. Gatekeeper Admission ControllerͰಈ࡞͢Δ ϚχϑΣετͷ؂ࠪͳͲ Ұॹʹ΍ͬͯ͘Δਓɺೖࣾͯ͘͠Εʙʙʙʙ

  29. ࣗಈApply ؂ࢹɺηΩϡϦςΟϙϦγʔ͸Ұ੪഑෍ tag cluster A cluster B cluster C apply

  30. CI/CD • ςετɺίϯςφϏϧυɺ੬ऑੑεΩϟϯ͸Github ActionsͷSelf Hosted Runner্Ͱ࣮ߦ • ίϯςφΠϝʔδͷεΩϟϯΤϯδϯ͸trivyΛར༻ • CD͸ArgoCD

    + argocd-image-updaterΛར༻
  31. ϩά؅ཧ Kafkaʹू໿ͯ͠ɺ༻్ʹԠͯ͡SaaS΁

  32. ͜͜·Ͱ࿩ͨ͜͠ͱ • NKEίϚϯυͷ։ൃʹΑͬͯΫϥελͷߏங΍ϝϯςφϯεΛࣗಈԽͯ͠ ͍Δ • ؂ࢹ΍ηΩϡϦςΟ؂ࠪʹ͍ͭͯ͸NKEͰϕʔεͱͳΔ΋ͷΛఏڙ • όʔδϣϯΞοϓʹ͍ͭͯ͸E2EͰಈ࡞Λ୲อͭͭ͠ɺ։ൃ༻ΫϥελͰ ໰୊͕ͳ͍͜ͱΛ֬ೝͯ͠ɺద༻͍ͯ͠Δ

  33. Hardwayͩͬͨ͜ͱ

  34. 1.12.7

  35. [࠶ܝ]KubernetesΫϥελ ٕज़ج൫νʔϜ Embedded SRE • NKEίϚϯυͷ։ൃ • ϓϦηοτϚχϑΣετͷߋ৽ • Ϋϥελ্Ͱಈ͘ιϑτ΢ΣΞͷಋೖ

    • όʔδϣϯΞοϓͳͲͷϝϯςφϯε ։ൃͱར༻ऀ͕ҟͳΔ
  36. όʔδϣϯΞοϓͷಈػ͕௿͍͜ͱ͕͋Δ • Ϋϥελͷ༻్ • ୲౰ऀ͕ଟ๩ • Kubernetesɺ͍͍ͩͨݹͯ͘΋ಈ͘ • όʔδϣϯΞοϓʹର͢Δ৺ཧোน

  37. όʔδϣϯ؅ཧࣗಈԽ͍ͨ͠ NKE Manifests Cluster A NKE Manifests Cluster B NKE

    Manifests Cluster C NKE Cluster A Cluster B Cluster C manifestͷఆٛʹج͍ͮͯࣗಈͰऩଋͯ͠΄͍͠
  38. ͋Δ೔ಥવͷ ”error: You must be logged in to the server

    (Unauthorized)”
  39. Կ͕ى͖͔ͨ kube-apiserver Service Account token ServiceAccountͷར༻͍ͯ͠ΔτʔΫϯ͕ࣦޮͯ͠ೝূΤϥʔ

  40. ͳͥى͖͔ͨ • Kubernetes ͷ SAτʔΫϯ͸༗ޮظݶ͕Forever • ϖύϘͷKubernetesͷSAτʔΫϯͷ伴͸ࣗಈͰϩʔςʔγϣϯ͍ͯ͠Δ

  41. Կ͕ى͖͔ͨ kube-apiserver Service Account token ূ໌ॻɺ伴ͷߋ৽ɺ഑෍͸HashicorpVaultͰࣗಈԽ kube-controller- manager τʔΫϯͷ෷͍ग़͠ τʔΫϯͷݕূ

    Hashicorp Vault Cert Key ূ໌ॻͱ伴ͷࣗಈ഑෍
  42. Կ͕ى͖͔ͨ kube-apiserver Service Account token ূ໌ॻɺ伴ͷߋ৽ɺ഑෍͸HashicorpVaultͰࣗಈԽ kube-controller- manager τʔΫϯͷ෷͍ग़͠ τʔΫϯͷݕূ

    Hashicorp Vault Cert Key ূ໌ॻͱ伴ͷࣗಈ഑෍ Ӭٱอଘʂʂʂ
  43. Կ͕ى͖͔ͨ kube-apiserver Service Account token ূ໌ॻɺ伴ͷߋ৽ɺ഑෍͸Hashicorp VaultͰࣗಈԽ kube-controller- manager τʔΫϯͷ෷͍ग़͠

    τʔΫϯͷݕূ Hashicorp Vault Cert Key ূ໌ॻͱ伴ͷࣗಈ഑෍ 伴͕ߋ৽͞ΕΔ͜ͱͰ ݕূ͕Ͱ͖ͳ͘ͳΔ
  44. ରॲʂѹ౗త࢑ఆରॲʂʂʂ ಈ͍͍ͯΔϙου͸ಈ͖ଓ͚Δ͕ɺϦεέδϡʔϧ͕Ͱ͖ͳ͍ͷͰ ·ͣ͸ShellͰରॲ ͜ͷ͋ͱɺূ໌ॻͷঢ়گΛ؂ࢹͯ͠ஔ͖׵͑ΔϓϩηεΛಈ͔͍ͯ͠·͢

  45. ࠷ޙͷॴײ • ΦϯϓϨKubernetes΍ΔͳΒ؅ཧιϑτ΢ΣΞΛ։ൃͨ͠΄͏͕౷߹తʹ؅ཧͰ͖ΔͷͰ
 ࠷ऴίετ͸མͱͤΔͱࢥ͏ • Kubernetesͦͷ΋ͷ͸ͱͯ΋ྑ͘Ͱ͖͍ͯͯɺKubernetesࣗମͷԿ͔Λ౿Ή͜ͱ͸ͦΜͳʹͳ͍ • ࠓ೔঺հ͍ͯ͠ͳ͍ωοτϫʔΫपΓͷΧʔωϧνϡʔχϯάͳͲɺඞཁͳέʔε΋͋ͬͨͷ Ͱɺͦ͏͍͏ྖҬΛݟΕΔਓ͕͍ͳ͍ͱݫ͍͠ͱ͸ࢥ͏ •

    େମͷϢʔεέʔε͸VM + DockerͰࣄ଍ΓΔͷͰɺ΄ΜͱʹͦΕKubernetes͍Δͷʁͱ͍͏ έʔε͸݁ߏ͋Δͱࢥ͏ • ͜Ε·Ͱͷιϑτ΢ΣΞʹՃ͑ͯɺKubernetesͱ͍͏ϨΠϠʔ͕ೖΔ͜ͱͰτϥϒϧγϡʔτ ΍؅ཧ͸େมʹͳΔ
  46. ͓͠·͍ ࠷৽ͷ࠾༻৘ใΛνΣοΫˠ !QC@SFDSVJU