Is Kubernetes On-premises Hardway?

Transcript

1. Is Kubernetes On-premises Hardway? ʙ݁ࠗɺͦΕ͸ର࿩Ͱ͋Δʙ

2. ࢁԼ
   ࿨඙
   
   !QZBNB
   (.0ϖύϘ
   ٕज़ج൫νʔϜ
   
   
   γχΞɾϓϦϯγύϧ
   
   
   
   UFOTOBQPODPN
   
   
   
   QZBNBGVO
   
   
   
   TUOTKQ

3. 45/4
   -JOVY
   /44
   4FSWFS
   TUOTKQ

4. 45/4

5. https://github.com/pyama86/github-replacer

6. ϗεςΟϯάࣄۀ &$ࢧԉࣄۀ ϋϯυϝΠυɾͦͷଞࣄۀ

7. Is Kubernetes On-premises Hardway?

8. ϚωʔδυαʔϏεͷϝϦοτ • Control Plane/Data Planeͷ؅ཧ • όʔδϣϯΞοϓͷ༰қ͞ • Ϋϥ΢υࣄۀऀ͕ఏڙ͢ΔͦͷଞͷϚωʔδυαʔϏεͱͷ࿈ܞͷ༰қ͞ •

   ແݶεέʔϦϯά(If you have much money)

9. ΦϯϓϨϛεͷϝϦοτ • ͢΂ͯΛ΍ΒͶ͹ͳΒͳ͍͕ނʹࣗ༝ • ਓ݅අΛআ͘ϥϯχϯάίετͷ҆͞

10. ࠓ೔࿩͢͜ͱ • ϖύϘͷKubernetesΫϥελͷ֓ཁ • Hardwayͩͬͨ͜ͱ • ࠓޙ΍Γ͍ͨ͜ͱ

11. KubernetesΫϥελ • OpenStack (Nyah) • Nyah Kubernetes Engine(NKE)

12. KubernetesΫϥελ ن໛ײ • ঎ࡐ͝ͱʹΫϥελΛ෼཭͓ͯ͠Γɺ23Ϋϥελ(ൃද࣌఺) • ঎ࡐʹΑͬͯ͸NKE / GKE / EKSͰͷϋΠϒϦουΫϥ΢υͰར༻

    • AWS Direct ConnectͰઐ༻ઢར༻

13. KubernetesΫϥελ ٕज़ج൫νʔϜ Embedded SRE • NKEίϚϯυͷ։ൃ • ϓϦηοτϚχϑΣετͷߋ৽ • Ϋϥελ্Ͱಈ͘ιϑτ΢ΣΞͷಋೖ

    • όʔδϣϯΞοϓͳͲͷϝϯςφϯε ։ൃͱར༻ऀ͕ҟͳΔ

14. NKE • ΫϥελͷߏஙɺόʔδϣϯΞοϓ • Ϋϥελ؅ཧϚχϑΣετͷద༻ • Data Planeͷ௥Ճɺ࡟আ • AnsibleΛ༻͍ͨϓϩϏδϣχϯά

    Ϋϥελ؅ཧΛίʔυԽ͠CLIΠϯλʔϑΣʔεʹͨ͠΋ͷ

15. NKE ઃఆϑΝΠϧɺൿಗ৘ใετΞʹج͖ͮɺ ΫϥελΛߏஙɺӡ༻ VM VM VM NKE tenant- con fi

    g.toml Hashicorp Vault conta iner conta iner conta iner

16. NKE • Golang • Hashicorp Vault • Consul • Packer

    ओཁίϯϙʔωϯτ

17. Kubernetesͷόʔδϣϯ؅ཧ

18. Kubernetesͷόʔδϣϯ؅ཧ • NKEͷϒϥϯν͝ͱʹόʔδϣϯ؅ཧ • trunk: ։ൃ༻ϒϥϯν • 1.20,1.21 ϦϦʔεϒϥϯν

19. Kubernetesͷόʔδϣϯ؅ཧ trunk 1.20 1.21 Unit Test E2E Test Unit Test

    E2E Test Unit Test E2E Test merge merge

20. Kubernetesͷόʔδϣϯ؅ཧ • CIΛར༻ͨ͠ςετΛύεͨ͠৔߹͸ɺ։ൃ༻Ϋϥελɺࣾ಺πʔϧ༻Ϋ ϥελͷόʔδϣϯΞοϓίϚϯυΛ࣮ߦ • ֤Ϋϥελͷ؅ཧऀ͕όʔδϣϯΞοϓίϚϯυΛ࣮ߦ • ΫϥελʹΑͬͯ͸2ܥ࣋ͭΑ͏ʹͯ͠ɺόʔδϣϯΞοϓ࡞ۀͳͲͷ
 μ΢ϯλΠϜΛආ͚Δ޻෉Λ͍ͯ͠Δ

21. Kubernetesͷόʔδϣϯ؅ཧ • Control Plane,Data Planeͱ΋ʹPodΛ௥͍ग़ͭͭ͠ɺ
 ϩʔϦϯάΞοϓσʔτ • Control PlaneɺEtcdʹ͍ͭͯ͸1୆ೖΕସ͑͝ͱʹϔϧενΣοΫΛ
 ͍Εͯμ΢ϯλΠϜΛආ͚͍ͯΔ

22. Kubernetesͷӡ༻؅ཧ

23. Kubernetesͷӡ༻؅ཧ • ؂ࢹ • ηΩϡϦςΟ؂ࠪ • CI/CD • ϩά؅ཧ

24. Kubernetesͷ؂ࢹ Prometheus Alert Manager Grafana mackerel-agent ࣌ܥྻσʔλͷอଘ ڞ௨ϧʔϧʹै͍ɺSlack௨஌ PrometheusͷσʔλͷϏδϡΞϥΠθʔγϣϯ Prometheus+AlertManagerͷ؂ࢹ

25. KubernetesͷηΩϡϦςΟ؂ࠪ • Wazuh • Falco • GateKeeper

26. Wazuh https://atmarkit.itmedia.co.jp/ait/articles/1902/18/news012.html OSͷઃఆ؂ࠪ ෆਖ਼ΞΫηεݕ஌ ੬ऑੑ؂ࠪ

27. Falco ίϯςφͷৼΔ෣͍؂ࠪɾݕ஌

28. Gatekeeper Admission ControllerͰಈ࡞͢Δ ϚχϑΣετͷ؂ࠪͳͲ Ұॹʹ΍ͬͯ͘Δਓɺೖࣾͯ͘͠Εʙʙʙʙ

29. ࣗಈApply ؂ࢹɺηΩϡϦςΟϙϦγʔ͸Ұ੪഑෍ tag cluster A cluster B cluster C apply

30. CI/CD • ςετɺίϯςφϏϧυɺ੬ऑੑεΩϟϯ͸Github ActionsͷSelf Hosted Runner্Ͱ࣮ߦ • ίϯςφΠϝʔδͷεΩϟϯΤϯδϯ͸trivyΛར༻ • CD͸ArgoCD

    + argocd-image-updaterΛར༻

31. ϩά؅ཧ Kafkaʹू໿ͯ͠ɺ༻్ʹԠͯ͡SaaS΁

32. ͜͜·Ͱ࿩ͨ͜͠ͱ • NKEίϚϯυͷ։ൃʹΑͬͯΫϥελͷߏங΍ϝϯςφϯεΛࣗಈԽͯ͠ ͍Δ • ؂ࢹ΍ηΩϡϦςΟ؂ࠪʹ͍ͭͯ͸NKEͰϕʔεͱͳΔ΋ͷΛఏڙ • όʔδϣϯΞοϓʹ͍ͭͯ͸E2EͰಈ࡞Λ୲อͭͭ͠ɺ։ൃ༻ΫϥελͰ ໰୊͕ͳ͍͜ͱΛ֬ೝͯ͠ɺద༻͍ͯ͠Δ

33. Hardwayͩͬͨ͜ͱ

34. 1.12.7

35. [࠶ܝ]KubernetesΫϥελ ٕज़ج൫νʔϜ Embedded SRE • NKEίϚϯυͷ։ൃ • ϓϦηοτϚχϑΣετͷߋ৽ • Ϋϥελ্Ͱಈ͘ιϑτ΢ΣΞͷಋೖ

    • όʔδϣϯΞοϓͳͲͷϝϯςφϯε ։ൃͱར༻ऀ͕ҟͳΔ

36. όʔδϣϯΞοϓͷಈػ͕௿͍͜ͱ͕͋Δ • Ϋϥελͷ༻్ • ୲౰ऀ͕ଟ๩ • Kubernetesɺ͍͍ͩͨݹͯ͘΋ಈ͘ • όʔδϣϯΞοϓʹର͢Δ৺ཧোน

37. όʔδϣϯ؅ཧࣗಈԽ͍ͨ͠ NKE Manifests Cluster A NKE Manifests Cluster B NKE

    Manifests Cluster C NKE Cluster A Cluster B Cluster C manifestͷఆٛʹج͍ͮͯࣗಈͰऩଋͯ͠΄͍͠

38. ͋Δ೔ಥવͷ ”error: You must be logged in to the server

    (Unauthorized)”

39. Կ͕ى͖͔ͨ kube-apiserver Service Account token ServiceAccountͷར༻͍ͯ͠ΔτʔΫϯ͕ࣦޮͯ͠ೝূΤϥʔ

40. ͳͥى͖͔ͨ • Kubernetes ͷ SAτʔΫϯ͸༗ޮظݶ͕Forever • ϖύϘͷKubernetesͷSAτʔΫϯͷ伴͸ࣗಈͰϩʔςʔγϣϯ͍ͯ͠Δ

41. Կ͕ى͖͔ͨ kube-apiserver Service Account token ূ໌ॻɺ伴ͷߋ৽ɺ഑෍͸HashicorpVaultͰࣗಈԽ kube-controller- manager τʔΫϯͷ෷͍ग़͠ τʔΫϯͷݕূ

    Hashicorp Vault Cert Key ূ໌ॻͱ伴ͷࣗಈ഑෍

42. Կ͕ى͖͔ͨ kube-apiserver Service Account token ূ໌ॻɺ伴ͷߋ৽ɺ഑෍͸HashicorpVaultͰࣗಈԽ kube-controller- manager τʔΫϯͷ෷͍ग़͠ τʔΫϯͷݕূ

    Hashicorp Vault Cert Key ূ໌ॻͱ伴ͷࣗಈ഑෍ Ӭٱอଘʂʂʂ

43. Կ͕ى͖͔ͨ kube-apiserver Service Account token ূ໌ॻɺ伴ͷߋ৽ɺ഑෍͸Hashicorp VaultͰࣗಈԽ kube-controller- manager τʔΫϯͷ෷͍ग़͠

    τʔΫϯͷݕূ Hashicorp Vault Cert Key ূ໌ॻͱ伴ͷࣗಈ഑෍ 伴͕ߋ৽͞ΕΔ͜ͱͰ ݕূ͕Ͱ͖ͳ͘ͳΔ

44. ରॲʂѹ౗త࢑ఆରॲʂʂʂ ಈ͍͍ͯΔϙου͸ಈ͖ଓ͚Δ͕ɺϦεέδϡʔϧ͕Ͱ͖ͳ͍ͷͰ ·ͣ͸ShellͰରॲ ͜ͷ͋ͱɺূ໌ॻͷঢ়گΛ؂ࢹͯ͠ஔ͖׵͑ΔϓϩηεΛಈ͔͍ͯ͠·͢

45. ࠷ޙͷॴײ • ΦϯϓϨKubernetes΍ΔͳΒ؅ཧιϑτ΢ΣΞΛ։ൃͨ͠΄͏͕౷߹తʹ؅ཧͰ͖ΔͷͰ
 ࠷ऴίετ͸མͱͤΔͱࢥ͏ • Kubernetesͦͷ΋ͷ͸ͱͯ΋ྑ͘Ͱ͖͍ͯͯɺKubernetesࣗମͷԿ͔Λ౿Ή͜ͱ͸ͦΜͳʹͳ͍ • ࠓ೔঺հ͍ͯ͠ͳ͍ωοτϫʔΫपΓͷΧʔωϧνϡʔχϯάͳͲɺඞཁͳέʔε΋͋ͬͨͷ Ͱɺͦ͏͍͏ྖҬΛݟΕΔਓ͕͍ͳ͍ͱݫ͍͠ͱ͸ࢥ͏ •

    େମͷϢʔεέʔε͸VM + DockerͰࣄ଍ΓΔͷͰɺ΄ΜͱʹͦΕKubernetes͍Δͷʁͱ͍͏ έʔε͸݁ߏ͋Δͱࢥ͏ • ͜Ε·Ͱͷιϑτ΢ΣΞʹՃ͑ͯɺKubernetesͱ͍͏ϨΠϠʔ͕ೖΔ͜ͱͰτϥϒϧγϡʔτ ΍؅ཧ͸େมʹͳΔ

46. ͓͠·͍ ࠷৽ͷ࠾༻৘ใΛνΣοΫˠ !QC@SFDSVJU