Is Kubernetes On-premises Hardway?

CloudNativeDays Tokyo 2021にて登壇した資料です。

1. Is Kubernetes On-premises Hardway? ʙ݁ࠗɺͦΕ͸ର࿩Ͱ͋Δʙ
2. ࢁԼ
   ࿨඙
   
   !QZBNB
   (.0ϖύϘ
   ٕज़ج൫νʔϜ
   
   
   γχΞɾϓϦϯγύϧ
   
   
   
   UFOTOBQPODPN
   
   
   
   QZBNBGVO
   
   
   
   TUOTKQ
3. 45/4
   -JOVY
   /44
   4FSWFS
   TUOTKQ
4. 45/4
5. https://github.com/pyama86/github-replacer
6. ϗεςΟϯάࣄۀ &$ࢧԉࣄۀ ϋϯυϝΠυɾͦͷଞࣄۀ
7. Is Kubernetes On-premises Hardway?
8. ϚωʔδυαʔϏεͷϝϦοτ • Control Plane/Data Planeͷ؅ཧ • όʔδϣϯΞοϓͷ༰қ͞ • Ϋϥ΢υࣄۀऀ͕ఏڙ͢ΔͦͷଞͷϚωʔδυαʔϏεͱͷ࿈ܞͷ༰қ͞ • ແݶεέʔϦϯά(If you have much money)
9. ΦϯϓϨϛεͷϝϦοτ • ͢΂ͯΛ΍ΒͶ͹ͳΒͳ͍͕ނʹࣗ༝ • ਓ݅අΛআ͘ϥϯχϯάίετͷ҆͞
10. ࠓ೔࿩͢͜ͱ • ϖύϘͷKubernetesΫϥελͷ֓ཁ • Hardwayͩͬͨ͜ͱ • ࠓޙ΍Γ͍ͨ͜ͱ
11. KubernetesΫϥελ • OpenStack (Nyah) • Nyah Kubernetes Engine(NKE)
12. KubernetesΫϥελ ن໛ײ • ঎ࡐ͝ͱʹΫϥελΛ෼཭͓ͯ͠Γɺ23Ϋϥελ(ൃද࣌఺) • ঎ࡐʹΑͬͯ͸NKE / GKE / EKSͰͷϋΠϒϦουΫϥ΢υͰར༻ • AWS Direct ConnectͰઐ༻ઢར༻
13. KubernetesΫϥελ ٕज़ج൫νʔϜ Embedded SRE • NKEίϚϯυͷ։ൃ • ϓϦηοτϚχϑΣετͷߋ৽ • Ϋϥελ্Ͱಈ͘ιϑτ΢ΣΞͷಋೖ • όʔδϣϯΞοϓͳͲͷϝϯςφϯε ։ൃͱར༻ऀ͕ҟͳΔ
14. NKE • ΫϥελͷߏஙɺόʔδϣϯΞοϓ • Ϋϥελ؅ཧϚχϑΣετͷద༻ • Data Planeͷ௥Ճɺ࡟আ • AnsibleΛ༻͍ͨϓϩϏδϣχϯά Ϋϥελ؅ཧΛίʔυԽ͠CLIΠϯλʔϑΣʔεʹͨ͠΋ͷ
15. NKE ઃఆϑΝΠϧɺൿಗ৘ใετΞʹج͖ͮɺ ΫϥελΛߏஙɺӡ༻ VM VM VM NKE tenant- con fi g.toml Hashicorp Vault conta iner conta iner conta iner
16. NKE • Golang • Hashicorp Vault • Consul • Packer ओཁίϯϙʔωϯτ
17. Kubernetesͷόʔδϣϯ؅ཧ
18. Kubernetesͷόʔδϣϯ؅ཧ • NKEͷϒϥϯν͝ͱʹόʔδϣϯ؅ཧ • trunk: ։ൃ༻ϒϥϯν • 1.20,1.21 ϦϦʔεϒϥϯν
19. Kubernetesͷόʔδϣϯ؅ཧ trunk 1.20 1.21 Unit Test E2E Test Unit Test E2E Test Unit Test E2E Test merge merge
20. Kubernetesͷόʔδϣϯ؅ཧ • CIΛར༻ͨ͠ςετΛύεͨ͠৔߹͸ɺ։ൃ༻Ϋϥελɺࣾ಺πʔϧ༻Ϋ ϥελͷόʔδϣϯΞοϓίϚϯυΛ࣮ߦ • ֤Ϋϥελͷ؅ཧऀ͕όʔδϣϯΞοϓίϚϯυΛ࣮ߦ • ΫϥελʹΑͬͯ͸2ܥ࣋ͭΑ͏ʹͯ͠ɺόʔδϣϯΞοϓ࡞ۀͳͲͷ
 μ΢ϯλΠϜΛආ͚Δ޻෉Λ͍ͯ͠Δ
21. Kubernetesͷόʔδϣϯ؅ཧ • Control Plane,Data Planeͱ΋ʹPodΛ௥͍ग़ͭͭ͠ɺ
 ϩʔϦϯάΞοϓσʔτ • Control PlaneɺEtcdʹ͍ͭͯ͸1୆ೖΕସ͑͝ͱʹϔϧενΣοΫΛ
 ͍Εͯμ΢ϯλΠϜΛආ͚͍ͯΔ
22. Kubernetesͷӡ༻؅ཧ
23. Kubernetesͷӡ༻؅ཧ • ؂ࢹ • ηΩϡϦςΟ؂ࠪ • CI/CD • ϩά؅ཧ
24. Kubernetesͷ؂ࢹ Prometheus Alert Manager Grafana mackerel-agent ࣌ܥྻσʔλͷอଘ ڞ௨ϧʔϧʹै͍ɺSlack௨஌ PrometheusͷσʔλͷϏδϡΞϥΠθʔγϣϯ Prometheus+AlertManagerͷ؂ࢹ
25. KubernetesͷηΩϡϦςΟ؂ࠪ • Wazuh • Falco • GateKeeper
26. Wazuh https://atmarkit.itmedia.co.jp/ait/articles/1902/18/news012.html OSͷઃఆ؂ࠪ ෆਖ਼ΞΫηεݕ஌ ੬ऑੑ؂ࠪ
27. Falco ίϯςφͷৼΔ෣͍؂ࠪɾݕ஌
28. Gatekeeper Admission ControllerͰಈ࡞͢Δ ϚχϑΣετͷ؂ࠪͳͲ Ұॹʹ΍ͬͯ͘Δਓɺೖࣾͯ͘͠Εʙʙʙʙ
29. ࣗಈApply ؂ࢹɺηΩϡϦςΟϙϦγʔ͸Ұ੪഑෍ tag cluster A cluster B cluster C apply
30. CI/CD • ςετɺίϯςφϏϧυɺ੬ऑੑεΩϟϯ͸Github ActionsͷSelf Hosted Runner্Ͱ࣮ߦ • ίϯςφΠϝʔδͷεΩϟϯΤϯδϯ͸trivyΛར༻ • CD͸ArgoCD + argocd-image-updaterΛར༻
31. ϩά؅ཧ Kafkaʹू໿ͯ͠ɺ༻్ʹԠͯ͡SaaS΁
32. ͜͜·Ͱ࿩ͨ͜͠ͱ • NKEίϚϯυͷ։ൃʹΑͬͯΫϥελͷߏங΍ϝϯςφϯεΛࣗಈԽͯ͠ ͍Δ • ؂ࢹ΍ηΩϡϦςΟ؂ࠪʹ͍ͭͯ͸NKEͰϕʔεͱͳΔ΋ͷΛఏڙ • όʔδϣϯΞοϓʹ͍ͭͯ͸E2EͰಈ࡞Λ୲อͭͭ͠ɺ։ൃ༻ΫϥελͰ ໰୊͕ͳ͍͜ͱΛ֬ೝͯ͠ɺద༻͍ͯ͠Δ
33. Hardwayͩͬͨ͜ͱ
34. 1.12.7
35. [࠶ܝ]KubernetesΫϥελ ٕज़ج൫νʔϜ Embedded SRE • NKEίϚϯυͷ։ൃ • ϓϦηοτϚχϑΣετͷߋ৽ • Ϋϥελ্Ͱಈ͘ιϑτ΢ΣΞͷಋೖ • όʔδϣϯΞοϓͳͲͷϝϯςφϯε ։ൃͱར༻ऀ͕ҟͳΔ
36. όʔδϣϯΞοϓͷಈػ͕௿͍͜ͱ͕͋Δ • Ϋϥελͷ༻్ • ୲౰ऀ͕ଟ๩ • Kubernetesɺ͍͍ͩͨݹͯ͘΋ಈ͘ • όʔδϣϯΞοϓʹର͢Δ৺ཧোน
37. όʔδϣϯ؅ཧࣗಈԽ͍ͨ͠ NKE Manifests Cluster A NKE Manifests Cluster B NKE Manifests Cluster C NKE Cluster A Cluster B Cluster C manifestͷఆٛʹج͍ͮͯࣗಈͰऩଋͯ͠΄͍͠
38. ͋Δ೔ಥવͷ ”error: You must be logged in to the server (Unauthorized)”
39. Կ͕ى͖͔ͨ kube-apiserver Service Account token ServiceAccountͷར༻͍ͯ͠ΔτʔΫϯ͕ࣦޮͯ͠ೝূΤϥʔ
40. ͳͥى͖͔ͨ • Kubernetes ͷ SAτʔΫϯ͸༗ޮظݶ͕Forever • ϖύϘͷKubernetesͷSAτʔΫϯͷ伴͸ࣗಈͰϩʔςʔγϣϯ͍ͯ͠Δ
41. Կ͕ى͖͔ͨ kube-apiserver Service Account token ূ໌ॻɺ伴ͷߋ৽ɺ഑෍͸HashicorpVaultͰࣗಈԽ kube-controller- manager τʔΫϯͷ෷͍ग़͠ τʔΫϯͷݕূ Hashicorp Vault Cert Key ূ໌ॻͱ伴ͷࣗಈ഑෍
42. Կ͕ى͖͔ͨ kube-apiserver Service Account token ূ໌ॻɺ伴ͷߋ৽ɺ഑෍͸HashicorpVaultͰࣗಈԽ kube-controller- manager τʔΫϯͷ෷͍ग़͠ τʔΫϯͷݕূ Hashicorp Vault Cert Key ূ໌ॻͱ伴ͷࣗಈ഑෍ Ӭٱอଘʂʂʂ
43. Կ͕ى͖͔ͨ kube-apiserver Service Account token ূ໌ॻɺ伴ͷߋ৽ɺ഑෍͸Hashicorp VaultͰࣗಈԽ kube-controller- manager τʔΫϯͷ෷͍ग़͠ τʔΫϯͷݕূ Hashicorp Vault Cert Key ূ໌ॻͱ伴ͷࣗಈ഑෍ 伴͕ߋ৽͞ΕΔ͜ͱͰ ݕূ͕Ͱ͖ͳ͘ͳΔ
44. ରॲʂѹ౗త࢑ఆରॲʂʂʂ ಈ͍͍ͯΔϙου͸ಈ͖ଓ͚Δ͕ɺϦεέδϡʔϧ͕Ͱ͖ͳ͍ͷͰ ·ͣ͸ShellͰରॲ ͜ͷ͋ͱɺূ໌ॻͷঢ়گΛ؂ࢹͯ͠ஔ͖׵͑ΔϓϩηεΛಈ͔͍ͯ͠·͢
45. ࠷ޙͷॴײ • ΦϯϓϨKubernetes΍ΔͳΒ؅ཧιϑτ΢ΣΞΛ։ൃͨ͠΄͏͕౷߹తʹ؅ཧͰ͖ΔͷͰ
 ࠷ऴίετ͸མͱͤΔͱࢥ͏ • Kubernetesͦͷ΋ͷ͸ͱͯ΋ྑ͘Ͱ͖͍ͯͯɺKubernetesࣗମͷԿ͔Λ౿Ή͜ͱ͸ͦΜͳʹͳ͍ • ࠓ೔঺հ͍ͯ͠ͳ͍ωοτϫʔΫपΓͷΧʔωϧνϡʔχϯάͳͲɺඞཁͳέʔε΋͋ͬͨͷ Ͱɺͦ͏͍͏ྖҬΛݟΕΔਓ͕͍ͳ͍ͱݫ͍͠ͱ͸ࢥ͏ • େମͷϢʔεέʔε͸VM + DockerͰࣄ଍ΓΔͷͰɺ΄ΜͱʹͦΕKubernetes͍Δͷʁͱ͍͏ έʔε͸݁ߏ͋Δͱࢥ͏ • ͜Ε·Ͱͷιϑτ΢ΣΞʹՃ͑ͯɺKubernetesͱ͍͏ϨΠϠʔ͕ೖΔ͜ͱͰτϥϒϧγϡʔτ ΍؅ཧ͸େมʹͳΔ
46. ͓͠·͍ ࠷৽ͷ࠾༻৘ใΛνΣοΫˠ !QC@SFDSVJU