CloudNativeDays Tokyo 2021にて登壇した資料です。
Is Kubernetes On-premisesHardway?ʙ݁ࠗɺͦΕରͰ͋Δʙ
View Slide
ࢁԼ!QZBNB(.0ϖύϘٕज़ج൫νʔϜγχΞɾϓϦϯγύϧUFOTOBQPODPNQZBNBGVOTUOTKQ
45/4-JOVY/444FSWFSTUOTKQ
45/4
https://github.com/pyama86/github-replacer
ϗεςΟϯάࣄۀ &$ࢧԉࣄۀ ϋϯυϝΠυɾͦͷଞࣄۀ
Is Kubernetes On-premisesHardway?
ϚωʔδυαʔϏεͷϝϦοτ• Control Plane/Data Planeͷཧ• όʔδϣϯΞοϓͷ༰қ͞• Ϋϥυࣄۀऀ͕ఏڙ͢ΔͦͷଞͷϚωʔδυαʔϏεͱͷ࿈ܞͷ༰қ͞• ແݶεέʔϦϯά(If you have much money)
ΦϯϓϨϛεͷϝϦοτ• ͯ͢ΛΒͶͳΒͳ͍͕ނʹࣗ༝• ਓ݅අΛআ͘ϥϯχϯάίετͷ҆͞
ࠓ͢͜ͱ• ϖύϘͷKubernetesΫϥελͷ֓ཁ• Hardwayͩͬͨ͜ͱ• ࠓޙΓ͍ͨ͜ͱ
KubernetesΫϥελ• OpenStack (Nyah)• Nyah Kubernetes Engine(NKE)
KubernetesΫϥελنײ• ࡐ͝ͱʹΫϥελΛ͓ͯ͠Γɺ23Ϋϥελ(ൃද࣌)• ࡐʹΑͬͯNKE / GKE / EKSͰͷϋΠϒϦουΫϥυͰར༻• AWS Direct ConnectͰઐ༻ઢར༻
KubernetesΫϥελٕज़ج൫νʔϜEmbedded SRE• NKEίϚϯυͷ։ൃ• ϓϦηοτϚχϑΣετͷߋ৽• Ϋϥελ্Ͱಈ͘ιϑτΣΞͷಋೖ• όʔδϣϯΞοϓͳͲͷϝϯςφϯε։ൃͱར༻ऀ͕ҟͳΔ
NKE• ΫϥελͷߏஙɺόʔδϣϯΞοϓ• ΫϥελཧϚχϑΣετͷద༻• Data PlaneͷՃɺআ• AnsibleΛ༻͍ͨϓϩϏδϣχϯάΫϥελཧΛίʔυԽ͠CLIΠϯλʔϑΣʔεʹͨ͠ͷ
NKEઃఆϑΝΠϧɺൿಗใετΞʹج͖ͮɺΫϥελΛߏஙɺӡ༻VMVMVMNKEtenant-config.tomlHashicorpVaultcontainercontainercontainer
NKE• Golang• Hashicorp Vault• Consul• Packerओཁίϯϙʔωϯτ
Kubernetesͷόʔδϣϯཧ
Kubernetesͷόʔδϣϯཧ• NKEͷϒϥϯν͝ͱʹόʔδϣϯཧ• trunk: ։ൃ༻ϒϥϯν• 1.20,1.21 ϦϦʔεϒϥϯν
Kubernetesͷόʔδϣϯཧtrunk1.201.21Unit TestE2E TestUnit TestE2E TestUnit TestE2E Testmergemerge
Kubernetesͷόʔδϣϯཧ• CIΛར༻ͨ͠ςετΛύεͨ͠߹ɺ։ൃ༻Ϋϥελɺࣾπʔϧ༻ΫϥελͷόʔδϣϯΞοϓίϚϯυΛ࣮ߦ• ֤Ϋϥελͷཧऀ͕όʔδϣϯΞοϓίϚϯυΛ࣮ߦ• ΫϥελʹΑͬͯ2ܥ࣋ͭΑ͏ʹͯ͠ɺόʔδϣϯΞοϓ࡞ۀͳͲͷ μϯλΠϜΛආ͚ΔΛ͍ͯ͠Δ
Kubernetesͷόʔδϣϯཧ• Control Plane,Data PlaneͱʹPodΛ͍ग़ͭͭ͠ɺ ϩʔϦϯάΞοϓσʔτ• Control PlaneɺEtcdʹ͍ͭͯ1ೖΕସ͑͝ͱʹϔϧενΣοΫΛ ͍ΕͯμϯλΠϜΛආ͚͍ͯΔ
Kubernetesͷӡ༻ཧ
Kubernetesͷӡ༻ཧ• ࢹ• ηΩϡϦςΟࠪ• CI/CD• ϩάཧ
KubernetesͷࢹPrometheusAlert ManagerGrafanamackerel-agent࣌ܥྻσʔλͷอଘڞ௨ϧʔϧʹै͍ɺSlack௨PrometheusͷσʔλͷϏδϡΞϥΠθʔγϣϯPrometheus+AlertManagerͷࢹ
KubernetesͷηΩϡϦςΟࠪ• Wazuh• Falco• GateKeeper
Wazuhhttps://atmarkit.itmedia.co.jp/ait/articles/1902/18/news012.htmlOSͷઃఆࠪෆਖ਼ΞΫηεݕ੬ऑੑࠪ
FalcoίϯςφͷৼΔ͍ࠪɾݕ
GatekeeperAdmission ControllerͰಈ࡞͢ΔϚχϑΣετͷࠪͳͲҰॹʹͬͯ͘Δਓɺೖࣾͯ͘͠Εʙʙʙʙ
ࣗಈApplyࢹɺηΩϡϦςΟϙϦγʔҰ੪tagcluster Acluster Bcluster Capply
CI/CD• ςετɺίϯςφϏϧυɺ੬ऑੑεΩϟϯGithub ActionsͷSelf HostedRunner্Ͱ࣮ߦ• ίϯςφΠϝʔδͷεΩϟϯΤϯδϯtrivyΛར༻• CDArgoCD + argocd-image-updaterΛར༻
ϩάཧKafkaʹूͯ͠ɺ༻్ʹԠͯ͡SaaS
͜͜·Ͱͨ͜͠ͱ• NKEίϚϯυͷ։ൃʹΑͬͯΫϥελͷߏஙϝϯςφϯεΛࣗಈԽ͍ͯ͠Δ• ࢹηΩϡϦςΟࠪʹ͍ͭͯNKEͰϕʔεͱͳΔͷΛఏڙ• όʔδϣϯΞοϓʹ͍ͭͯE2EͰಈ࡞Λ୲อͭͭ͠ɺ։ൃ༻ΫϥελͰ͕ͳ͍͜ͱΛ֬ೝͯ͠ɺద༻͍ͯ͠Δ
Hardwayͩͬͨ͜ͱ
1.12.7
[࠶ܝ]KubernetesΫϥελٕज़ج൫νʔϜEmbedded SRE• NKEίϚϯυͷ։ൃ• ϓϦηοτϚχϑΣετͷߋ৽• Ϋϥελ্Ͱಈ͘ιϑτΣΞͷಋೖ• όʔδϣϯΞοϓͳͲͷϝϯςφϯε։ൃͱར༻ऀ͕ҟͳΔ
όʔδϣϯΞοϓͷಈػ͕͍͜ͱ͕͋Δ• Ϋϥελͷ༻్• ୲ऀ͕ଟ• Kubernetesɺ͍͍ͩͨݹͯ͘ಈ͘• όʔδϣϯΞοϓʹର͢Δ৺ཧোน
όʔδϣϯཧࣗಈԽ͍ͨ͠NKE ManifestsCluster ANKE ManifestsCluster BNKE ManifestsCluster CNKECluster ACluster BCluster Cmanifestͷఆٛʹج͍ͮͯࣗಈͰऩଋͯ͠΄͍͠
͋Δಥવͷ”error: You must be logged in to theserver (Unauthorized)”
Կ͕ى͖͔ͨkube-apiserverServiceAccounttokenServiceAccountͷར༻͍ͯ͠ΔτʔΫϯ͕ࣦޮͯ͠ೝূΤϥʔ
ͳͥى͖͔ͨ• Kubernetes ͷ SAτʔΫϯ༗ޮظݶ͕Forever• ϖύϘͷKubernetesͷSAτʔΫϯͷ伴ࣗಈͰϩʔςʔγϣϯ͍ͯ͠Δ
Կ͕ى͖͔ͨkube-apiserverServiceAccounttokenূ໌ॻɺ伴ͷߋ৽ɺHashicorpVaultͰࣗಈԽkube-controller-managerτʔΫϯͷ͍ग़͠ τʔΫϯͷݕূHashicorpVaultCert Keyূ໌ॻͱ伴ͷࣗಈ
Կ͕ى͖͔ͨkube-apiserverServiceAccounttokenূ໌ॻɺ伴ͷߋ৽ɺHashicorpVaultͰࣗಈԽkube-controller-managerτʔΫϯͷ͍ग़͠ τʔΫϯͷݕূHashicorpVaultCert Keyূ໌ॻͱ伴ͷࣗಈӬٱอଘʂʂʂ
Կ͕ى͖͔ͨkube-apiserverServiceAccounttokenূ໌ॻɺ伴ͷߋ৽ɺHashicorp VaultͰࣗಈԽkube-controller-managerτʔΫϯͷ͍ग़͠ τʔΫϯͷݕূHashicorpVaultCert Keyূ໌ॻͱ伴ͷࣗಈ伴͕ߋ৽͞ΕΔ͜ͱͰݕূ͕Ͱ͖ͳ͘ͳΔ
ରॲʂѹతఆରॲʂʂʂಈ͍͍ͯΔϙουಈ͖ଓ͚Δ͕ɺϦεέδϡʔϧ͕Ͱ͖ͳ͍ͷͰ·ͣShellͰରॲ͜ͷ͋ͱɺূ໌ॻͷঢ়گΛࢹͯ͠ஔ͖͑ΔϓϩηεΛಈ͔͍ͯ͠·͢
࠷ޙͷॴײ• ΦϯϓϨKubernetesΔͳΒཧιϑτΣΞΛ։ൃͨ͠΄͏͕౷߹తʹཧͰ͖ΔͷͰ ࠷ऴίετམͱͤΔͱࢥ͏• Kubernetesͦͷͷͱͯྑ͘Ͱ͖͍ͯͯɺKubernetesࣗମͷԿ͔Λ౿Ή͜ͱͦΜͳʹͳ͍• ࠓհ͍ͯ͠ͳ͍ωοτϫʔΫपΓͷΧʔωϧνϡʔχϯάͳͲɺඞཁͳέʔε͋ͬͨͷͰɺͦ͏͍͏ྖҬΛݟΕΔਓ͕͍ͳ͍ͱݫ͍͠ͱࢥ͏• େମͷϢʔεέʔεVM + DockerͰࣄΓΔͷͰɺ΄ΜͱʹͦΕKubernetes͍Δͷʁͱ͍͏έʔε݁ߏ͋Δͱࢥ͏• ͜Ε·ͰͷιϑτΣΞʹՃ͑ͯɺKubernetesͱ͍͏ϨΠϠʔ͕ೖΔ͜ͱͰτϥϒϧγϡʔτཧେมʹͳΔ
͓͠·͍࠷৽ͷ࠾༻ใΛνΣοΫˠ !QC@SFDSVJU