Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Is Kubernetes On-premises Hardway?
Search
Kazuhiko Yamashita
November 05, 2021
Technology
2
500
Is Kubernetes On-premises Hardway?
CloudNativeDays Tokyo 2021にて登壇した資料です。
Kazuhiko Yamashita
November 05, 2021
Tweet
Share
More Decks by Kazuhiko Yamashita
See All by Kazuhiko Yamashita
ペパボOpenTelemetry革命
pyama86
2
1.5k
Site Reliability Engineering for GMO
pyama86
9
1.1k
PHPアプリケーションのスケーラビリティと 信頼性を革新する nginx+ngx_mrubyとGoの融合
pyama86
2
330
ペパボOpenTelemetry革命
pyama86
0
750
ChatGPTの活用を体現し、 組織に浸透する技術
pyama86
1
960
リリースフラグと プログレッシブデリバリーを活用した 安全なWEBアプリケーションデプロイ
pyama86
2
200
PHPerのためのOpenTelemetry入門
pyama86
1
240
生成AIで仕事を もっとおもしろくする事例と詳解
pyama86
0
150
Goで実装された高速な 仮想待合室サーバの実装と詳解
pyama86
15
7.2k
Other Decks in Technology
See All in Technology
未知の原因によるDB負荷の上昇とその対応プロセス
tshimizu470
0
310
エレガントパズル エンジニアのマネジメントという難問にあなたはどう立ち向かうのか / Elegant Puzzle
iwashi86
13
2.4k
Kubernetesで作るIaaS基盤/KubeVirt Deep Dive
oracle4engineer
PRO
11
2.1k
Custom logging with slog Making Logging Fun Again!
masumomo
3
360
ポスター & デモと総括 / Posters & Demonstrations, and Conclusions
ks91
PRO
0
100
A perfect combo: content that sells from engineers who write
holymay
0
100
Oracle Base Database Service 技術詳細
oracle4engineer
PRO
5
39k
20240530_洲崎 Amazon Connectの概要と活用パターン
suzakiyoshito
0
210
OCI Functionsについて
dbtec
0
110
生成AIの活用方法 - 2024年現在、 結局どのように使えばいいのだろうか? / How to use Generative AI in 2024
nttcom
3
720
Cleanup handling in Go / Go Conference 2024
k1low
5
2k
自動テストは何の役に立つのか そして役に立たないのか
undefined_name
2
930
Featured
See All Featured
Into the Great Unknown - MozCon
thekraken
16
1.2k
Building Adaptive Systems
keathley
33
2k
4 Signs Your Business is Dying
shpigford
176
21k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
13
1.1k
Teambox: Starting and Learning
jrom
129
8.5k
Become a Pro
speakerdeck
PRO
13
4.7k
The Brand Is Dead. Long Live the Brand.
mthomps
51
33k
Debugging Ruby Performance
tmm1
70
11k
How To Stay Up To Date on Web Technology
chriscoyier
783
250k
ParisWeb 2013: Learning to Love: Crash Course in Emotional UX Design
dotmariusz
104
6.7k
Building a Scalable Design System with Sketch
lauravandoore
457
32k
Intergalactic Javascript Robots from Outer Space
tanoku
266
26k
Transcript
Is Kubernetes On-premises Hardway? ʙ݁ࠗɺͦΕରͰ͋Δʙ
ࢁԼ!QZBNB (.0ϖύϘٕज़ج൫νʔϜ γχΞɾϓϦϯγύϧ UFOTOBQPODPN QZBNBGVO TUOTKQ
45/4 -JOVY/444FSWFS TUOTKQ
45/4
https://github.com/pyama86/github-replacer
ϗεςΟϯάࣄۀ &$ࢧԉࣄۀ ϋϯυϝΠυɾͦͷଞࣄۀ
Is Kubernetes On-premises Hardway?
ϚωʔδυαʔϏεͷϝϦοτ • Control Plane/Data Planeͷཧ • όʔδϣϯΞοϓͷ༰қ͞ • Ϋϥυࣄۀऀ͕ఏڙ͢ΔͦͷଞͷϚωʔδυαʔϏεͱͷ࿈ܞͷ༰қ͞ •
ແݶεέʔϦϯά(If you have much money)
ΦϯϓϨϛεͷϝϦοτ • ͯ͢ΛΒͶͳΒͳ͍͕ނʹࣗ༝ • ਓ݅අΛআ͘ϥϯχϯάίετͷ҆͞
ࠓ͢͜ͱ • ϖύϘͷKubernetesΫϥελͷ֓ཁ • Hardwayͩͬͨ͜ͱ • ࠓޙΓ͍ͨ͜ͱ
KubernetesΫϥελ • OpenStack (Nyah) • Nyah Kubernetes Engine(NKE)
KubernetesΫϥελ نײ • ࡐ͝ͱʹΫϥελΛ͓ͯ͠Γɺ23Ϋϥελ(ൃද࣌) • ࡐʹΑͬͯNKE / GKE / EKSͰͷϋΠϒϦουΫϥυͰར༻
• AWS Direct ConnectͰઐ༻ઢར༻
KubernetesΫϥελ ٕज़ج൫νʔϜ Embedded SRE • NKEίϚϯυͷ։ൃ • ϓϦηοτϚχϑΣετͷߋ৽ • Ϋϥελ্Ͱಈ͘ιϑτΣΞͷಋೖ
• όʔδϣϯΞοϓͳͲͷϝϯςφϯε ։ൃͱར༻ऀ͕ҟͳΔ
NKE • ΫϥελͷߏஙɺόʔδϣϯΞοϓ • ΫϥελཧϚχϑΣετͷద༻ • Data PlaneͷՃɺআ • AnsibleΛ༻͍ͨϓϩϏδϣχϯά
ΫϥελཧΛίʔυԽ͠CLIΠϯλʔϑΣʔεʹͨ͠ͷ
NKE ઃఆϑΝΠϧɺൿಗใετΞʹج͖ͮɺ ΫϥελΛߏஙɺӡ༻ VM VM VM NKE tenant- con fi
g.toml Hashicorp Vault conta iner conta iner conta iner
NKE • Golang • Hashicorp Vault • Consul • Packer
ओཁίϯϙʔωϯτ
Kubernetesͷόʔδϣϯཧ
Kubernetesͷόʔδϣϯཧ • NKEͷϒϥϯν͝ͱʹόʔδϣϯཧ • trunk: ։ൃ༻ϒϥϯν • 1.20,1.21 ϦϦʔεϒϥϯν
Kubernetesͷόʔδϣϯཧ trunk 1.20 1.21 Unit Test E2E Test Unit Test
E2E Test Unit Test E2E Test merge merge
Kubernetesͷόʔδϣϯཧ • CIΛར༻ͨ͠ςετΛύεͨ͠߹ɺ։ൃ༻Ϋϥελɺࣾπʔϧ༻Ϋ ϥελͷόʔδϣϯΞοϓίϚϯυΛ࣮ߦ • ֤Ϋϥελͷཧऀ͕όʔδϣϯΞοϓίϚϯυΛ࣮ߦ • ΫϥελʹΑͬͯ2ܥ࣋ͭΑ͏ʹͯ͠ɺόʔδϣϯΞοϓ࡞ۀͳͲͷ μϯλΠϜΛආ͚ΔΛ͍ͯ͠Δ
Kubernetesͷόʔδϣϯཧ • Control Plane,Data PlaneͱʹPodΛ͍ग़ͭͭ͠ɺ ϩʔϦϯάΞοϓσʔτ • Control PlaneɺEtcdʹ͍ͭͯ1ೖΕସ͑͝ͱʹϔϧενΣοΫΛ ͍ΕͯμϯλΠϜΛආ͚͍ͯΔ
Kubernetesͷӡ༻ཧ
Kubernetesͷӡ༻ཧ • ࢹ • ηΩϡϦςΟࠪ • CI/CD • ϩάཧ
Kubernetesͷࢹ Prometheus Alert Manager Grafana mackerel-agent ࣌ܥྻσʔλͷอଘ ڞ௨ϧʔϧʹै͍ɺSlack௨ PrometheusͷσʔλͷϏδϡΞϥΠθʔγϣϯ Prometheus+AlertManagerͷࢹ
KubernetesͷηΩϡϦςΟࠪ • Wazuh • Falco • GateKeeper
Wazuh https://atmarkit.itmedia.co.jp/ait/articles/1902/18/news012.html OSͷઃఆࠪ ෆਖ਼ΞΫηεݕ ੬ऑੑࠪ
Falco ίϯςφͷৼΔ͍ࠪɾݕ
Gatekeeper Admission ControllerͰಈ࡞͢Δ ϚχϑΣετͷࠪͳͲ Ұॹʹͬͯ͘Δਓɺೖࣾͯ͘͠Εʙʙʙʙ
ࣗಈApply ࢹɺηΩϡϦςΟϙϦγʔҰ੪ tag cluster A cluster B cluster C apply
CI/CD • ςετɺίϯςφϏϧυɺ੬ऑੑεΩϟϯGithub ActionsͷSelf Hosted Runner্Ͱ࣮ߦ • ίϯςφΠϝʔδͷεΩϟϯΤϯδϯtrivyΛར༻ • CDArgoCD
+ argocd-image-updaterΛར༻
ϩάཧ Kafkaʹूͯ͠ɺ༻్ʹԠͯ͡SaaS
͜͜·Ͱͨ͜͠ͱ • NKEίϚϯυͷ։ൃʹΑͬͯΫϥελͷߏஙϝϯςφϯεΛࣗಈԽͯ͠ ͍Δ • ࢹηΩϡϦςΟࠪʹ͍ͭͯNKEͰϕʔεͱͳΔͷΛఏڙ • όʔδϣϯΞοϓʹ͍ͭͯE2EͰಈ࡞Λ୲อͭͭ͠ɺ։ൃ༻ΫϥελͰ ͕ͳ͍͜ͱΛ֬ೝͯ͠ɺద༻͍ͯ͠Δ
Hardwayͩͬͨ͜ͱ
1.12.7
[࠶ܝ]KubernetesΫϥελ ٕज़ج൫νʔϜ Embedded SRE • NKEίϚϯυͷ։ൃ • ϓϦηοτϚχϑΣετͷߋ৽ • Ϋϥελ্Ͱಈ͘ιϑτΣΞͷಋೖ
• όʔδϣϯΞοϓͳͲͷϝϯςφϯε ։ൃͱར༻ऀ͕ҟͳΔ
όʔδϣϯΞοϓͷಈػ͕͍͜ͱ͕͋Δ • Ϋϥελͷ༻్ • ୲ऀ͕ଟ • Kubernetesɺ͍͍ͩͨݹͯ͘ಈ͘ • όʔδϣϯΞοϓʹର͢Δ৺ཧোน
όʔδϣϯཧࣗಈԽ͍ͨ͠ NKE Manifests Cluster A NKE Manifests Cluster B NKE
Manifests Cluster C NKE Cluster A Cluster B Cluster C manifestͷఆٛʹج͍ͮͯࣗಈͰऩଋͯ͠΄͍͠
͋Δಥવͷ ”error: You must be logged in to the server
(Unauthorized)”
Կ͕ى͖͔ͨ kube-apiserver Service Account token ServiceAccountͷར༻͍ͯ͠ΔτʔΫϯ͕ࣦޮͯ͠ೝূΤϥʔ
ͳͥى͖͔ͨ • Kubernetes ͷ SAτʔΫϯ༗ޮظݶ͕Forever • ϖύϘͷKubernetesͷSAτʔΫϯͷ伴ࣗಈͰϩʔςʔγϣϯ͍ͯ͠Δ
Կ͕ى͖͔ͨ kube-apiserver Service Account token ূ໌ॻɺ伴ͷߋ৽ɺHashicorpVaultͰࣗಈԽ kube-controller- manager τʔΫϯͷ͍ग़͠ τʔΫϯͷݕূ
Hashicorp Vault Cert Key ূ໌ॻͱ伴ͷࣗಈ
Կ͕ى͖͔ͨ kube-apiserver Service Account token ূ໌ॻɺ伴ͷߋ৽ɺHashicorpVaultͰࣗಈԽ kube-controller- manager τʔΫϯͷ͍ग़͠ τʔΫϯͷݕূ
Hashicorp Vault Cert Key ূ໌ॻͱ伴ͷࣗಈ Ӭٱอଘʂʂʂ
Կ͕ى͖͔ͨ kube-apiserver Service Account token ূ໌ॻɺ伴ͷߋ৽ɺHashicorp VaultͰࣗಈԽ kube-controller- manager τʔΫϯͷ͍ग़͠
τʔΫϯͷݕূ Hashicorp Vault Cert Key ূ໌ॻͱ伴ͷࣗಈ 伴͕ߋ৽͞ΕΔ͜ͱͰ ݕূ͕Ͱ͖ͳ͘ͳΔ
ରॲʂѹతఆରॲʂʂʂ ಈ͍͍ͯΔϙουಈ͖ଓ͚Δ͕ɺϦεέδϡʔϧ͕Ͱ͖ͳ͍ͷͰ ·ͣShellͰରॲ ͜ͷ͋ͱɺূ໌ॻͷঢ়گΛࢹͯ͠ஔ͖͑ΔϓϩηεΛಈ͔͍ͯ͠·͢
࠷ޙͷॴײ • ΦϯϓϨKubernetesΔͳΒཧιϑτΣΞΛ։ൃͨ͠΄͏͕౷߹తʹཧͰ͖ΔͷͰ ࠷ऴίετམͱͤΔͱࢥ͏ • Kubernetesͦͷͷͱͯྑ͘Ͱ͖͍ͯͯɺKubernetesࣗମͷԿ͔Λ౿Ή͜ͱͦΜͳʹͳ͍ • ࠓհ͍ͯ͠ͳ͍ωοτϫʔΫपΓͷΧʔωϧνϡʔχϯάͳͲɺඞཁͳέʔε͋ͬͨͷ Ͱɺͦ͏͍͏ྖҬΛݟΕΔਓ͕͍ͳ͍ͱݫ͍͠ͱࢥ͏ •
େମͷϢʔεέʔεVM + DockerͰࣄΓΔͷͰɺ΄ΜͱʹͦΕKubernetes͍Δͷʁͱ͍͏ έʔε݁ߏ͋Δͱࢥ͏ • ͜Ε·ͰͷιϑτΣΞʹՃ͑ͯɺKubernetesͱ͍͏ϨΠϠʔ͕ೖΔ͜ͱͰτϥϒϧγϡʔτ ཧେมʹͳΔ
͓͠·͍ ࠷৽ͷ࠾༻ใΛνΣοΫˠ !QC@SFDSVJU