我が家のKubernetesクラスタ

Transcript

1. K8s@home #2 (2022/12/14) Fumihiro Ito զ͕ՈͷKubernetesΫϥελ
2. ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster Who? • ҏ౻ ࢙ߒ • https://github.com/f110 • Z Lab Corporation • ϚωʔδυK8sαʔϏεͷ։ൃ • લ৬Ͱ͸εϚʔτϑΥϯ޲͚ήʔϜͷαʔόʔαΠυͷӡ༻ɾ։ൃ
3. ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster
4. ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster ςΩετ൛ͷϦϯΫΛৗʹࠨԼʹද͓͖ࣔͯ͠·͢
5. ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster ͳͥࣗ୐ΫϥελΛߏஙͯ͠͠·ͬͨͷ͔
6. ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster ͳͥࣗ୐ΫϥελΛߏஙͯ͠͠·ͬͨͷ͔ • K8sΤίγεςϜΛࢼͨ͠ΓϚχϑΣετΛॻ࣌͘ʹ࢖͑Δ؀ڥ͕ཉ͍͠ • minikube / kind ࣌୅ʢ2017ʙ2019೥ʣ • ࠷ॳ͸MacBook ProͰ • Ϧιʔε͕଍Γͳ͘ͳͬͨͷͰRyzen Threadripper 2950X΁ʢ16C32Tʣ • ׂͱແ஡Λͯ͠΋ಈ͘ɺҰ෦͚ͩʹো֐͕ൃੜ͢Δͱ͍͏͜ͱ͕ͳ͍ • ͦ͏ͩNUCͰΫϥελΛߏங͠Α͏
7. ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster ίϯηϓτ • ͦͦ͜͜K8sʹ৐ͤΔ • ͋Δఔ౓͸Ϋϥελ্Ͱಈ͔͍ͨ͠ • ͚ͲؤுΓ͗͢ͳ͍ • ͿͬյΕ্౳ • յΕʢյ͠ʣͯम෮͢Δܦݧ͔ΒֶͿ΋ͷ͸ଟ͍ • ͪΐͬͱେࣄͳ΋ͷΛಈ͔͓ͯ͘͜͠ͱͰम෮ͷϞνϕʔγϣϯʹ͢Δ • ຊ౰ʹࠔΔ΋ͷ͸֎΁ग़͢
8. ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster جຊํ਑ • ϛχϚϜ͔Β࢝Ίͯঃʑʹ֦େ͍ͯ͘͠ • ͦΕͳΓʹ҆ఆͯ͠ಈ͘΋ͷΛ࢖͏ • ॳΊͯߏங͢ΔͷͰ߈Ίͨ͜ͱ͸͠ͳ͍ • ߏஙɾӡ༻͕ΰʔϧͰ͸ͳ͍ • ͦͷ্Ͱಈ͔͢΋ͷΛࢼͨ͠Γ։ൃͨ͠Γ͍ͨ͠
9. ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster $ kubectl get node whale1 -o jsonpath="{.metadata.creationTimestamp}" 2019-10-21T02:59:56Z 2019೥ʹ2୆ߏ੒Ͱ࢝ΊͨΫϥελ ࠓͰ͸ίϯτϩʔϧϓϨʔϯx3 ϫʔΧʔx4ͷߏ੒ʹ $ kubectl get node NAME STATUS ROLES AGE VERSION rudder1 Ready control-plane 146d v1.24.3 rudder2 Ready control-plane 2y118d v1.24.3 rudder3 Ready control-plane 2y28d v1.24.3 whale1 Ready <none> 3y50d v1.24.3 whale2 Ready <none> 3y50d v1.24.3 whale3 Ready <none> 344d v1.24.3 whale4 Ready <none> 2y216d v1.24.3
10. ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster ϋʔυ΢ΣΞߏ੒ UniFi Dream Machine Pro UniFi Switch 16 Control Plane Raspberry Pi 4 mem: 4GB disk: 64GB Raspberry Pi 4 mem: 4GB disk: 64GB Raspberry Pi 4 mem: 4GB disk: 64GB Worker Intel NUC Core i3-8109U Mem: 16GB Intel NUC Core i3-10110U Mem: 16GB Minisforum Ryzen 7 2700U Mem: 16GB Intel NUC Core i3-8109U Mem: 16GB
12. UniFi Dream Machine Pro • podmanͰίϯςφΛಈ͔ͤͨΓศརͳ΍ͭ • FRRoutingΛಈ͔ͤ͹ܦ࿏Λަ׵͢Δ͜ͱ΋Մ
13. UniFi Switch 16
14. ϊʔυ
15. ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster ιϑτ΢ΣΞߏ੒ʢίϯτϩʔϧϓϨʔϯʣ rudder1 keepalived haproxy kube-apiserver etcd rudder2 keepalived haproxy kube-apiserver etcd rudder3 keepalived haproxy kube-apiserver etcd kube-controller-manager kube-scheduler kube-controller-manager kube-scheduler kube-controller-manager kube-scheduler • ΦʔιυοΫεͳ haproxy / keepalived Λ࢖ͬͨ HA ߏ੒ • controller-manager ͱ scheduler ΋ಉډ • apiserver ΁͸ϩʔΧϧΞΫηε
16. ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster K8sج൫ ߏ੒πʔϧ kubeadm CNI calico ετϨʔδ Rook / Ceph ϩʔυόϥϯα MetalLB ίϯςφϨδετϦ Harbor γʔΫϨοτ؅ཧ Vault
17. ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster ϞχλϦϯά • Prometheus / Grafana / Grafana Loki ͷ Α͋͘Δߏ੒ • alertmanager ͸ Slack ʹ௨஌͢ΔͷΈ • Loki ʹϩάΛอଘ͍ͯ͠Δ͕΄ͱΜͲ ݟͯͳ͍ prometheus loki promtail node-exporter promtail alertmanager prometheus loki grafana grafana Slack kube-state-metrics
18. ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster GitOps • GitHub ʹϓϥΠϕʔτϦϙδτϦͱͯ͠ϚχϑΣετΛஔ͍͍ͯΔ • γʔΫϨοτ΋ϚχϑΣετͱͯ͠ೖΕ໊͍ͯͨ࢒ • git-crypt ͰϑΝΠϧࣗମ͸҉߸Խ͍͕ͯͨ͠ສ͕Ұͷ҉߸Խ๨ΕΛߟྀ͠ ͯϓϥΠϕʔτͰӡ༻ • ArgoCD ͰΫϥελʹద༻͢Δ
19. ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster ϊʔυͷηοτΞοϓ • OSͷΠϯετʔϧ͸σΟεϓϨΠͱΩʔϘʔυΛͭͳ͍ͰUSBϒʔτ • OSΠϯετʔϧͷࣗಈԽ͸࢖͏ػձ͕গͳ͍ͷͰ͋͑ͯߦΘͣ • ϓϩϏδϣχϯά͸itamae • ϓϩϏδϣχϯάπʔϧ͸͍ͣΕࣗ࡞͍ͨ͠
20. ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster Ξοϓσʔτ • γϯϓϧʹkubeadmͰ • Ξοϓσʔτ࣌ʹϊʔυ͸Ұ౓ࢭΊΔ • 1୆ͣͭҰ࣌తʹ֎ͯ͠૟আΛ͢Δ • NUC͸ϗίϦ͕ཷ·Δͱ͙͢ʹྫྷ٫ੑೳ͕མͪΔ • OSͷΞοϓσʔτ͕͋Ε͹͜ͷλΠϛϯά
21. ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster ೔ʑͷӡ༻ • جຊతʹGitOps • γʔΫϨοτ͸Vaultʹอଘͯ͠argocd-vault-plugin • ࣗಈͰ൓ө͢Δ΋ͷͱࠩ෼Λ֬ೝͯ͠खಈͰ൓ө͢Δ΋ͷ͕͋Δ • PrometheusͰ؂ࢹͯ͠Ξϥʔτ͸Slack΁ • Ξϥʔτϧʔϧ͸গͳΊ • Ξϥʔτ͕ͳ͍ͱؾ͕෇͔ͳ͍Α͏ͳ΍ͭͷΈΛઃఆ͍ͯ͠Δ • ྫ͑͹ϊʔυͷCPU࢖༻཰ͳͲ͸؂ࢹ͠ͳ͍ • CPUϑΝϯͷԻ͕ฉ͑͜ΔͷͰҟৗ͸෼͔Δ
22. ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster ࣗ࡞Ingress controller • Zero Trust ProxyΛࣗ࡞͍ͯ͠ΔͷͰগ֦͠ுͯ͠Ingress controllerͱͯ͠΋ৼ Δ෣͏Α͏ʹͨ͠ • ৄ͘͠͸ https://bit.ly/f110-ingress-controller • IngressϦιʔε͸ೝূෆཁͰΞΫηεɺಠࣗϦιʔε͸ೝূͳͲ༷ʑͳઃఆ ͕Ͱ͖Δ • GitHubͷWebhookΛઃఆ͢ΔػೳͳͲ΋͋Δ
23. ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster ࣗ࡞Ingress controller cert-manager cert-manager heimdallr operator proxy etcd api 自作コンポーネント OSSコンポーネント
24. ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster ࣗ࡞Controller܊ • ӡ༻ͷॿ͚ʹͳΔίϯτϩʔϥΛ͍͔ͭࣗ͘࡞ • MinIOͷόέοτΛ؅ཧ͢ΔίϯτϩʔϥͳͲ • ίϯτϩʔϥ͸·ͱΊͯϚωʔδϟͱͯ͠ҰͭͷPodͰಈ࡞͍ͤͯ͞Δ
25. ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster όοΫΞοϓ • γϯϓϧʹetcdͷεφοϓγϣοτΛऔಘ͢ΔͷΈ • อଘઌ͸Google Cloud Storage • ετϨʔδͷσʔλ͸ఘΊΔ • όοΫΞοϓͷϓϩάϥϜ͸ࣗ࡞ • CronJobͰ3ִ࣌ؒؒͰ࣮ߦ • όοΫΞοϓϑΝΠϧ͸Ұिؒ෼ͷΈอ͍࣋ͯ͠Δ
26. ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster ΞϓϦέʔγϣϯ • ࣗ࡞Zero Trust Proxy • ࣗ࡞CI • Notion༻ࣗ࡞πʔϧ • ίʔυݕࡧπʔϧ • Թ౓ܭ༻ΤΫεϙʔλʔ
27. ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster ࠓޙ • L3εΠονΛಋೖͯ͠CLOSߏ੒ʹ͍ͨ͠ • ্Ґϧʔλ͕SPOFͳͷͰωοτϫʔΫͷ৑௕ԽΛ͍ͨ͠ • ετϨʔδͷ෼཭ • ҆ఆՔಇͰ͖ͦ͏