我が家のKubernetesクラスタ

Slide 1

Slide 1 text

K8s@home #2 (2022/12/14) Fumihiro Ito զ͕ՈͷKubernetesΫϥελ

Slide 2

Slide 2 text

ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster Who? • ҏ౻ ࢙ߒ • https://github.com/f110 • Z Lab Corporation • ϚωʔδυK8sαʔϏεͷ։ൃ • લ৬Ͱ͸εϚʔτϑΥϯ޲͚ήʔϜͷαʔόʔαΠυͷӡ༻ɾ։ൃ

Slide 3

Slide 3 text

ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster

Slide 4

Slide 4 text

ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster ςΩετ൛ͷϦϯΫΛৗʹࠨԼʹද͓͖ࣔͯ͠·͢

Slide 5

Slide 5 text

ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster ͳͥࣗ୐ΫϥελΛߏஙͯ͠͠·ͬͨͷ͔

Slide 6

Slide 6 text

ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster ͳͥࣗ୐ΫϥελΛߏஙͯ͠͠·ͬͨͷ͔ • K8sΤίγεςϜΛࢼͨ͠ΓϚχϑΣετΛॻ࣌͘ʹ࢖͑Δ؀ڥ͕ཉ͍͠ • minikube / kind ࣌୅ʢ2017ʙ2019೥ʣ • ࠷ॳ͸MacBook ProͰ • Ϧιʔε͕଍Γͳ͘ͳͬͨͷͰRyzen Threadripper 2950X΁ʢ16C32Tʣ • ׂͱແ஡Λͯ͠΋ಈ͘ɺҰ෦͚ͩʹো֐͕ൃੜ͢Δͱ͍͏͜ͱ͕ͳ͍ • ͦ͏ͩNUCͰΫϥελΛߏங͠Α͏

Slide 7

Slide 7 text

ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster ίϯηϓτ • ͦͦ͜͜K8sʹ৐ͤΔ • ͋Δఔ౓͸Ϋϥελ্Ͱಈ͔͍ͨ͠ • ͚ͲؤுΓ͗͢ͳ͍ • ͿͬյΕ্౳ • յΕʢյ͠ʣͯम෮͢Δܦݧ͔ΒֶͿ΋ͷ͸ଟ͍ • ͪΐͬͱେࣄͳ΋ͷΛಈ͔͓ͯ͘͜͠ͱͰम෮ͷϞνϕʔγϣϯʹ͢Δ • ຊ౰ʹࠔΔ΋ͷ͸֎΁ग़͢

Slide 8

Slide 8 text

ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster جຊํ਑ • ϛχϚϜ͔Β࢝Ίͯঃʑʹ֦େ͍ͯ͘͠ • ͦΕͳΓʹ҆ఆͯ͠ಈ͘΋ͷΛ࢖͏ • ॳΊͯߏங͢ΔͷͰ߈Ίͨ͜ͱ͸͠ͳ͍ • ߏஙɾӡ༻͕ΰʔϧͰ͸ͳ͍ • ͦͷ্Ͱಈ͔͢΋ͷΛࢼͨ͠Γ։ൃͨ͠Γ͍ͨ͠

Slide 9

Slide 9 text

ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster $ kubectl get node whale1 -o jsonpath="{.metadata.creationTimestamp}" 2019-10-21T02:59:56Z 2019೥ʹ2୆ߏ੒Ͱ࢝ΊͨΫϥελ ࠓͰ͸ίϯτϩʔϧϓϨʔϯx3 ϫʔΧʔx4ͷߏ੒ʹ $ kubectl get node NAME STATUS ROLES AGE VERSION rudder1 Ready control-plane 146d v1.24.3 rudder2 Ready control-plane 2y118d v1.24.3 rudder3 Ready control-plane 2y28d v1.24.3 whale1 Ready 3y50d v1.24.3 whale2 Ready 3y50d v1.24.3 whale3 Ready 344d v1.24.3 whale4 Ready 2y216d v1.24.3

Slide 10

Slide 10 text

ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster ϋʔυ΢ΣΞߏ੒ UniFi Dream Machine Pro UniFi Switch 16 Control Plane Raspberry Pi 4 mem: 4GB disk: 64GB Raspberry Pi 4 mem: 4GB disk: 64GB Raspberry Pi 4 mem: 4GB disk: 64GB Worker Intel NUC Core i3-8109U Mem: 16GB Intel NUC Core i3-10110U Mem: 16GB Minisforum Ryzen 7 2700U Mem: 16GB Intel NUC Core i3-8109U Mem: 16GB

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

UniFi Dream Machine Pro • podmanͰίϯςφΛಈ͔ͤͨΓศརͳ΍ͭ • FRRoutingΛಈ͔ͤ͹ܦ࿏Λަ׵͢Δ͜ͱ΋Մ

Slide 13

Slide 13 text

UniFi Switch 16

Slide 14

Slide 14 text

ϊʔυ

Slide 15

Slide 15 text

ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster ιϑτ΢ΣΞߏ੒ʢίϯτϩʔϧϓϨʔϯʣ rudder1 keepalived haproxy kube-apiserver etcd rudder2 keepalived haproxy kube-apiserver etcd rudder3 keepalived haproxy kube-apiserver etcd kube-controller-manager kube-scheduler kube-controller-manager kube-scheduler kube-controller-manager kube-scheduler • ΦʔιυοΫεͳ haproxy / keepalived Λ࢖ͬͨ HA ߏ੒ • controller-manager ͱ scheduler ΋ಉډ • apiserver ΁͸ϩʔΧϧΞΫηε

Slide 16

Slide 16 text

ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster K8sج൫ ߏ੒πʔϧ kubeadm CNI calico ετϨʔδ Rook / Ceph ϩʔυόϥϯα MetalLB ίϯςφϨδετϦ Harbor γʔΫϨοτ؅ཧ Vault

Slide 17

Slide 17 text

ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster ϞχλϦϯά • Prometheus / Grafana / Grafana Loki ͷ Α͋͘Δߏ੒ • alertmanager ͸ Slack ʹ௨஌͢ΔͷΈ • Loki ʹϩάΛอଘ͍ͯ͠Δ͕΄ͱΜͲ ݟͯͳ͍ prometheus loki promtail node-exporter promtail alertmanager prometheus loki grafana grafana Slack kube-state-metrics

Slide 18

Slide 18 text

ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster GitOps • GitHub ʹϓϥΠϕʔτϦϙδτϦͱͯ͠ϚχϑΣετΛஔ͍͍ͯΔ • γʔΫϨοτ΋ϚχϑΣετͱͯ͠ೖΕ໊͍ͯͨ࢒ • git-crypt ͰϑΝΠϧࣗମ͸҉߸Խ͍͕ͯͨ͠ສ͕Ұͷ҉߸Խ๨ΕΛߟྀ͠ ͯϓϥΠϕʔτͰӡ༻ • ArgoCD ͰΫϥελʹద༻͢Δ

Slide 19

Slide 19 text

ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster ϊʔυͷηοτΞοϓ • OSͷΠϯετʔϧ͸σΟεϓϨΠͱΩʔϘʔυΛͭͳ͍ͰUSBϒʔτ • OSΠϯετʔϧͷࣗಈԽ͸࢖͏ػձ͕গͳ͍ͷͰ͋͑ͯߦΘͣ • ϓϩϏδϣχϯά͸itamae • ϓϩϏδϣχϯάπʔϧ͸͍ͣΕࣗ࡞͍ͨ͠

Slide 20

Slide 20 text

ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster Ξοϓσʔτ • γϯϓϧʹkubeadmͰ • Ξοϓσʔτ࣌ʹϊʔυ͸Ұ౓ࢭΊΔ • 1୆ͣͭҰ࣌తʹ֎ͯ͠૟আΛ͢Δ • NUC͸ϗίϦ͕ཷ·Δͱ͙͢ʹྫྷ٫ੑೳ͕མͪΔ • OSͷΞοϓσʔτ͕͋Ε͹͜ͷλΠϛϯά

Slide 21

Slide 21 text

ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster ೔ʑͷӡ༻ • جຊతʹGitOps • γʔΫϨοτ͸Vaultʹอଘͯ͠argocd-vault-plugin • ࣗಈͰ൓ө͢Δ΋ͷͱࠩ෼Λ֬ೝͯ͠खಈͰ൓ө͢Δ΋ͷ͕͋Δ • PrometheusͰ؂ࢹͯ͠Ξϥʔτ͸Slack΁ • Ξϥʔτϧʔϧ͸গͳΊ • Ξϥʔτ͕ͳ͍ͱؾ͕෇͔ͳ͍Α͏ͳ΍ͭͷΈΛઃఆ͍ͯ͠Δ • ྫ͑͹ϊʔυͷCPU࢖༻཰ͳͲ͸؂ࢹ͠ͳ͍ • CPUϑΝϯͷԻ͕ฉ͑͜ΔͷͰҟৗ͸෼͔Δ

Slide 22

Slide 22 text

ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster ࣗ࡞Ingress controller • Zero Trust ProxyΛࣗ࡞͍ͯ͠ΔͷͰগ֦͠ுͯ͠Ingress controllerͱͯ͠΋ৼ Δ෣͏Α͏ʹͨ͠ • ৄ͘͠͸ https://bit.ly/f110-ingress-controller • IngressϦιʔε͸ೝূෆཁͰΞΫηεɺಠࣗϦιʔε͸ೝূͳͲ༷ʑͳઃఆ ͕Ͱ͖Δ • GitHubͷWebhookΛઃఆ͢ΔػೳͳͲ΋͋Δ

Slide 23

Slide 23 text

ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster ࣗ࡞Ingress controller cert-manager cert-manager heimdallr operator proxy etcd api 自作コンポーネント OSSコンポーネント

Slide 24

Slide 24 text

ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster ࣗ࡞Controller܊ • ӡ༻ͷॿ͚ʹͳΔίϯτϩʔϥΛ͍͔ͭࣗ͘࡞ • MinIOͷόέοτΛ؅ཧ͢ΔίϯτϩʔϥͳͲ • ίϯτϩʔϥ͸·ͱΊͯϚωʔδϟͱͯ͠ҰͭͷPodͰಈ࡞͍ͤͯ͞Δ

Slide 25

Slide 25 text

ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster όοΫΞοϓ • γϯϓϧʹetcdͷεφοϓγϣοτΛऔಘ͢ΔͷΈ • อଘઌ͸Google Cloud Storage • ετϨʔδͷσʔλ͸ఘΊΔ • όοΫΞοϓͷϓϩάϥϜ͸ࣗ࡞ • CronJobͰ3ִ࣌ؒؒͰ࣮ߦ • όοΫΞοϓϑΝΠϧ͸Ұिؒ෼ͷΈอ͍࣋ͯ͠Δ

Slide 26

Slide 26 text

ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster ΞϓϦέʔγϣϯ • ࣗ࡞Zero Trust Proxy • ࣗ࡞CI • Notion༻ࣗ࡞πʔϧ • ίʔυݕࡧπʔϧ • Թ౓ܭ༻ΤΫεϙʔλʔ

Slide 27

Slide 27 text

ςΩετ൛͸ 👉 https://bit.ly/f110-homecluster ࠓޙ • L3εΠονΛಋೖͯ͠CLOSߏ੒ʹ͍ͨ͠ • ্Ґϧʔλ͕SPOFͳͷͰωοτϫʔΫͷ৑௕ԽΛ͍ͨ͠ • ετϨʔδͷ෼཭ • ҆ఆՔಇͰ͖ͦ͏