Is Kubernetes On-premises Hardway?

CloudNativeDays Tokyo 2021にて登壇した資料です。

Transcript

1. Is Kubernetes On-premises
   Hardway?
   ʙ݁ࠗɺͦΕ͸ର࿩Ͱ͋Δʙ
   

   View Slide

2. ࢁԼ࿨඙!QZBNB
   (.0ϖύϘٕज़ج൫νʔϜ
   γχΞɾϓϦϯγύϧ
   UFOTOBQPODPN
   QZBNBGVO
   TUOTKQ
   

   View Slide

3. 45/4
   -JOVY/444FSWFS
   TUOTKQ
   

   View Slide

4. 45/4
   

   View Slide

5. https://github.com/pyama86/github-replacer
   

   View Slide

6. ϗεςΟϯάࣄۀ &$ࢧԉࣄۀ ϋϯυϝΠυɾͦͷଞࣄۀ
   

   View Slide

7. Is Kubernetes On-premises
   Hardway?
   

   View Slide

8. ϚωʔδυαʔϏεͷϝϦοτ
   • Control Plane/Data Planeͷ؅ཧ
   
   • όʔδϣϯΞοϓͷ༰қ͞
   
   • Ϋϥ΢υࣄۀऀ͕ఏڙ͢ΔͦͷଞͷϚωʔδυαʔϏεͱͷ࿈ܞͷ༰қ͞
   
   • ແݶεέʔϦϯά(If you have much money)
   

   View Slide

9. ΦϯϓϨϛεͷϝϦοτ
   • ͢΂ͯΛ΍ΒͶ͹ͳΒͳ͍͕ނʹࣗ༝
   
   • ਓ݅අΛআ͘ϥϯχϯάίετͷ҆͞
   

   View Slide

10. ࠓ೔࿩͢͜ͱ
    • ϖύϘͷKubernetesΫϥελͷ֓ཁ
    
    • Hardwayͩͬͨ͜ͱ
    
    • ࠓޙ΍Γ͍ͨ͜ͱ
    

    View Slide

11. KubernetesΫϥελ
    • OpenStack (Nyah)
    
    • Nyah Kubernetes Engine(NKE)
    

    View Slide

12. KubernetesΫϥελ
    ن໛ײ
    • ঎ࡐ͝ͱʹΫϥελΛ෼཭͓ͯ͠Γɺ23Ϋϥελ(ൃද࣌఺)
    
    • ঎ࡐʹΑͬͯ͸NKE / GKE / EKSͰͷϋΠϒϦουΫϥ΢υͰར༻
    
    • AWS Direct ConnectͰઐ༻ઢར༻
    

    View Slide

13. KubernetesΫϥελ
    ٕज़ج൫νʔϜ
    Embedded SRE
    • NKEίϚϯυͷ։ൃ
    
    • ϓϦηοτϚχϑΣετͷߋ৽
    • Ϋϥελ্Ͱಈ͘ιϑτ΢ΣΞͷಋೖ
    
    • όʔδϣϯΞοϓͳͲͷϝϯςφϯε
    ։ൃͱར༻ऀ͕ҟͳΔ
    

    View Slide

14. NKE
    • ΫϥελͷߏஙɺόʔδϣϯΞοϓ
    
    • Ϋϥελ؅ཧϚχϑΣετͷద༻
    
    • Data Planeͷ௥Ճɺ࡟আ
    
    • AnsibleΛ༻͍ͨϓϩϏδϣχϯά
    Ϋϥελ؅ཧΛίʔυԽ͠CLIΠϯλʔϑΣʔεʹͨ͠΋ͷ
    

    View Slide

15. NKE
    ઃఆϑΝΠϧɺൿಗ৘ใετΞʹج͖ͮɺ
    
    
    ΫϥελΛߏஙɺӡ༻
    VM
    VM
    VM
    NKE
    tenant-
    con
    fi
    g.toml
    Hashicorp
    
    
    Vault
    conta
    iner
    conta
    iner
    conta
    iner
    

    View Slide

16. NKE
    • Golang
    
    • Hashicorp Vault
    
    • Consul
    
    • Packer
    ओཁίϯϙʔωϯτ
    

    View Slide

17. Kubernetesͷόʔδϣϯ؅ཧ
    

    View Slide

18. Kubernetesͷόʔδϣϯ؅ཧ
    • NKEͷϒϥϯν͝ͱʹόʔδϣϯ؅ཧ
    
    • trunk: ։ൃ༻ϒϥϯν
    
    • 1.20,1.21 ϦϦʔεϒϥϯν
    

    View Slide

19. Kubernetesͷόʔδϣϯ؅ཧ
    trunk
    1.20
    1.21
    Unit Test
    
    
    E2E Test
    Unit Test
    
    
    E2E Test
    Unit Test
    
    
    E2E Test
    merge
    merge
    

    View Slide

20. Kubernetesͷόʔδϣϯ؅ཧ
    • CIΛར༻ͨ͠ςετΛύεͨ͠৔߹͸ɺ։ൃ༻Ϋϥελɺࣾ಺πʔϧ༻Ϋ
    ϥελͷόʔδϣϯΞοϓίϚϯυΛ࣮ߦ
    
    • ֤Ϋϥελͷ؅ཧऀ͕όʔδϣϯΞοϓίϚϯυΛ࣮ߦ
    
    • ΫϥελʹΑͬͯ͸2ܥ࣋ͭΑ͏ʹͯ͠ɺόʔδϣϯΞοϓ࡞ۀͳͲͷ

    μ΢ϯλΠϜΛආ͚Δ޻෉Λ͍ͯ͠Δ
    

    View Slide

21. Kubernetesͷόʔδϣϯ؅ཧ
    • Control Plane,Data Planeͱ΋ʹPodΛ௥͍ग़ͭͭ͠ɺ

    ϩʔϦϯάΞοϓσʔτ
    
    • Control PlaneɺEtcdʹ͍ͭͯ͸1୆ೖΕସ͑͝ͱʹϔϧενΣοΫΛ

    ͍Εͯμ΢ϯλΠϜΛආ͚͍ͯΔ
    

    View Slide

22. Kubernetesͷӡ༻؅ཧ
    

    View Slide

23. Kubernetesͷӡ༻؅ཧ
    • ؂ࢹ
    
    • ηΩϡϦςΟ؂ࠪ
    
    • CI/CD
    
    • ϩά؅ཧ
    

    View Slide

24. Kubernetesͷ؂ࢹ
    Prometheus
    
    Alert Manager
    
    Grafana
    
    mackerel-agent
    ࣌ܥྻσʔλͷอଘ
    ڞ௨ϧʔϧʹै͍ɺSlack௨஌
    PrometheusͷσʔλͷϏδϡΞϥΠθʔγϣϯ
    Prometheus+AlertManagerͷ؂ࢹ
    

    View Slide

25. KubernetesͷηΩϡϦςΟ؂ࠪ
    • Wazuh
    
    • Falco
    
    • GateKeeper
    

    View Slide

26. Wazuh
    https://atmarkit.itmedia.co.jp/ait/articles/1902/18/news012.html
    OSͷઃఆ؂ࠪ
    
    ෆਖ਼ΞΫηεݕ஌
    
    ੬ऑੑ؂ࠪ
    

    View Slide

27. Falco
    ίϯςφͷৼΔ෣͍؂ࠪɾݕ஌
    

    View Slide

28. Gatekeeper
    Admission ControllerͰಈ࡞͢Δ
    
    ϚχϑΣετͷ؂ࠪͳͲ
    Ұॹʹ΍ͬͯ͘Δਓɺೖࣾͯ͘͠Εʙʙʙʙ
    

    View Slide

29. ࣗಈApply
    ؂ࢹɺηΩϡϦςΟϙϦγʔ͸Ұ੪഑෍
    tag
    cluster A
    cluster B
    cluster C
    apply
    

    View Slide

30. CI/CD
    • ςετɺίϯςφϏϧυɺ੬ऑੑεΩϟϯ͸Github ActionsͷSelf Hosted
    Runner্Ͱ࣮ߦ
    
    • ίϯςφΠϝʔδͷεΩϟϯΤϯδϯ͸trivyΛར༻
    
    • CD͸ArgoCD + argocd-image-updaterΛར༻
    

    View Slide

31. ϩά؅ཧ
    Kafkaʹू໿ͯ͠ɺ༻్ʹԠͯ͡SaaS΁
    

    View Slide

32. ͜͜·Ͱ࿩ͨ͜͠ͱ
    • NKEίϚϯυͷ։ൃʹΑͬͯΫϥελͷߏங΍ϝϯςφϯεΛࣗಈԽͯ͠
    ͍Δ
    
    • ؂ࢹ΍ηΩϡϦςΟ؂ࠪʹ͍ͭͯ͸NKEͰϕʔεͱͳΔ΋ͷΛఏڙ
    
    • όʔδϣϯΞοϓʹ͍ͭͯ͸E2EͰಈ࡞Λ୲อͭͭ͠ɺ։ൃ༻ΫϥελͰ
    ໰୊͕ͳ͍͜ͱΛ֬ೝͯ͠ɺద༻͍ͯ͠Δ
    

    View Slide

33. Hardwayͩͬͨ͜ͱ
    

    View Slide

34. 1.12.7
    

    View Slide

35. [࠶ܝ]KubernetesΫϥελ
    ٕज़ج൫νʔϜ
    Embedded SRE
    • NKEίϚϯυͷ։ൃ
    
    • ϓϦηοτϚχϑΣετͷߋ৽
    • Ϋϥελ্Ͱಈ͘ιϑτ΢ΣΞͷಋೖ
    
    • όʔδϣϯΞοϓͳͲͷϝϯςφϯε
    ։ൃͱར༻ऀ͕ҟͳΔ
    

    View Slide

36. όʔδϣϯΞοϓͷಈػ͕௿͍͜ͱ͕͋Δ
    • Ϋϥελͷ༻్
    
    • ୲౰ऀ͕ଟ๩
    
    • Kubernetesɺ͍͍ͩͨݹͯ͘΋ಈ͘
    
    • όʔδϣϯΞοϓʹର͢Δ৺ཧোน
    

    View Slide

37. όʔδϣϯ؅ཧࣗಈԽ͍ͨ͠
    NKE Manifests
    
    
    Cluster A
    NKE Manifests
    
    
    Cluster B
    NKE Manifests
    
    
    Cluster C
    NKE
    Cluster A
    Cluster B
    Cluster C
    manifestͷఆٛʹج͍ͮͯࣗಈͰऩଋͯ͠΄͍͠
    

    View Slide

38. ͋Δ೔ಥવͷ
    
    
    ”error: You must be logged in to the
    server (Unauthorized)”
    

    View Slide

39. Կ͕ى͖͔ͨ
    kube-apiserver
    Service
    
    
    Account
    token
    ServiceAccountͷར༻͍ͯ͠ΔτʔΫϯ͕ࣦޮͯ͠ೝূΤϥʔ
    

    View Slide

40. ͳͥى͖͔ͨ
    • Kubernetes ͷ SAτʔΫϯ͸༗ޮظݶ͕Forever
    
    • ϖύϘͷKubernetesͷSAτʔΫϯͷ伴͸ࣗಈͰϩʔςʔγϣϯ͍ͯ͠Δ
    

    View Slide

41. Կ͕ى͖͔ͨ
    kube-apiserver
    Service
    
    
    Account
    token
    ূ໌ॻɺ伴ͷߋ৽ɺ഑෍͸HashicorpVaultͰࣗಈԽ
    kube-controller-
    manager
    τʔΫϯͷ෷͍ग़͠ τʔΫϯͷݕূ
    Hashicorp
    
    
    Vault
    Cert Key
    ূ໌ॻͱ伴ͷࣗಈ഑෍
    

    View Slide

42. Կ͕ى͖͔ͨ
    kube-apiserver
    Service
    
    
    Account
    token
    ূ໌ॻɺ伴ͷߋ৽ɺ഑෍͸HashicorpVaultͰࣗಈԽ
    kube-controller-
    manager
    τʔΫϯͷ෷͍ग़͠ τʔΫϯͷݕূ
    Hashicorp
    
    
    Vault
    Cert Key
    ূ໌ॻͱ伴ͷࣗಈ഑෍
    Ӭٱอଘʂʂʂ
    

    View Slide

43. Կ͕ى͖͔ͨ
    kube-apiserver
    Service
    
    
    Account
    token
    ূ໌ॻɺ伴ͷߋ৽ɺ഑෍͸Hashicorp VaultͰࣗಈԽ
    kube-controller-
    manager
    τʔΫϯͷ෷͍ग़͠ τʔΫϯͷݕূ
    Hashicorp
    
    
    Vault
    Cert Key
    ূ໌ॻͱ伴ͷࣗಈ഑෍
    伴͕ߋ৽͞ΕΔ͜ͱͰ
    
    
    ݕূ͕Ͱ͖ͳ͘ͳΔ
    

    View Slide

44. ରॲʂѹ౗త࢑ఆରॲʂʂʂ
    ಈ͍͍ͯΔϙου͸ಈ͖ଓ͚Δ͕ɺϦεέδϡʔϧ͕Ͱ͖ͳ͍ͷͰ
    ·ͣ͸ShellͰରॲ
    ͜ͷ͋ͱɺূ໌ॻͷঢ়گΛ؂ࢹͯ͠ஔ͖׵͑ΔϓϩηεΛಈ͔͍ͯ͠·͢
    

    View Slide

45. ࠷ޙͷॴײ
    • ΦϯϓϨKubernetes΍ΔͳΒ؅ཧιϑτ΢ΣΞΛ։ൃͨ͠΄͏͕౷߹తʹ؅ཧͰ͖ΔͷͰ

    ࠷ऴίετ͸མͱͤΔͱࢥ͏
    
    • Kubernetesͦͷ΋ͷ͸ͱͯ΋ྑ͘Ͱ͖͍ͯͯɺKubernetesࣗମͷԿ͔Λ౿Ή͜ͱ͸ͦΜͳʹͳ͍
    
    • ࠓ೔঺հ͍ͯ͠ͳ͍ωοτϫʔΫपΓͷΧʔωϧνϡʔχϯάͳͲɺඞཁͳέʔε΋͋ͬͨͷ
    Ͱɺͦ͏͍͏ྖҬΛݟΕΔਓ͕͍ͳ͍ͱݫ͍͠ͱ͸ࢥ͏
    
    • େମͷϢʔεέʔε͸VM + DockerͰࣄ଍ΓΔͷͰɺ΄ΜͱʹͦΕKubernetes͍Δͷʁͱ͍͏
    έʔε͸݁ߏ͋Δͱࢥ͏
    
    • ͜Ε·Ͱͷιϑτ΢ΣΞʹՃ͑ͯɺKubernetesͱ͍͏ϨΠϠʔ͕ೖΔ͜ͱͰτϥϒϧγϡʔτ
    ΍؅ཧ͸େมʹͳΔ
    

    View Slide

46. ͓͠·͍
    ࠷৽ͷ࠾༻৘ใΛνΣοΫˠ !QC@SFDSVJU
    

    View Slide