Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Is Kubernetes On-premises Hardway?

Is Kubernetes On-premises Hardway?

CloudNativeDays Tokyo 2021にて登壇した資料です。

Kazuhiko Yamashita

November 05, 2021
Tweet

More Decks by Kazuhiko Yamashita

Other Decks in Technology

Transcript

  1. Is Kubernetes On-premises
    Hardway?
    ʙ݁ࠗɺͦΕ͸ର࿩Ͱ͋Δʙ

    View Slide

  2. ࢁԼ࿨඙!QZBNB
    (.0ϖύϘٕज़ج൫νʔϜ
    γχΞɾϓϦϯγύϧ
    UFOTOBQPODPN
    QZBNBGVO
    TUOTKQ

    View Slide

  3. 45/4
    -JOVY/444FSWFS
    TUOTKQ

    View Slide

  4. 45/4

    View Slide

  5. https://github.com/pyama86/github-replacer

    View Slide

  6. ϗεςΟϯάࣄۀ &$ࢧԉࣄۀ ϋϯυϝΠυɾͦͷଞࣄۀ

    View Slide

  7. Is Kubernetes On-premises
    Hardway?

    View Slide

  8. ϚωʔδυαʔϏεͷϝϦοτ
    • Control Plane/Data Planeͷ؅ཧ

    • όʔδϣϯΞοϓͷ༰қ͞

    • Ϋϥ΢υࣄۀऀ͕ఏڙ͢ΔͦͷଞͷϚωʔδυαʔϏεͱͷ࿈ܞͷ༰қ͞

    • ແݶεέʔϦϯά(If you have much money)

    View Slide

  9. ΦϯϓϨϛεͷϝϦοτ
    • ͢΂ͯΛ΍ΒͶ͹ͳΒͳ͍͕ނʹࣗ༝

    • ਓ݅අΛআ͘ϥϯχϯάίετͷ҆͞

    View Slide

  10. ࠓ೔࿩͢͜ͱ
    • ϖύϘͷKubernetesΫϥελͷ֓ཁ

    • Hardwayͩͬͨ͜ͱ

    • ࠓޙ΍Γ͍ͨ͜ͱ

    View Slide

  11. KubernetesΫϥελ
    • OpenStack (Nyah)

    • Nyah Kubernetes Engine(NKE)

    View Slide

  12. KubernetesΫϥελ
    ن໛ײ
    • ঎ࡐ͝ͱʹΫϥελΛ෼཭͓ͯ͠Γɺ23Ϋϥελ(ൃද࣌఺)

    • ঎ࡐʹΑͬͯ͸NKE / GKE / EKSͰͷϋΠϒϦουΫϥ΢υͰར༻

    • AWS Direct ConnectͰઐ༻ઢར༻

    View Slide

  13. KubernetesΫϥελ
    ٕज़ج൫νʔϜ
    Embedded SRE
    • NKEίϚϯυͷ։ൃ

    • ϓϦηοτϚχϑΣετͷߋ৽
    • Ϋϥελ্Ͱಈ͘ιϑτ΢ΣΞͷಋೖ

    • όʔδϣϯΞοϓͳͲͷϝϯςφϯε
    ։ൃͱར༻ऀ͕ҟͳΔ

    View Slide

  14. NKE
    • ΫϥελͷߏஙɺόʔδϣϯΞοϓ

    • Ϋϥελ؅ཧϚχϑΣετͷద༻

    • Data Planeͷ௥Ճɺ࡟আ

    • AnsibleΛ༻͍ͨϓϩϏδϣχϯά
    Ϋϥελ؅ཧΛίʔυԽ͠CLIΠϯλʔϑΣʔεʹͨ͠΋ͷ

    View Slide

  15. NKE
    ઃఆϑΝΠϧɺൿಗ৘ใετΞʹج͖ͮɺ


    ΫϥελΛߏஙɺӡ༻
    VM
    VM
    VM
    NKE
    tenant-
    con
    fi
    g.toml
    Hashicorp


    Vault
    conta
    iner
    conta
    iner
    conta
    iner

    View Slide

  16. NKE
    • Golang

    • Hashicorp Vault

    • Consul

    • Packer
    ओཁίϯϙʔωϯτ

    View Slide

  17. Kubernetesͷόʔδϣϯ؅ཧ

    View Slide

  18. Kubernetesͷόʔδϣϯ؅ཧ
    • NKEͷϒϥϯν͝ͱʹόʔδϣϯ؅ཧ

    • trunk: ։ൃ༻ϒϥϯν

    • 1.20,1.21 ϦϦʔεϒϥϯν

    View Slide

  19. Kubernetesͷόʔδϣϯ؅ཧ
    trunk
    1.20
    1.21
    Unit Test


    E2E Test
    Unit Test


    E2E Test
    Unit Test


    E2E Test
    merge
    merge

    View Slide

  20. Kubernetesͷόʔδϣϯ؅ཧ
    • CIΛར༻ͨ͠ςετΛύεͨ͠৔߹͸ɺ։ൃ༻Ϋϥελɺࣾ಺πʔϧ༻Ϋ
    ϥελͷόʔδϣϯΞοϓίϚϯυΛ࣮ߦ

    • ֤Ϋϥελͷ؅ཧऀ͕όʔδϣϯΞοϓίϚϯυΛ࣮ߦ

    • ΫϥελʹΑͬͯ͸2ܥ࣋ͭΑ͏ʹͯ͠ɺόʔδϣϯΞοϓ࡞ۀͳͲͷ

    μ΢ϯλΠϜΛආ͚Δ޻෉Λ͍ͯ͠Δ

    View Slide

  21. Kubernetesͷόʔδϣϯ؅ཧ
    • Control Plane,Data Planeͱ΋ʹPodΛ௥͍ग़ͭͭ͠ɺ

    ϩʔϦϯάΞοϓσʔτ

    • Control PlaneɺEtcdʹ͍ͭͯ͸1୆ೖΕସ͑͝ͱʹϔϧενΣοΫΛ

    ͍Εͯμ΢ϯλΠϜΛආ͚͍ͯΔ

    View Slide

  22. Kubernetesͷӡ༻؅ཧ

    View Slide

  23. Kubernetesͷӡ༻؅ཧ
    • ؂ࢹ

    • ηΩϡϦςΟ؂ࠪ

    • CI/CD

    • ϩά؅ཧ

    View Slide

  24. Kubernetesͷ؂ࢹ
    Prometheus

    Alert Manager

    Grafana

    mackerel-agent
    ࣌ܥྻσʔλͷอଘ
    ڞ௨ϧʔϧʹै͍ɺSlack௨஌
    PrometheusͷσʔλͷϏδϡΞϥΠθʔγϣϯ
    Prometheus+AlertManagerͷ؂ࢹ

    View Slide

  25. KubernetesͷηΩϡϦςΟ؂ࠪ
    • Wazuh

    • Falco

    • GateKeeper

    View Slide

  26. Wazuh
    https://atmarkit.itmedia.co.jp/ait/articles/1902/18/news012.html
    OSͷઃఆ؂ࠪ

    ෆਖ਼ΞΫηεݕ஌

    ੬ऑੑ؂ࠪ

    View Slide

  27. Falco
    ίϯςφͷৼΔ෣͍؂ࠪɾݕ஌

    View Slide

  28. Gatekeeper
    Admission ControllerͰಈ࡞͢Δ

    ϚχϑΣετͷ؂ࠪͳͲ
    Ұॹʹ΍ͬͯ͘Δਓɺೖࣾͯ͘͠Εʙʙʙʙ

    View Slide

  29. ࣗಈApply
    ؂ࢹɺηΩϡϦςΟϙϦγʔ͸Ұ੪഑෍
    tag
    cluster A
    cluster B
    cluster C
    apply

    View Slide

  30. CI/CD
    • ςετɺίϯςφϏϧυɺ੬ऑੑεΩϟϯ͸Github ActionsͷSelf Hosted
    Runner্Ͱ࣮ߦ

    • ίϯςφΠϝʔδͷεΩϟϯΤϯδϯ͸trivyΛར༻

    • CD͸ArgoCD + argocd-image-updaterΛར༻

    View Slide

  31. ϩά؅ཧ
    Kafkaʹू໿ͯ͠ɺ༻్ʹԠͯ͡SaaS΁

    View Slide

  32. ͜͜·Ͱ࿩ͨ͜͠ͱ
    • NKEίϚϯυͷ։ൃʹΑͬͯΫϥελͷߏங΍ϝϯςφϯεΛࣗಈԽͯ͠
    ͍Δ

    • ؂ࢹ΍ηΩϡϦςΟ؂ࠪʹ͍ͭͯ͸NKEͰϕʔεͱͳΔ΋ͷΛఏڙ

    • όʔδϣϯΞοϓʹ͍ͭͯ͸E2EͰಈ࡞Λ୲อͭͭ͠ɺ։ൃ༻ΫϥελͰ
    ໰୊͕ͳ͍͜ͱΛ֬ೝͯ͠ɺద༻͍ͯ͠Δ

    View Slide

  33. Hardwayͩͬͨ͜ͱ

    View Slide

  34. 1.12.7

    View Slide

  35. [࠶ܝ]KubernetesΫϥελ
    ٕज़ج൫νʔϜ
    Embedded SRE
    • NKEίϚϯυͷ։ൃ

    • ϓϦηοτϚχϑΣετͷߋ৽
    • Ϋϥελ্Ͱಈ͘ιϑτ΢ΣΞͷಋೖ

    • όʔδϣϯΞοϓͳͲͷϝϯςφϯε
    ։ൃͱར༻ऀ͕ҟͳΔ

    View Slide

  36. όʔδϣϯΞοϓͷಈػ͕௿͍͜ͱ͕͋Δ
    • Ϋϥελͷ༻్

    • ୲౰ऀ͕ଟ๩

    • Kubernetesɺ͍͍ͩͨݹͯ͘΋ಈ͘

    • όʔδϣϯΞοϓʹର͢Δ৺ཧোน

    View Slide

  37. όʔδϣϯ؅ཧࣗಈԽ͍ͨ͠
    NKE Manifests


    Cluster A
    NKE Manifests


    Cluster B
    NKE Manifests


    Cluster C
    NKE
    Cluster A
    Cluster B
    Cluster C
    manifestͷఆٛʹج͍ͮͯࣗಈͰऩଋͯ͠΄͍͠

    View Slide

  38. ͋Δ೔ಥવͷ


    ”error: You must be logged in to the
    server (Unauthorized)”

    View Slide

  39. Կ͕ى͖͔ͨ
    kube-apiserver
    Service


    Account
    token
    ServiceAccountͷར༻͍ͯ͠ΔτʔΫϯ͕ࣦޮͯ͠ೝূΤϥʔ

    View Slide

  40. ͳͥى͖͔ͨ
    • Kubernetes ͷ SAτʔΫϯ͸༗ޮظݶ͕Forever

    • ϖύϘͷKubernetesͷSAτʔΫϯͷ伴͸ࣗಈͰϩʔςʔγϣϯ͍ͯ͠Δ

    View Slide

  41. Կ͕ى͖͔ͨ
    kube-apiserver
    Service


    Account
    token
    ূ໌ॻɺ伴ͷߋ৽ɺ഑෍͸HashicorpVaultͰࣗಈԽ
    kube-controller-
    manager
    τʔΫϯͷ෷͍ग़͠ τʔΫϯͷݕূ
    Hashicorp


    Vault
    Cert Key
    ূ໌ॻͱ伴ͷࣗಈ഑෍

    View Slide

  42. Կ͕ى͖͔ͨ
    kube-apiserver
    Service


    Account
    token
    ূ໌ॻɺ伴ͷߋ৽ɺ഑෍͸HashicorpVaultͰࣗಈԽ
    kube-controller-
    manager
    τʔΫϯͷ෷͍ग़͠ τʔΫϯͷݕূ
    Hashicorp


    Vault
    Cert Key
    ূ໌ॻͱ伴ͷࣗಈ഑෍
    Ӭٱอଘʂʂʂ

    View Slide

  43. Կ͕ى͖͔ͨ
    kube-apiserver
    Service


    Account
    token
    ূ໌ॻɺ伴ͷߋ৽ɺ഑෍͸Hashicorp VaultͰࣗಈԽ
    kube-controller-
    manager
    τʔΫϯͷ෷͍ग़͠ τʔΫϯͷݕূ
    Hashicorp


    Vault
    Cert Key
    ূ໌ॻͱ伴ͷࣗಈ഑෍
    伴͕ߋ৽͞ΕΔ͜ͱͰ


    ݕূ͕Ͱ͖ͳ͘ͳΔ

    View Slide

  44. ରॲʂѹ౗త࢑ఆରॲʂʂʂ
    ಈ͍͍ͯΔϙου͸ಈ͖ଓ͚Δ͕ɺϦεέδϡʔϧ͕Ͱ͖ͳ͍ͷͰ
    ·ͣ͸ShellͰରॲ
    ͜ͷ͋ͱɺূ໌ॻͷঢ়گΛ؂ࢹͯ͠ஔ͖׵͑ΔϓϩηεΛಈ͔͍ͯ͠·͢

    View Slide

  45. ࠷ޙͷॴײ
    • ΦϯϓϨKubernetes΍ΔͳΒ؅ཧιϑτ΢ΣΞΛ։ൃͨ͠΄͏͕౷߹తʹ؅ཧͰ͖ΔͷͰ

    ࠷ऴίετ͸མͱͤΔͱࢥ͏

    • Kubernetesͦͷ΋ͷ͸ͱͯ΋ྑ͘Ͱ͖͍ͯͯɺKubernetesࣗମͷԿ͔Λ౿Ή͜ͱ͸ͦΜͳʹͳ͍

    • ࠓ೔঺հ͍ͯ͠ͳ͍ωοτϫʔΫपΓͷΧʔωϧνϡʔχϯάͳͲɺඞཁͳέʔε΋͋ͬͨͷ
    Ͱɺͦ͏͍͏ྖҬΛݟΕΔਓ͕͍ͳ͍ͱݫ͍͠ͱ͸ࢥ͏

    • େମͷϢʔεέʔε͸VM + DockerͰࣄ଍ΓΔͷͰɺ΄ΜͱʹͦΕKubernetes͍Δͷʁͱ͍͏
    έʔε͸݁ߏ͋Δͱࢥ͏

    • ͜Ε·Ͱͷιϑτ΢ΣΞʹՃ͑ͯɺKubernetesͱ͍͏ϨΠϠʔ͕ೖΔ͜ͱͰτϥϒϧγϡʔτ
    ΍؅ཧ͸େมʹͳΔ

    View Slide

  46. ͓͠·͍
    ࠷৽ͷ࠾༻৘ใΛνΣοΫˠ !QC@SFDSVJU

    View Slide