Ansible、Terraform、Packerで作るSelf-Hosted Kubernetes / JKD1812

C84357a21083c81c5ccd5550422abc8d?s=47 Ryo Takaishi
December 04, 2018

Ansible、Terraform、Packerで作るSelf-Hosted Kubernetes / JKD1812

C84357a21083c81c5ccd5550422abc8d?s=128

Ryo Takaishi

December 04, 2018
Tweet

Transcript

  1. 2.

    ∁ੴྒ!S@UBLBJTIJ TAKAISHI Ryo Software Engineer @ GMO Pepabo, Inc. Belgian

    Beer Go, Ruby Kubernetes, OpenStack https://repl.info/
  2. 4.

    • OpenStackΛ༻͍ͨPrivateCloudΛӡ༻͍ͯ͠Δ • NyahʢIaaSɺLBaaSʣ • AWSͷΑ͏ͳPublicCloud΋࢖༻ • ϋΠϒϦουΫϥ΢υߏ੒ͷαʔϏε΋͋Δ !4 લఏ

    ϖύϘͷϓϥΠϕʔτΫϥ΢υ “Nyah” ͦͷޙ by @tnmt https://speakerdeck.com/tnmt/pepabos-privatecloud-nyah-after-that
 minne ͰͷϋΠϒϦουΫϥ΢υӡ༻ by @shiro16 https://speakerdeck.com/shiro16/minne-defalsehaiburitudokuraudoyun-yong
  3. 8.

    • ͔͠͠ɺKubernetesͷߏஙɾӡ༻͸ۃྗলྗԽ͍ͨ͠ • Kubernetesͷӡ༻Ͱফ໣͢Δͷ͸ຊ຤స౗ • kubelet / apiserver / controller-manager

    / scheduler / etcd / etc… • 1Ϋϥελͱ͸ݶΒͳ͍ • શͯࣗ෼ͨͪͰ؅ཧ͢Δͷ͸େม • ઃఆมߋɺόʔδϣϯΞοϓ • ৑௕Խ • ো֐࣌ͷ෮چ࡞ۀ !8 Ͳ͏΍ͬͯߏஙɾӡ༻͢Δ͔
  4. 9.

    • Ϛωʔδυ Self-Hosted Kubernetes͸Ͳ͏ͩΖ͏ʁ • Kubernetesࣗ਎ʹࣗ෼ࣗ਎Λ؅ཧͤ͞Δߟ͑ํ • Kubernetesͷ࣋ͭߴ౓ͳػೳΛ࢖͏͜ͱ͕Ͱ͖Δ • Deployment,

    Auto Healing, etc… • Kubernetes as a Serviceͱ͸ҧ͏ͷ͔ • ͪ͜Β͸ɺෳ਺ͷΫϥελΛKubernetesͰ؅ཧ͢Δ΋ͷ !9 ,VCFSOFUFTͷ؅ཧΛΑΓָʹ͢Δʹ͸
  5. 11.

    !11 1. Small Dependencies 2. Deployment consistency 3. Introspection 4.

    Cluster Upgrades 5. Easier Highly-Available Configurations 1SPQPTBM4FMGIPTUFE$POUSPM1MBOF IUUQTHJUIVCDPNLVCFSOFUFTDPNNVOJUZCMPCNBTUFSDPOUSJCVUPSTEFTJHOQSPQPTBMTDMVTUFSMJGFDZDMFTFMGIPTUFELVCFSOFUFTNE
  6. 17.

    !17 ηϧϑϗετ͢Δൣғʹ͍ͭͯ 0 1 2 3 4 kubelet etcd apiserver

    scheduler controller- manager proxy dns addons 2-4 cluster 1-4 cluster
  7. 20.
  8. 21.

    • 2-4Ϋϥελ • جຊతʹ Kubernetes on Kubernetes on OpenStack (Ծ)

    ʹ४ڌ • ΊͬͪΌࢀߟʹ͍͍͍ͤͯͨͩͯ͞·͢ :kansya: !21 ࠓճνϟϨϯδͨ͠4FMG)PTUFEʹ͍ͭͯ kubelet Systemd؅ཧԼ etcd StaticPodΛ༻͍ͯkubelet؅ཧԼ apiserver
 controller-manager scheduler Kubernetes APIΛ༻͍ͯKubernetes؅ཧԼ Kubernetes on Kubernetes on OpenStack (Ծ) by @yuanying https://docs.google.com/presentation/d/1VKk89MaNkGRSlpBsOOHJt8cLD6mpZ5V55GEJqIDu2Sk/edit#slide=id.g28eb8071bc_0_1119
  9. 22.

    • Packer • ϕʔεΠϝʔδʹDocker΍kubeletɺͦͷଞશϊʔυͰ࢖͏ιϑτ΢ΣΞͷΠϯ ετʔϧʹ࢖༻ɻ • Ϋϥελ࡞੒࣌ɺϊʔυ௥Ճ࣌ͷ࣌ؒ୹ॖΛݟࠐΜͩ • Terraform •

    αʔόʔɺূ໌ॻൃߦʹ࢖༻ • OpenStack Provider, TLS Provider • Ansible • Ϋϥελʔߏஙɺߋ৽ʹ࢖༻ • Dynamic InventoryΛ࢖͏ͱTerraform͔Βద༻ઌIPΞυϨεΛऔಘͰ͖ͯศར !22 1BDLFS5FSSBGPSN"OTJCMF͸Ͳ͜Ͱ࢖͔ͬͨ
  10. 23.

    1. Packer / AnsibleͰϕʔεΠϝʔδΛ࡞੒ʢDocker, kubeletʣ 2. TerraformͰαʔόʔΛىಈ 3. AnsibleͰBootstrap ClusterΛىಈ

    4. AnsibleͰBootstrap Cluster͔ΒSelf-Hosted Cluster΁੾Γସ͑Δ !23 ߏஙͷྲྀΕ
  11. 24.

    1. Packer / AnsibleͰϕʔεΠϝʔδΛ࡞੒ʢDocker, kubeletʣ 2. TerraformͰαʔόʔΛىಈ 3. AnsibleͰBootstrap ClusterΛىಈ

    4. AnsibleͰBootstrap Cluster͔ΒSelf-Hosted Cluster΁੾Γସ͑Δ !24 ߏஙͷྲྀΕ
  12. 26.

    1. Packer / AnsibleͰϕʔεΠϝʔδΛ࡞੒ʢDocker, kubeletʣ 2. TerraformͰαʔόʔΛىಈ 3. AnsibleͰBootstrap ClusterΛىಈ

    4. AnsibleͰBootstrap Cluster͔ΒSelf-Hosted Cluster΁੾Γସ͑Δ !26 ߏஙͷྲྀΕ
  13. 27.

    5FSSBGPSNͰαʔόʔΛىಈ͢Δ Docker kubelet master Docker kubelet master Docker kubelet master

    Docker kubelet node Docker kubelet node Terraform Launch servers certificates and private key
  14. 28.

    5FSSBGPSNͰαʔόʔΛىಈ͢Δ Docker kubelet master Docker kubelet master Docker kubelet master

    Docker kubelet node Docker kubelet node Terraform Launch servers certificates and private key • ֤छূ໌ॻɾൿີ伴͸͜ͷλΠϛϯάͰੜ੒ • TLS Provider • repl.infoɿTerraformͷTLS ProviderΛࢼ͢ • αʔόʔىಈ࣌ʹcloud-configͰ౉ͯ͠഑ஔ • tfstateʹฏจͰอଘ͞ΕΔ(!)ͨΊɺࠓޙํࣜΛม͑Δ༧ఆ
  15. 29.

    1. Packer / AnsibleͰϕʔεΠϝʔδΛ࡞੒ʢDocker, kubeletʣ 2. TerraformͰαʔόʔΛىಈ 3. AnsibleͰBootstrap ClusterΛىಈ

    4. AnsibleͰBootstrap Cluster͔ΒSelf-Hosted Cluster΁੾Γସ͑Δ !29 ߏஙͷྲྀΕ
  16. 30.

    "OTJCMFͰ#PPUTUSBQ$MVTUFSΛىಈ͢Δ Docker kubelet etcd master Docker kubelet etcd master Docker

    kubelet etcd master Docker kubelet node Docker kubelet node Ansible Launch etcd cluster Static Pod manifest files
  17. 31.

    "OTJCMFͰ#PPUTUSBQ$MVTUFSΛىಈ͢Δ Docker kubelet etcd master Docker kubelet etcd master Docker

    kubelet etcd master Docker kubelet node Docker kubelet node • etcd clusterΛߏங͢Δ • masterʹStatic Pod༻ϚχϑΣετϑΝΠϧΛஔ͘ • Static Pod ? Ansible Launch etcd cluster Static Pod manifest via files
  18. 32.

    • kubelet͸ࢦఆͨ͠σΟϨΫτϦΛ؂ࢹͯ͠ɺஔ͔ΕͨϚχϑΣετ ϑΝΠϧΛݩʹPodΛىಈ͢ΔػೳΛ࣋ͭ • ͜ΕΛStatic PodͱݺͿ !32 4UBUJD1PEʁʁʁ kubelet /etc/kubernetes/manifests

    XBUDI ϚχϑΣετϑΝΠϧ͕ஔ͔ΕͨΒ1PEΛ࡞੒
 ফ͞ΕͨΒ1PEΛ࡟আ Docker IUUQTLVCFSOFUFTJPEPDTUBTLTBENJOJTUFSDMVTUFSTUBUJDQPE ŠQPENBOJGFTUQBUI ŠNBOJGFTUVSM http://pod.example.com/
  19. 33.

    "OTJCMFͰ#PPUTUSBQ$MVTUFSΛىಈ͢Δ Docker kubelet etcd bootstrap apiserver bootstrap controller-manager bootstrap scheduler

    master Docker kubelet etcd master Docker kubelet etcd master Docker kubelet node Docker kubelet node Ansible Launch bootstrap cluster StaticPod manifest via files apiserverɿ໊લͷ௨ΓɺKubernetes APIͷΤϯυϙΠϯτ schedulerɿPodΛNodeʹׂΓ౰ͯΔ controller-managerɿReplicaͳͲΛίϯτϩʔϧ͢Δ
  20. 34.

    "OTJCMFͰ#PPUTUSBQ$MVTUFSΛىಈ͢Δ Docker kubelet etcd bootstrap apiserver bootstrap controller-0manager bootstrap scheduler

    master Docker kubelet etcd master Docker kubelet etcd master Docker kubelet node Docker kubelet node • apiserver, controller-manager, schedulerΛStatic PodͰىಈ • ͜ͷ࣌఺ͰKubernetes API͕ར༻ՄೳͱͳΔ • ֤αʔόʔͷkubelet͕apiserver΁ొ࿥ɺAPIͰݟ͑ΔΑ͏ʹͳΔ • controller-manager, scheduler΋ಈ͍͍ͯΔͷͰDeploymentͳͲ΋࢖͑Δ Ansible Launch bootstrap cluster StaticPod manifest via files
  21. 35.

    "OTJCMFͰ#PPUTUSBQ$MVTUFSΛىಈ͢Δ Docker kubelet etcd bootstrap apiserver bootstrap controller-manager bootstrap scheduler

    master Docker kubelet etcd master Docker kubelet etcd master Docker kubelet node Docker kubelet node register node Ansible Launch bootstrap cluster StaticPod manifest via files
  22. 36.

    1. Packer / AnsibleͰϕʔεΠϝʔδΛ࡞੒ʢDocker, kubeletʣ 2. TerraformͰαʔόʔΛىಈ 3. AnsibleͰBootstrap ClusterΛىಈ

    4. AnsibleͰBootstrap Cluster͔ΒSelf-Hosted Cluster΁੾Γ ସ͑Δ !36 ߏஙͷྲྀΕ
  23. 37.

    4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ Docker kubelet etcd master Docker kubelet etcd master Docker

    kubelet etcd master Docker kubelet node Docker kubelet node bootstrap apiserver bootstrap controller-manager bootstrap scheduler Ansible Launch self-hosted cluster Manifest files via Kubernetes API
  24. 38.

    4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ Docker kubelet etcd kube-proxy master Docker kubelet etcd kube-proxy

    master Docker kubelet etcd kube-proxy master Docker kubelet kube-proxy node Docker kubelet kube-proxy node bootstrap apiserver bootstrap controller-manager bootstrap scheduler Ansible Launch self-hosted cluster Manifest files via Kubernetes API
  25. 39.

    4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ Docker kubelet etcd kube-proxy flannel master Docker kubelet etcd

    kube-proxy flannel master Docker kubelet etcd kube-proxy flannel master Docker kubelet kube-proxy flannel node Docker kubelet kube-proxy flannel node bootstrap apiserver bootstrap controller-manager bootstrap scheduler Ansible Launch self-hosted cluster Manifest files via Kubernetes API
  26. 40.

    4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ Docker kubelet etcd apiserver kube-proxy flannel master Docker kubelet

    etcd apiserver kube-proxy flannel master Docker kubelet etcd apiserver kube-proxy flannel master Docker kubelet kube-proxy flannel node Docker kubelet kube-proxy flannel node bootstrap apiserver bootstrap controller-manager bootstrap scheduler Ansible Launch self-hosted cluster Manifest files via Kubernetes API NodeAffinityΛ࢖ͬͯmasterͷΈىಈ
  27. 41.

    4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ Docker kubelet etcd apiserver kube-proxy flannel scheduler master Docker

    kubelet etcd apiserver kube-proxy flannel scheduler master Docker kubelet etcd apiserver kube-proxy flannel master Docker kubelet kube-proxy flannel node Docker kubelet kube-proxy flannel node bootstrap apiserver bootstrap controller-manager bootstrap scheduler Ansible Launch self-hosted cluster Manifest files via Kubernetes API
  28. 42.

    4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ Docker kubelet etcd apiserver kube-proxy flannel scheduler controller-manager master

    Docker kubelet etcd apiserver kube-proxy flannel scheduler master Docker kubelet etcd apiserver kube-proxy flannel controller-manager master Docker kubelet kube-proxy flannel node Docker kubelet kube-proxy flannel node bootstrap apiserver bootstrap controller-manager bootstrap scheduler Ansible Launch self-hosted cluster Manifest files via Kubernetes API NodeAffinityΛ࢖ͬͯmasterͷΈىಈ
  29. 43.

    4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ Docker kubelet etcd apiserver kube-proxy flannel scheduler controller-manager master

    Docker kubelet etcd apiserver kube-proxy flannel scheduler master Docker kubelet etcd apiserver kube-proxy flannel controller-manager master Docker kubelet kube-proxy flannel node Docker kubelet kube-proxy flannel node bootstrap apiserver bootstrap controller-manager bootstrap scheduler • DaemonSetɺDeploymentͰىಈ Ansible Launch self-hosted cluster Manifest files via Kubernetes API
  30. 44.

    4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ Docker kubelet etcd apiserver kube-proxy flannel scheduler controller-manager master

    Docker kubelet etcd apiserver kube-proxy flannel scheduler master Docker kubelet etcd apiserver kube-proxy flannel controller-manager master Docker kubelet kube-proxy flannel node Docker kubelet kube-proxy flannel node bootstrap apiserver bootstrap controller-manager bootstrap scheduler • bootstrap apiserverͱapiserverͰ࢖༻ϙʔτ͕ িಥͦ͠͏ • ࣮ࡍɺ͜ͷapiserver͸ىಈʹࣦഊ͢Δ • ͔͠͠ɺ͜ͷޙແࣄʹىಈͯ͘͠Δ Ansible Launch self-hosted cluster Manifest files via Kubernetes API
  31. 45.

    #PPUTUSBQ$MVTUFS͔Β4FMG)PTUFE$MVTUFS΁੾Γସ͑Δ Docker kubelet etcd apiserver kube-proxy flannel scheduler controller-manager master

    Docker kubelet etcd apiserver kube-proxy flannel scheduler master Docker kubelet etcd apiserver kube-proxy flannel controller-manager master Docker kubelet kube-proxy flannel node Docker kubelet kube-proxy flannel node bootstrap apiserver bootstrap controller-manager bootstrap scheduler Ansible Delete bootstrap cluster Delete StaticPod manifest files
  32. 46.

    #PPUTUSBQ$MVTUFS͔Β4FMG)PTUFE$MVTUFS΁੾Γସ͑Δ Docker kubelet etcd apiserver kube-proxy flannel scheduler controller-manager master

    Docker kubelet etcd apiserver kube-proxy flannel scheduler master Docker kubelet etcd apiserver kube-proxy flannel controller-manager master Docker kubelet kube-proxy flannel node Docker kubelet kube-proxy flannel node Ansible Delete bootstrap cluster Delete StaticPod manifest files
  33. 47.

    #PPUTUSBQ$MVTUFS͔Β4FMG)PTUFE$MVTUFS΁੾Γସ͑Δ Docker kubelet etcd apiserver kube-proxy flannel scheduler controller-manager master

    Docker kubelet etcd apiserver kube-proxy flannel scheduler master Docker kubelet etcd apiserver kube-proxy flannel controller-manager master Docker kubelet kube-proxy flannel node Docker kubelet kube-proxy flannel node • bootstrap apiserverͷϓϩηε͕͍ͳ͘ͳͬͨ ͷͰɺapiserver͸ىಈϦτϥΠʹΑΓىಈͯ͠ ͘Δ Ansible Delete bootstrap cluster Delete StaticPod manifest files
  34. 53.

    • TerraformͰϊʔυͷ૿ݮ͸Մೳ • ݮΒ࣌͢ʹdrain͢Δඞཁ͕͋Δ • Kubernetes؅ཧԼʹஔ͚ΔͱΑΓΑ͘Ͱ͖ͦ͏ • Kubernetes APIܦ༝ͰεέʔϧΠϯɾΞ΢τ •

    HPAͷΑ͏ͳΦʔτεέʔϧػߏͱ૊Έ߹ΘͤΔ͜ͱ΋Ͱ͖ͦ͏ • ͪ͜Β΋Custom ControllerԽ͢Δ͜ͱΛݕ౼ !53 ϊʔυͷ૿ݮΛͲ͏͢Δ͔
  35. 54.
  36. 55.
  37. 56.