Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Ansible、Terraform、Packerで作るSelf-Hosted Kubernetes / JKD1812

Ryo Takaishi
December 04, 2018

Ansible、Terraform、Packerで作るSelf-Hosted Kubernetes / JKD1812

Ryo Takaishi

December 04, 2018
Tweet

More Decks by Ryo Takaishi

Other Decks in Technology

Transcript

  1. ∁ੴྒ / GMO Pepabo, Inc.
    2018-12-04 JapanContainerDays v18.12
    ฏ੒࠷ޙͷ12݄ʂʂ1
    "OTJCMFɺ5FSSBGPSNɺ1BDLFS
    Ͱ࡞Δ4FMG)PTUFE,VCFSOFUFT

    View Slide

  2. ∁ੴྒ!S@UBLBJTIJ
    TAKAISHI Ryo
    Software Engineer @ GMO Pepabo, Inc.
    Belgian Beer
    Go, Ruby
    Kubernetes, OpenStack
    https://repl.info/

    View Slide

  3. !3
    • KubernetesΫϥελΛࣗ෼Ͱߏங͍ͨ͠ɻͳͥʁ
    • Self-Hosted Kubernetesͱ͸Կ͔
    • ͲͷΑ͏ͳํ๏Ͱߏங͢Δ͔
    • ΍ͬͯΈͯಘΒΕͨ͜ͱɺࠓޙͷ՝୊ʹ͍ͭͯ
    ࠓ೔͓࿩͢Δ಺༰

    View Slide

  4. • OpenStackΛ༻͍ͨPrivateCloudΛӡ༻͍ͯ͠Δ
    • NyahʢIaaSɺLBaaSʣ
    • AWSͷΑ͏ͳPublicCloud΋࢖༻
    • ϋΠϒϦουΫϥ΢υߏ੒ͷαʔϏε΋͋Δ
    !4
    લఏ
    ϖύϘͷϓϥΠϕʔτΫϥ΢υ “Nyah” ͦͷޙ by @tnmt
    https://speakerdeck.com/tnmt/pepabos-privatecloud-nyah-after-that

    minne ͰͷϋΠϒϦουΫϥ΢υӡ༻ by @shiro16
    https://speakerdeck.com/shiro16/minne-defalsehaiburitudokuraudoyun-yong

    View Slide

  5. • PublicCloudʹ͋ΔɺΑΓޮ཰తʹ։ൃͰ͖ΔϓϥοτϑΥʔϜ͕
    ͳ͍
    !5
    ΑΓޮ཰తʹαʔϏε։ൃɾӡ༻Λ͢Δʹ͸
    IUUQTDMPVEHPPHMFDPNCMPHQSPEVDUTHDQUJNFUPIFMMPXPSMEWNTWTDPOUBJOFSTWTQBBTWTGBBT

    View Slide

  6. • ੍໿Λڧ͘͢Δ͜ͱͰΑΓߴ͍ੜ࢈ੑΛಘΔ
    • Կ͔ΒखΛ͚ͭΔ͔
    • => ·ͣ͸CaaS͕Α͔Ζ͏ɻͳΒKubernetes͔ͳʁ
    • ΑΓखܰʹ࢖͑ΔΠϯϑϥΛ໨ࢦͨ͢ΊɺIaaS෦෼ΛӅṭ͍ͨ͠
    • PaaS΍FaaS͸CaaS্ʹߏஙͰ͖Δ͕ٯ͸೉͍͠
    !6
    $BB41BB4'BB4Λఏڙ͢Δ͜ͱͰੜ࢈ੑ61ˢ

    View Slide

  7. • CI༻్͕த৺ͰɺαʔϏε͸ಈ͔͍ͯ͠ͳ͍
    • SwarmͰ͸ବ໨ͳͷ͔ʁ
    • Kubernetesͷ࣋ͭαʔϏεσΟεΧόϦػೳ΍ӡ༻ࣗಈԽɺ֦ுੑͱ
    ͍ͬͨར఺Λ׆༻͍ͨ͠ؾ͕࣋ͪڧ͍
    !7
    ࣮͸ࣾ಺޲͚ʹ%PDLFS4XBSNΛఏڙத

    View Slide

  8. • ͔͠͠ɺKubernetesͷߏஙɾӡ༻͸ۃྗলྗԽ͍ͨ͠
    • Kubernetesͷӡ༻Ͱফ໣͢Δͷ͸ຊ຤స౗
    • kubelet / apiserver / controller-manager / scheduler / etcd / etc…
    • 1Ϋϥελͱ͸ݶΒͳ͍
    • શͯࣗ෼ͨͪͰ؅ཧ͢Δͷ͸େม
    • ઃఆมߋɺόʔδϣϯΞοϓ
    • ৑௕Խ
    • ো֐࣌ͷ෮چ࡞ۀ
    !8
    Ͳ͏΍ͬͯߏஙɾӡ༻͢Δ͔

    View Slide

  9. • Ϛωʔδυ Self-Hosted Kubernetes͸Ͳ͏ͩΖ͏ʁ
    • Kubernetesࣗ਎ʹࣗ෼ࣗ਎Λ؅ཧͤ͞Δߟ͑ํ
    • Kubernetesͷ࣋ͭߴ౓ͳػೳΛ࢖͏͜ͱ͕Ͱ͖Δ
    • Deployment, Auto Healing, etc…
    • Kubernetes as a Serviceͱ͸ҧ͏ͷ͔
    • ͪ͜Β͸ɺෳ਺ͷΫϥελΛKubernetesͰ؅ཧ͢Δ΋ͷ
    !9
    ,VCFSOFUFTͷ؅ཧΛΑΓָʹ͢Δʹ͸

    View Slide

  10. 4FMG)PTUFE,VCFSOFUFTͱ͸ʁ

    View Slide

  11. !11
    1. Small Dependencies
    2. Deployment consistency
    3. Introspection
    4. Cluster Upgrades
    5. Easier Highly-Available Configurations
    1SPQPTBM4FMGIPTUFE$POUSPM1MBOF
    IUUQTHJUIVCDPNLVCFSOFUFTDPNNVOJUZCMPCNBTUFSDPOUSJCVUPSTEFTJHOQSPQPTBMTDMVTUFSMJGFDZDMFTFMGIPTUFELVCFSOFUFTNE

    View Slide

  12. • kubeletͰ֤ίϯϙʔωϯτΛಈ͔͢͜ͱͰɺϗετ্ʹඞཁͳί
    ϯϙʔωϯτΛݮΒ͢͜ͱ͕Ͱ͖Δ
    • KubernetesͷηοτΞοϓʹ͓͚Δෳࡶͷղফʹܨ͕Δ
    • ࠓճߏஙͨ͠؀ڥͷ৔߹
    • DockerͱkubeletͷΈsystemdͰ؅ཧ
    • ଞͷίϯϙʔωϯτ͸kubelet؅ཧԽ
    !12
    4NBMM%FQFOEFODJFT

    View Slide

  13. • σΟεΫ্ͷϑΝΠϧ΍SSHΛ༻͍ͨઃఆ؅ཧΛݮΒͤΔ
    • ϗετOS΁ͷґଘΛݮΒ͠ɺσϓϩΠͷҰ؏ੑΛಘΔ
    • ࠓճߏஙͨ͠؀ڥͷ৔߹
    • ϑΝΠϧͱͯ͠ஔ͘ඞཁ͕͋ΔϚχϑΣετΛSSHͰ഑ஔ
    • ଞ͸Kubernetes APIܦ༝
    !13
    %FQMPZNFOUDPOTJTUFODZ

    View Slide

  14. • kubectl logsͷΑ͏ͳKubenetes APIΛ࢖ͬͯίϯϙʔωϯτͷσ
    όοά΍ௐ͕ࠪͰ͖Δ
    • ͜Ε·Ͱͩͱɺ֤αʔόʔʹೖΔorू໿͢ΔͨΊͷπʔϧΛೖΕΔ
    ඞཁ͕͋ͬͨ
    • ϩάऩूͷ࢓૊Έ΋Kubernetes্ʹ࡞Ε͹1΍2ͷΑ͏ͳԸܙΛड͚ΒΕ
    Δ
    !14
    *OUSPTQFDUJPO

    View Slide

  15. • Kubernetes APIܦ༝ͰΫϥελʔͷΞοϓάϨʔυ͕Ͱ͖Δ
    • Kubernetes্Ͱಈ͔͢αʔϏεͱಉ͡ํ๏Λ࢖͑Δ
    • RollingUpdate΍RollbackɺศརͰ͢Ͷ
    !15
    $MVTUFS6QHSBEFT

    View Slide

  16. • ؂ࢹ΍HAߏ੒ͷߏஙΛ֎෦πʔϧͳ͠Ͱ࣮ݱͰ͖Δ
    • Kubernetes্Ͱಈ͔͢αʔϏεͱಉ͡ํ๏Ͱ࣮ݱͰ͖Δͷ͕Good
    • ಉ͡ςΫχοΫΛ࢖͑Δͷ͸࠷ߴ
    !16
    &BTJFS)JHIMZ"WBJMBCMF$POpHVSBUJPOT

    View Slide

  17. !17
    ηϧϑϗετ͢Δൣғʹ͍ͭͯ
    0
    1
    2
    3
    4
    kubelet
    etcd
    apiserver
    scheduler
    controller-
    manager
    proxy
    dns addons
    2-4 cluster
    1-4 cluster

    View Slide

  18. Ͳ͏΍ͬͯ࡞Ζ͏ʁ

    View Slide

  19. • bootkubeͷΑ͏ͳطଘͷߏஙπʔϧΛ࢖ͬͯ΋΋ͪΖΜΑ͍
    • ͜Ε·Ͱ࢖͖ͬͯͨπʔϧͰͲ͜·Ͱ΍ΕΔͷ͔ɺࢼͯ͠ΈΑ͏
    • KubernetesࣗମͷཧղΛਂΊ͍ͨ
    • Packer / Terraform / AnsibleΛ࢖͍ͬͯΕ͹ձࣾͷଞϝϯόʔ΋ѻ
    ͍΍͍͢ͷͰ͸ɺͱߟ͑ͨ
    !19
    ࠓճ͸ࣗ࡞͢Δ͜ͱʹͨ͠

    View Slide

  20. ࢓૊Έ

    View Slide

  21. • 2-4Ϋϥελ
    • جຊతʹ Kubernetes on Kubernetes on OpenStack (Ծ) ʹ४ڌ
    • ΊͬͪΌࢀߟʹ͍͍͍ͤͯͨͩͯ͞·͢ :kansya:
    !21
    ࠓճνϟϨϯδͨ͠4FMG)PTUFEʹ͍ͭͯ
    kubelet Systemd؅ཧԼ
    etcd StaticPodΛ༻͍ͯkubelet؅ཧԼ
    apiserver

    controller-manager
    scheduler
    Kubernetes APIΛ༻͍ͯKubernetes؅ཧԼ
    Kubernetes on Kubernetes on OpenStack (Ծ) by @yuanying
    https://docs.google.com/presentation/d/1VKk89MaNkGRSlpBsOOHJt8cLD6mpZ5V55GEJqIDu2Sk/edit#slide=id.g28eb8071bc_0_1119

    View Slide

  22. • Packer
    • ϕʔεΠϝʔδʹDocker΍kubeletɺͦͷଞશϊʔυͰ࢖͏ιϑτ΢ΣΞͷΠϯ
    ετʔϧʹ࢖༻ɻ
    • Ϋϥελ࡞੒࣌ɺϊʔυ௥Ճ࣌ͷ࣌ؒ୹ॖΛݟࠐΜͩ
    • Terraform
    • αʔόʔɺূ໌ॻൃߦʹ࢖༻
    • OpenStack Provider, TLS Provider
    • Ansible
    • Ϋϥελʔߏஙɺߋ৽ʹ࢖༻
    • Dynamic InventoryΛ࢖͏ͱTerraform͔Βద༻ઌIPΞυϨεΛऔಘͰ͖ͯศར
    !22
    1BDLFS5FSSBGPSN"OTJCMF͸Ͳ͜Ͱ࢖͔ͬͨ

    View Slide

  23. 1. Packer / AnsibleͰϕʔεΠϝʔδΛ࡞੒ʢDocker, kubeletʣ
    2. TerraformͰαʔόʔΛىಈ
    3. AnsibleͰBootstrap ClusterΛىಈ
    4. AnsibleͰBootstrap Cluster͔ΒSelf-Hosted Cluster΁੾Γସ͑Δ
    !23
    ߏஙͷྲྀΕ

    View Slide

  24. 1. Packer / AnsibleͰϕʔεΠϝʔδΛ࡞੒ʢDocker, kubeletʣ
    2. TerraformͰαʔόʔΛىಈ
    3. AnsibleͰBootstrap ClusterΛىಈ
    4. AnsibleͰBootstrap Cluster͔ΒSelf-Hosted Cluster΁੾Γସ͑Δ
    !24
    ߏஙͷྲྀΕ

    View Slide

  25. • શαʔόʔ͕ඞཁͱ͢ΔίϯϙʔωϯτΛ༧ΊΠϯετʔϧ
    • Docker, kubeletͳͲ
    • Ϋϥελʔߏஙɺϊʔυ௥Ճ࣌ͷ࣌ؒ୹ॖ͕໨త
    • Πϝʔδߋ৽ͰDockerɾkubeletͷೖΕସ͑Λߦ͏৔߹޻෉͕ඞཁ
    !25
    1BDLFS"OTJCMFͰϕʔεΠϝʔδΛ࡞੒
    Image
    Server
    Packer
    Ansible
    Install Docker, kubelet, etc
    Launch Server
    Save Base Image

    View Slide

  26. 1. Packer / AnsibleͰϕʔεΠϝʔδΛ࡞੒ʢDocker, kubeletʣ
    2. TerraformͰαʔόʔΛىಈ
    3. AnsibleͰBootstrap ClusterΛىಈ
    4. AnsibleͰBootstrap Cluster͔ΒSelf-Hosted Cluster΁੾Γସ͑Δ
    !26
    ߏஙͷྲྀΕ

    View Slide

  27. 5FSSBGPSNͰαʔόʔΛىಈ͢Δ
    Docker
    kubelet
    master
    Docker
    kubelet
    master
    Docker
    kubelet
    master
    Docker
    kubelet
    node
    Docker
    kubelet
    node
    Terraform
    Launch servers
    certificates and private key

    View Slide

  28. 5FSSBGPSNͰαʔόʔΛىಈ͢Δ
    Docker
    kubelet
    master
    Docker
    kubelet
    master
    Docker
    kubelet
    master
    Docker
    kubelet
    node
    Docker
    kubelet
    node
    Terraform
    Launch servers
    certificates and private key
    • ֤छূ໌ॻɾൿີ伴͸͜ͷλΠϛϯάͰੜ੒
    • TLS Provider
    • repl.infoɿTerraformͷTLS ProviderΛࢼ͢
    • αʔόʔىಈ࣌ʹcloud-configͰ౉ͯ͠഑ஔ
    • tfstateʹฏจͰอଘ͞ΕΔ(!)ͨΊɺࠓޙํࣜΛม͑Δ༧ఆ

    View Slide

  29. 1. Packer / AnsibleͰϕʔεΠϝʔδΛ࡞੒ʢDocker, kubeletʣ
    2. TerraformͰαʔόʔΛىಈ
    3. AnsibleͰBootstrap ClusterΛىಈ
    4. AnsibleͰBootstrap Cluster͔ΒSelf-Hosted Cluster΁੾Γସ͑Δ
    !29
    ߏஙͷྲྀΕ

    View Slide

  30. "OTJCMFͰ#PPUTUSBQ$MVTUFSΛىಈ͢Δ
    Docker
    kubelet
    etcd
    master
    Docker
    kubelet
    etcd
    master
    Docker
    kubelet
    etcd
    master
    Docker
    kubelet
    node
    Docker
    kubelet
    node
    Ansible
    Launch etcd cluster
    Static Pod manifest files

    View Slide

  31. "OTJCMFͰ#PPUTUSBQ$MVTUFSΛىಈ͢Δ
    Docker
    kubelet
    etcd
    master
    Docker
    kubelet
    etcd
    master
    Docker
    kubelet
    etcd
    master
    Docker
    kubelet
    node
    Docker
    kubelet
    node
    • etcd clusterΛߏங͢Δ
    • masterʹStatic Pod༻ϚχϑΣετϑΝΠϧΛஔ͘
    • Static Pod ?
    Ansible
    Launch etcd cluster
    Static Pod manifest via files

    View Slide

  32. • kubelet͸ࢦఆͨ͠σΟϨΫτϦΛ؂ࢹͯ͠ɺஔ͔ΕͨϚχϑΣετ
    ϑΝΠϧΛݩʹPodΛىಈ͢ΔػೳΛ࣋ͭ
    • ͜ΕΛStatic PodͱݺͿ
    !32
    4UBUJD1PEʁʁʁ
    kubelet /etc/kubernetes/manifests
    XBUDI
    ϚχϑΣετϑΝΠϧ͕ஔ͔ΕͨΒ1PEΛ࡞੒

    ফ͞ΕͨΒ1PEΛ࡟আ
    Docker IUUQTLVCFSOFUFTJPEPDTUBTLTBENJOJTUFSDMVTUFSTUBUJDQPE
    ŠQPENBOJGFTUQBUI
    ŠNBOJGFTUVSM
    http://pod.example.com/

    View Slide

  33. "OTJCMFͰ#PPUTUSBQ$MVTUFSΛىಈ͢Δ
    Docker
    kubelet
    etcd
    bootstrap apiserver
    bootstrap controller-manager
    bootstrap scheduler
    master
    Docker
    kubelet
    etcd
    master
    Docker
    kubelet
    etcd
    master
    Docker
    kubelet
    node
    Docker
    kubelet
    node
    Ansible
    Launch bootstrap cluster
    StaticPod manifest via files
    apiserverɿ໊લͷ௨ΓɺKubernetes APIͷΤϯυϙΠϯτ
    schedulerɿPodΛNodeʹׂΓ౰ͯΔ
    controller-managerɿReplicaͳͲΛίϯτϩʔϧ͢Δ

    View Slide

  34. "OTJCMFͰ#PPUTUSBQ$MVTUFSΛىಈ͢Δ
    Docker
    kubelet
    etcd
    bootstrap apiserver
    bootstrap controller-0manager
    bootstrap scheduler
    master
    Docker
    kubelet
    etcd
    master
    Docker
    kubelet
    etcd
    master
    Docker
    kubelet
    node
    Docker
    kubelet
    node
    • apiserver, controller-manager, schedulerΛStatic PodͰىಈ
    • ͜ͷ࣌఺ͰKubernetes API͕ར༻ՄೳͱͳΔ
    • ֤αʔόʔͷkubelet͕apiserver΁ొ࿥ɺAPIͰݟ͑ΔΑ͏ʹͳΔ
    • controller-manager, scheduler΋ಈ͍͍ͯΔͷͰDeploymentͳͲ΋࢖͑Δ
    Ansible
    Launch bootstrap cluster
    StaticPod manifest via files

    View Slide

  35. "OTJCMFͰ#PPUTUSBQ$MVTUFSΛىಈ͢Δ
    Docker
    kubelet
    etcd
    bootstrap apiserver
    bootstrap controller-manager
    bootstrap scheduler
    master
    Docker
    kubelet
    etcd
    master
    Docker
    kubelet
    etcd
    master
    Docker
    kubelet
    node
    Docker
    kubelet
    node
    register node
    Ansible
    Launch bootstrap cluster
    StaticPod manifest via files

    View Slide

  36. 1. Packer / AnsibleͰϕʔεΠϝʔδΛ࡞੒ʢDocker, kubeletʣ
    2. TerraformͰαʔόʔΛىಈ
    3. AnsibleͰBootstrap ClusterΛىಈ
    4. AnsibleͰBootstrap Cluster͔ΒSelf-Hosted Cluster΁੾Γ
    ସ͑Δ
    !36
    ߏஙͷྲྀΕ

    View Slide

  37. 4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ
    Docker
    kubelet
    etcd
    master
    Docker
    kubelet
    etcd
    master
    Docker
    kubelet
    etcd
    master
    Docker
    kubelet
    node
    Docker
    kubelet
    node
    bootstrap apiserver
    bootstrap controller-manager
    bootstrap scheduler
    Ansible
    Launch self-hosted cluster
    Manifest files via Kubernetes API

    View Slide

  38. 4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ
    Docker
    kubelet
    etcd
    kube-proxy
    master
    Docker
    kubelet
    etcd
    kube-proxy
    master
    Docker
    kubelet
    etcd
    kube-proxy
    master
    Docker
    kubelet
    kube-proxy
    node
    Docker
    kubelet
    kube-proxy
    node
    bootstrap apiserver
    bootstrap controller-manager
    bootstrap scheduler
    Ansible
    Launch self-hosted cluster
    Manifest files via Kubernetes API

    View Slide

  39. 4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ
    Docker
    kubelet
    etcd
    kube-proxy
    flannel
    master
    Docker
    kubelet
    etcd
    kube-proxy
    flannel
    master
    Docker
    kubelet
    etcd
    kube-proxy
    flannel
    master
    Docker
    kubelet
    kube-proxy
    flannel
    node
    Docker
    kubelet
    kube-proxy
    flannel
    node
    bootstrap apiserver
    bootstrap controller-manager
    bootstrap scheduler
    Ansible
    Launch self-hosted cluster
    Manifest files via Kubernetes API

    View Slide

  40. 4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    master
    Docker
    kubelet
    kube-proxy
    flannel
    node
    Docker
    kubelet
    kube-proxy
    flannel
    node
    bootstrap apiserver
    bootstrap controller-manager
    bootstrap scheduler
    Ansible
    Launch self-hosted cluster
    Manifest files via Kubernetes API
    NodeAffinityΛ࢖ͬͯmasterͷΈىಈ

    View Slide

  41. 4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    scheduler
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    scheduler
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    master
    Docker
    kubelet
    kube-proxy
    flannel
    node
    Docker
    kubelet
    kube-proxy
    flannel
    node
    bootstrap apiserver
    bootstrap controller-manager
    bootstrap scheduler
    Ansible
    Launch self-hosted cluster
    Manifest files via Kubernetes API

    View Slide

  42. 4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    scheduler
    controller-manager
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    scheduler
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    controller-manager
    master
    Docker
    kubelet
    kube-proxy
    flannel
    node
    Docker
    kubelet
    kube-proxy
    flannel
    node
    bootstrap apiserver
    bootstrap controller-manager
    bootstrap scheduler
    Ansible
    Launch self-hosted cluster
    Manifest files via Kubernetes API
    NodeAffinityΛ࢖ͬͯmasterͷΈىಈ

    View Slide

  43. 4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    scheduler
    controller-manager
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    scheduler
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    controller-manager
    master
    Docker
    kubelet
    kube-proxy
    flannel
    node
    Docker
    kubelet
    kube-proxy
    flannel
    node
    bootstrap apiserver
    bootstrap controller-manager
    bootstrap scheduler
    • DaemonSetɺDeploymentͰىಈ
    Ansible
    Launch self-hosted cluster
    Manifest files via Kubernetes API

    View Slide

  44. 4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    scheduler
    controller-manager
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    scheduler
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    controller-manager
    master
    Docker
    kubelet
    kube-proxy
    flannel
    node
    Docker
    kubelet
    kube-proxy
    flannel
    node
    bootstrap apiserver
    bootstrap controller-manager
    bootstrap scheduler
    • bootstrap apiserverͱapiserverͰ࢖༻ϙʔτ͕
    িಥͦ͠͏
    • ࣮ࡍɺ͜ͷapiserver͸ىಈʹࣦഊ͢Δ
    • ͔͠͠ɺ͜ͷޙແࣄʹىಈͯ͘͠Δ
    Ansible
    Launch self-hosted cluster
    Manifest files via Kubernetes API

    View Slide

  45. #PPUTUSBQ$MVTUFS͔Β4FMG)PTUFE$MVTUFS΁੾Γସ͑Δ
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    scheduler
    controller-manager
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    scheduler
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    controller-manager
    master
    Docker
    kubelet
    kube-proxy
    flannel
    node
    Docker
    kubelet
    kube-proxy
    flannel
    node
    bootstrap apiserver
    bootstrap controller-manager
    bootstrap scheduler
    Ansible
    Delete bootstrap cluster
    Delete StaticPod manifest files

    View Slide

  46. #PPUTUSBQ$MVTUFS͔Β4FMG)PTUFE$MVTUFS΁੾Γସ͑Δ
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    scheduler
    controller-manager
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    scheduler
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    controller-manager
    master
    Docker
    kubelet
    kube-proxy
    flannel
    node
    Docker
    kubelet
    kube-proxy
    flannel
    node
    Ansible
    Delete bootstrap cluster
    Delete StaticPod manifest files

    View Slide

  47. #PPUTUSBQ$MVTUFS͔Β4FMG)PTUFE$MVTUFS΁੾Γସ͑Δ
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    scheduler
    controller-manager
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    scheduler
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    controller-manager
    master
    Docker
    kubelet
    kube-proxy
    flannel
    node
    Docker
    kubelet
    kube-proxy
    flannel
    node
    • bootstrap apiserverͷϓϩηε͕͍ͳ͘ͳͬͨ
    ͷͰɺapiserver͸ىಈϦτϥΠʹΑΓىಈͯ͠
    ͘Δ
    Ansible
    Delete bootstrap cluster
    Delete StaticPod manifest files

    View Slide

  48. !48
    )JHI"WBJMBCJMJUZ
    etcd StaticPodΛ࢖͍ɺmaster3୆ʙͰΫϥελԽ
    apiserver
    DaemonSetΛ࢖͍ɺmasterશ୆ͰՔಇ
    keepalived + haproxyͰ৑௕Խ
    (Ϋϥελ֎ʹLBaaSͰLoadBalancerΛઃஔ͢Δͷ΋Մʣ
    controller-manager
    scheduler
    DeploymentΛ࢖͍ɺmasterෳ਺୆ͰՔಇ

    View Slide

  49. ಘΒΕͨ΋ͷɺࠓޙͷ՝୊

    View Slide

  50. • ཧ۶Ͱ͸෼͔ͬͯ΋ɺखΛಈ͔ͯ͠ΈΔ͜ͱ͸େ੾
    • Bootstrap Cluster͔ΒSelf-Hosted Clusterʹ੾Γସ͑Δ͜ͱ͕Ͱ͖
    ͨ࣌͸ײಈͨ͠
    • Ͳ͜·Ͱ͸؆୯ʹ࣮૷Ͱ͖ͯɺͲ͔͜Β೉͘͠ͳΔͷ͔΋஌Δ͜ͱ
    ͕Ͱ͖ͨ
    !50
    ࣗ෼ࣗ਎Λ؅ཧ͢Δɺͱ͍͏ํ๏ʹ͍ͭͯͷཧղ

    View Slide

  51. • Ϋϥελࣗମͷߏஙɾӡ༻
    • Կͷίϯϙʔωϯτ͕ඞཁ͔ɺԿͷ໾ׂΛ͍࣋ͬͯΔͷ͔
    • ֤ίϯϙʔωϯτͷΦϓγϣϯʹ͍ͭͯͷཧղʢΊͪΌଟ͍Ͱ͢ΑͶ…ʣ
    • Ϋϥελ্Ͱιϑτ΢ΣΞΛ͏·͘ಈ͔͢
    • controller-manager΍schedulerͳͲͷPodΛ࠷௿1ͭ͸ҡ͍࣋ͨ͠
    • PodͷೖΕସ͑΍NodeͷDrain
    !51
    Ϋϥελͷ্ɾԼ྆ํͷ஌ݟΛಉ࣌ʹಘΒΕͨ

    View Slide

  52. • ݱࡏ͸Ansibleɿ1ϊʔυͣͭαʔϏεΞ΢τ&ߋ৽&αʔϏεΠϯ
    • ࠷ॳ͸Immutable Infrastructureͷߟ͑ํΛద༻Ͱ͖ͳ͍͔ߟ͑ͨ
    • ϕʔεΠϝʔδΛೖΕସ͑ͯߋ৽
    • TerraformͰ΍Ζ͏ͱ͢Δͱ݁ߏେม
    • masterϊʔυ΍etcdΛͲ͏͢Δʁ
    • ࠓޙ͸Custom ControllerΛ࢖ͬͯ؅ཧ͢Δ͜ͱΛݕ౼͍ͯ͠Δ
    !52
    %PDLFS΍LVCFMFUͷߋ৽ΛͲ͏͢Δ͔

    View Slide

  53. • TerraformͰϊʔυͷ૿ݮ͸Մೳ
    • ݮΒ࣌͢ʹdrain͢Δඞཁ͕͋Δ
    • Kubernetes؅ཧԼʹஔ͚ΔͱΑΓΑ͘Ͱ͖ͦ͏
    • Kubernetes APIܦ༝ͰεέʔϧΠϯɾΞ΢τ
    • HPAͷΑ͏ͳΦʔτεέʔϧػߏͱ૊Έ߹ΘͤΔ͜ͱ΋Ͱ͖ͦ͏
    • ͪ͜Β΋Custom ControllerԽ͢Δ͜ͱΛݕ౼
    !53
    ϊʔυͷ૿ݮΛͲ͏͢Δ͔

    View Slide

  54. ·ͱΊ

    View Slide

  55. • ൚༻తͳπʔϧΛ࢖ͬͯSelf-Hosted KubernetesͷߏஙΛߦͬͨ
    • ͔ͳΓ͍͍ײ͡ʹಈ࡞͢ΔΫϥελʔΛ࡞Δ͜ͱ͕Ͱ͖ͨ
    • ͔͠͠ɺݶք΋ײͨ͡
    • ϊʔυͷೖΕସ͑
    • ϊʔυ಺ίϯϙʔωϯτͷߋ৽
    • ࠓޙ͸Custom ControllerΛ׆༻ͯ͠ΑΓޮ཰తͳ؅ཧΛ࣮ݱ͍ͨ͠
    !55
    ·ͱΊ

    View Slide

  56. View Slide