Ansible、Terraform、Packerで作るSelf-Hosted Kubernetes / JKD1812

Transcript

1. ∁ੴྒ / GMO Pepabo, Inc.
   2018-12-04 JapanContainerDays v18.12
   ฏ੒࠷ޙͷ12݄ʂʂ1
   "OTJCMFɺ5FSSBGPSNɺ1BDLFS
   Ͱ࡞Δ4FMG)PTUFE,VCFSOFUFT
   

   View Slide

2. ∁ੴྒ!S@UBLBJTIJ
   TAKAISHI Ryo
   Software Engineer @ GMO Pepabo, Inc.
   Belgian Beer
   Go, Ruby
   Kubernetes, OpenStack
   https://repl.info/
   

   View Slide

3. !3
   • KubernetesΫϥελΛࣗ෼Ͱߏங͍ͨ͠ɻͳͥʁ
   • Self-Hosted Kubernetesͱ͸Կ͔
   • ͲͷΑ͏ͳํ๏Ͱߏங͢Δ͔
   • ΍ͬͯΈͯಘΒΕͨ͜ͱɺࠓޙͷ՝୊ʹ͍ͭͯ
   ࠓ೔͓࿩͢Δ಺༰
   

   View Slide

4. • OpenStackΛ༻͍ͨPrivateCloudΛӡ༻͍ͯ͠Δ
   • NyahʢIaaSɺLBaaSʣ
   • AWSͷΑ͏ͳPublicCloud΋࢖༻
   • ϋΠϒϦουΫϥ΢υߏ੒ͷαʔϏε΋͋Δ
   !4
   લఏ
   ϖύϘͷϓϥΠϕʔτΫϥ΢υ “Nyah” ͦͷޙ by @tnmt
   https://speakerdeck.com/tnmt/pepabos-privatecloud-nyah-after-that

   minne ͰͷϋΠϒϦουΫϥ΢υӡ༻ by @shiro16
   https://speakerdeck.com/shiro16/minne-defalsehaiburitudokuraudoyun-yong
   

   View Slide

5. • PublicCloudʹ͋ΔɺΑΓޮ཰తʹ։ൃͰ͖ΔϓϥοτϑΥʔϜ͕
   ͳ͍
   !5
   ΑΓޮ཰తʹαʔϏε։ൃɾӡ༻Λ͢Δʹ͸
   IUUQTDMPVEHPPHMFDPNCMPHQSPEVDUTHDQUJNFUPIFMMPXPSMEWNTWTDPOUBJOFSTWTQBBTWTGBBT
   

   View Slide

6. • ੍໿Λڧ͘͢Δ͜ͱͰΑΓߴ͍ੜ࢈ੑΛಘΔ
   • Կ͔ΒखΛ͚ͭΔ͔
   • => ·ͣ͸CaaS͕Α͔Ζ͏ɻͳΒKubernetes͔ͳʁ
   • ΑΓखܰʹ࢖͑ΔΠϯϑϥΛ໨ࢦͨ͢ΊɺIaaS෦෼ΛӅṭ͍ͨ͠
   • PaaS΍FaaS͸CaaS্ʹߏஙͰ͖Δ͕ٯ͸೉͍͠
   !6
   $BB41BB4'BB4Λఏڙ͢Δ͜ͱͰੜ࢈ੑ61ˢ
   

   View Slide

7. • CI༻్͕த৺ͰɺαʔϏε͸ಈ͔͍ͯ͠ͳ͍
   • SwarmͰ͸ବ໨ͳͷ͔ʁ
   • Kubernetesͷ࣋ͭαʔϏεσΟεΧόϦػೳ΍ӡ༻ࣗಈԽɺ֦ுੑͱ
   ͍ͬͨར఺Λ׆༻͍ͨ͠ؾ͕࣋ͪڧ͍
   !7
   ࣮͸ࣾ಺޲͚ʹ%PDLFS4XBSNΛఏڙத
   

   View Slide

8. • ͔͠͠ɺKubernetesͷߏஙɾӡ༻͸ۃྗলྗԽ͍ͨ͠
   • Kubernetesͷӡ༻Ͱফ໣͢Δͷ͸ຊ຤స౗
   • kubelet / apiserver / controller-manager / scheduler / etcd / etc…
   • 1Ϋϥελͱ͸ݶΒͳ͍
   • શͯࣗ෼ͨͪͰ؅ཧ͢Δͷ͸େม
   • ઃఆมߋɺόʔδϣϯΞοϓ
   • ৑௕Խ
   • ো֐࣌ͷ෮چ࡞ۀ
   !8
   Ͳ͏΍ͬͯߏஙɾӡ༻͢Δ͔
   

   View Slide

9. • Ϛωʔδυ Self-Hosted Kubernetes͸Ͳ͏ͩΖ͏ʁ
   • Kubernetesࣗ਎ʹࣗ෼ࣗ਎Λ؅ཧͤ͞Δߟ͑ํ
   • Kubernetesͷ࣋ͭߴ౓ͳػೳΛ࢖͏͜ͱ͕Ͱ͖Δ
   • Deployment, Auto Healing, etc…
   • Kubernetes as a Serviceͱ͸ҧ͏ͷ͔
   • ͪ͜Β͸ɺෳ਺ͷΫϥελΛKubernetesͰ؅ཧ͢Δ΋ͷ
   !9
   ,VCFSOFUFTͷ؅ཧΛΑΓָʹ͢Δʹ͸
   

   View Slide

10. 4FMG)PTUFE,VCFSOFUFTͱ͸ʁ
    

    View Slide

11. !11
    1. Small Dependencies
    2. Deployment consistency
    3. Introspection
    4. Cluster Upgrades
    5. Easier Highly-Available Configurations
    1SPQPTBM4FMGIPTUFE$POUSPM1MBOF
    IUUQTHJUIVCDPNLVCFSOFUFTDPNNVOJUZCMPCNBTUFSDPOUSJCVUPSTEFTJHOQSPQPTBMTDMVTUFSMJGFDZDMFTFMGIPTUFELVCFSOFUFTNE
    

    View Slide

12. • kubeletͰ֤ίϯϙʔωϯτΛಈ͔͢͜ͱͰɺϗετ্ʹඞཁͳί
    ϯϙʔωϯτΛݮΒ͢͜ͱ͕Ͱ͖Δ
    • KubernetesͷηοτΞοϓʹ͓͚Δෳࡶͷղফʹܨ͕Δ
    • ࠓճߏஙͨ͠؀ڥͷ৔߹
    • DockerͱkubeletͷΈsystemdͰ؅ཧ
    • ଞͷίϯϙʔωϯτ͸kubelet؅ཧԽ
    !12
    4NBMM%FQFOEFODJFT
    

    View Slide

13. • σΟεΫ্ͷϑΝΠϧ΍SSHΛ༻͍ͨઃఆ؅ཧΛݮΒͤΔ
    • ϗετOS΁ͷґଘΛݮΒ͠ɺσϓϩΠͷҰ؏ੑΛಘΔ
    • ࠓճߏஙͨ͠؀ڥͷ৔߹
    • ϑΝΠϧͱͯ͠ஔ͘ඞཁ͕͋ΔϚχϑΣετΛSSHͰ഑ஔ
    • ଞ͸Kubernetes APIܦ༝
    !13
    %FQMPZNFOUDPOTJTUFODZ
    

    View Slide

14. • kubectl logsͷΑ͏ͳKubenetes APIΛ࢖ͬͯίϯϙʔωϯτͷσ
    όοά΍ௐ͕ࠪͰ͖Δ
    • ͜Ε·Ͱͩͱɺ֤αʔόʔʹೖΔorू໿͢ΔͨΊͷπʔϧΛೖΕΔ
    ඞཁ͕͋ͬͨ
    • ϩάऩूͷ࢓૊Έ΋Kubernetes্ʹ࡞Ε͹1΍2ͷΑ͏ͳԸܙΛड͚ΒΕ
    Δ
    !14
    *OUSPTQFDUJPO
    

    View Slide

15. • Kubernetes APIܦ༝ͰΫϥελʔͷΞοϓάϨʔυ͕Ͱ͖Δ
    • Kubernetes্Ͱಈ͔͢αʔϏεͱಉ͡ํ๏Λ࢖͑Δ
    • RollingUpdate΍RollbackɺศརͰ͢Ͷ
    !15
    $MVTUFS6QHSBEFT
    

    View Slide

16. • ؂ࢹ΍HAߏ੒ͷߏஙΛ֎෦πʔϧͳ͠Ͱ࣮ݱͰ͖Δ
    • Kubernetes্Ͱಈ͔͢αʔϏεͱಉ͡ํ๏Ͱ࣮ݱͰ͖Δͷ͕Good
    • ಉ͡ςΫχοΫΛ࢖͑Δͷ͸࠷ߴ
    !16
    &BTJFS)JHIMZ"WBJMBCMF$POpHVSBUJPOT
    

    View Slide

17. !17
    ηϧϑϗετ͢Δൣғʹ͍ͭͯ
    0
    1
    2
    3
    4
    kubelet
    etcd
    apiserver
    scheduler
    controller-
    manager
    proxy
    dns addons
    2-4 cluster
    1-4 cluster
    

    View Slide

18. Ͳ͏΍ͬͯ࡞Ζ͏ʁ
    

    View Slide

19. • bootkubeͷΑ͏ͳطଘͷߏஙπʔϧΛ࢖ͬͯ΋΋ͪΖΜΑ͍
    • ͜Ε·Ͱ࢖͖ͬͯͨπʔϧͰͲ͜·Ͱ΍ΕΔͷ͔ɺࢼͯ͠ΈΑ͏
    • KubernetesࣗମͷཧղΛਂΊ͍ͨ
    • Packer / Terraform / AnsibleΛ࢖͍ͬͯΕ͹ձࣾͷଞϝϯόʔ΋ѻ
    ͍΍͍͢ͷͰ͸ɺͱߟ͑ͨ
    !19
    ࠓճ͸ࣗ࡞͢Δ͜ͱʹͨ͠
    

    View Slide

20. ࢓૊Έ
    

    View Slide

21. • 2-4Ϋϥελ
    • جຊతʹ Kubernetes on Kubernetes on OpenStack (Ծ) ʹ४ڌ
    • ΊͬͪΌࢀߟʹ͍͍͍ͤͯͨͩͯ͞·͢ :kansya:
    !21
    ࠓճνϟϨϯδͨ͠4FMG)PTUFEʹ͍ͭͯ
    kubelet Systemd؅ཧԼ
    etcd StaticPodΛ༻͍ͯkubelet؅ཧԼ
    apiserver

    controller-manager
    scheduler
    Kubernetes APIΛ༻͍ͯKubernetes؅ཧԼ
    Kubernetes on Kubernetes on OpenStack (Ծ) by @yuanying
    https://docs.google.com/presentation/d/1VKk89MaNkGRSlpBsOOHJt8cLD6mpZ5V55GEJqIDu2Sk/edit#slide=id.g28eb8071bc_0_1119
    

    View Slide

22. • Packer
    • ϕʔεΠϝʔδʹDocker΍kubeletɺͦͷଞશϊʔυͰ࢖͏ιϑτ΢ΣΞͷΠϯ
    ετʔϧʹ࢖༻ɻ
    • Ϋϥελ࡞੒࣌ɺϊʔυ௥Ճ࣌ͷ࣌ؒ୹ॖΛݟࠐΜͩ
    • Terraform
    • αʔόʔɺূ໌ॻൃߦʹ࢖༻
    • OpenStack Provider, TLS Provider
    • Ansible
    • Ϋϥελʔߏஙɺߋ৽ʹ࢖༻
    • Dynamic InventoryΛ࢖͏ͱTerraform͔Βద༻ઌIPΞυϨεΛऔಘͰ͖ͯศར
    !22
    1BDLFS5FSSBGPSN"OTJCMF͸Ͳ͜Ͱ࢖͔ͬͨ
    

    View Slide

23. 1. Packer / AnsibleͰϕʔεΠϝʔδΛ࡞੒ʢDocker, kubeletʣ
    2. TerraformͰαʔόʔΛىಈ
    3. AnsibleͰBootstrap ClusterΛىಈ
    4. AnsibleͰBootstrap Cluster͔ΒSelf-Hosted Cluster΁੾Γସ͑Δ
    !23
    ߏஙͷྲྀΕ
    

    View Slide

24. 1. Packer / AnsibleͰϕʔεΠϝʔδΛ࡞੒ʢDocker, kubeletʣ
    2. TerraformͰαʔόʔΛىಈ
    3. AnsibleͰBootstrap ClusterΛىಈ
    4. AnsibleͰBootstrap Cluster͔ΒSelf-Hosted Cluster΁੾Γସ͑Δ
    !24
    ߏஙͷྲྀΕ
    

    View Slide

25. • શαʔόʔ͕ඞཁͱ͢ΔίϯϙʔωϯτΛ༧ΊΠϯετʔϧ
    • Docker, kubeletͳͲ
    • Ϋϥελʔߏஙɺϊʔυ௥Ճ࣌ͷ࣌ؒ୹ॖ͕໨త
    • Πϝʔδߋ৽ͰDockerɾkubeletͷೖΕସ͑Λߦ͏৔߹޻෉͕ඞཁ
    !25
    1BDLFS"OTJCMFͰϕʔεΠϝʔδΛ࡞੒
    Image
    Server
    Packer
    Ansible
    Install Docker, kubelet, etc
    Launch Server
    Save Base Image
    

    View Slide

26. 1. Packer / AnsibleͰϕʔεΠϝʔδΛ࡞੒ʢDocker, kubeletʣ
    2. TerraformͰαʔόʔΛىಈ
    3. AnsibleͰBootstrap ClusterΛىಈ
    4. AnsibleͰBootstrap Cluster͔ΒSelf-Hosted Cluster΁੾Γସ͑Δ
    !26
    ߏஙͷྲྀΕ
    

    View Slide

27. 5FSSBGPSNͰαʔόʔΛىಈ͢Δ
    Docker
    kubelet
    master
    Docker
    kubelet
    master
    Docker
    kubelet
    master
    Docker
    kubelet
    node
    Docker
    kubelet
    node
    Terraform
    Launch servers
    certificates and private key
    

    View Slide

28. 5FSSBGPSNͰαʔόʔΛىಈ͢Δ
    Docker
    kubelet
    master
    Docker
    kubelet
    master
    Docker
    kubelet
    master
    Docker
    kubelet
    node
    Docker
    kubelet
    node
    Terraform
    Launch servers
    certificates and private key
    • ֤छূ໌ॻɾൿີ伴͸͜ͷλΠϛϯάͰੜ੒
    • TLS Provider
    • repl.infoɿTerraformͷTLS ProviderΛࢼ͢
    • αʔόʔىಈ࣌ʹcloud-configͰ౉ͯ͠഑ஔ
    • tfstateʹฏจͰอଘ͞ΕΔ(!)ͨΊɺࠓޙํࣜΛม͑Δ༧ఆ
    

    View Slide

29. 1. Packer / AnsibleͰϕʔεΠϝʔδΛ࡞੒ʢDocker, kubeletʣ
    2. TerraformͰαʔόʔΛىಈ
    3. AnsibleͰBootstrap ClusterΛىಈ
    4. AnsibleͰBootstrap Cluster͔ΒSelf-Hosted Cluster΁੾Γସ͑Δ
    !29
    ߏஙͷྲྀΕ
    

    View Slide

30. "OTJCMFͰ#PPUTUSBQ$MVTUFSΛىಈ͢Δ
    Docker
    kubelet
    etcd
    master
    Docker
    kubelet
    etcd
    master
    Docker
    kubelet
    etcd
    master
    Docker
    kubelet
    node
    Docker
    kubelet
    node
    Ansible
    Launch etcd cluster
    Static Pod manifest files
    

    View Slide

31. "OTJCMFͰ#PPUTUSBQ$MVTUFSΛىಈ͢Δ
    Docker
    kubelet
    etcd
    master
    Docker
    kubelet
    etcd
    master
    Docker
    kubelet
    etcd
    master
    Docker
    kubelet
    node
    Docker
    kubelet
    node
    • etcd clusterΛߏங͢Δ
    • masterʹStatic Pod༻ϚχϑΣετϑΝΠϧΛஔ͘
    • Static Pod ?
    Ansible
    Launch etcd cluster
    Static Pod manifest via files
    

    View Slide

32. • kubelet͸ࢦఆͨ͠σΟϨΫτϦΛ؂ࢹͯ͠ɺஔ͔ΕͨϚχϑΣετ
    ϑΝΠϧΛݩʹPodΛىಈ͢ΔػೳΛ࣋ͭ
    • ͜ΕΛStatic PodͱݺͿ
    !32
    4UBUJD1PEʁʁʁ
    kubelet /etc/kubernetes/manifests
    XBUDI
    ϚχϑΣετϑΝΠϧ͕ஔ͔ΕͨΒ1PEΛ࡞੒

    ফ͞ΕͨΒ1PEΛ࡟আ
    Docker IUUQTLVCFSOFUFTJPEPDTUBTLTBENJOJTUFSDMVTUFSTUBUJDQPE
    ŠQPENBOJGFTUQBUI
    ŠNBOJGFTUVSM
    http://pod.example.com/
    

    View Slide

33. "OTJCMFͰ#PPUTUSBQ$MVTUFSΛىಈ͢Δ
    Docker
    kubelet
    etcd
    bootstrap apiserver
    bootstrap controller-manager
    bootstrap scheduler
    master
    Docker
    kubelet
    etcd
    master
    Docker
    kubelet
    etcd
    master
    Docker
    kubelet
    node
    Docker
    kubelet
    node
    Ansible
    Launch bootstrap cluster
    StaticPod manifest via files
    apiserverɿ໊લͷ௨ΓɺKubernetes APIͷΤϯυϙΠϯτ
    schedulerɿPodΛNodeʹׂΓ౰ͯΔ
    controller-managerɿReplicaͳͲΛίϯτϩʔϧ͢Δ
    

    View Slide

34. "OTJCMFͰ#PPUTUSBQ$MVTUFSΛىಈ͢Δ
    Docker
    kubelet
    etcd
    bootstrap apiserver
    bootstrap controller-0manager
    bootstrap scheduler
    master
    Docker
    kubelet
    etcd
    master
    Docker
    kubelet
    etcd
    master
    Docker
    kubelet
    node
    Docker
    kubelet
    node
    • apiserver, controller-manager, schedulerΛStatic PodͰىಈ
    • ͜ͷ࣌఺ͰKubernetes API͕ར༻ՄೳͱͳΔ
    • ֤αʔόʔͷkubelet͕apiserver΁ొ࿥ɺAPIͰݟ͑ΔΑ͏ʹͳΔ
    • controller-manager, scheduler΋ಈ͍͍ͯΔͷͰDeploymentͳͲ΋࢖͑Δ
    Ansible
    Launch bootstrap cluster
    StaticPod manifest via files
    

    View Slide

35. "OTJCMFͰ#PPUTUSBQ$MVTUFSΛىಈ͢Δ
    Docker
    kubelet
    etcd
    bootstrap apiserver
    bootstrap controller-manager
    bootstrap scheduler
    master
    Docker
    kubelet
    etcd
    master
    Docker
    kubelet
    etcd
    master
    Docker
    kubelet
    node
    Docker
    kubelet
    node
    register node
    Ansible
    Launch bootstrap cluster
    StaticPod manifest via files
    

    View Slide

36. 1. Packer / AnsibleͰϕʔεΠϝʔδΛ࡞੒ʢDocker, kubeletʣ
    2. TerraformͰαʔόʔΛىಈ
    3. AnsibleͰBootstrap ClusterΛىಈ
    4. AnsibleͰBootstrap Cluster͔ΒSelf-Hosted Cluster΁੾Γ
    ସ͑Δ
    !36
    ߏஙͷྲྀΕ
    

    View Slide

37. 4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ
    Docker
    kubelet
    etcd
    master
    Docker
    kubelet
    etcd
    master
    Docker
    kubelet
    etcd
    master
    Docker
    kubelet
    node
    Docker
    kubelet
    node
    bootstrap apiserver
    bootstrap controller-manager
    bootstrap scheduler
    Ansible
    Launch self-hosted cluster
    Manifest files via Kubernetes API
    

    View Slide

38. 4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ
    Docker
    kubelet
    etcd
    kube-proxy
    master
    Docker
    kubelet
    etcd
    kube-proxy
    master
    Docker
    kubelet
    etcd
    kube-proxy
    master
    Docker
    kubelet
    kube-proxy
    node
    Docker
    kubelet
    kube-proxy
    node
    bootstrap apiserver
    bootstrap controller-manager
    bootstrap scheduler
    Ansible
    Launch self-hosted cluster
    Manifest files via Kubernetes API
    

    View Slide

39. 4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ
    Docker
    kubelet
    etcd
    kube-proxy
    flannel
    master
    Docker
    kubelet
    etcd
    kube-proxy
    flannel
    master
    Docker
    kubelet
    etcd
    kube-proxy
    flannel
    master
    Docker
    kubelet
    kube-proxy
    flannel
    node
    Docker
    kubelet
    kube-proxy
    flannel
    node
    bootstrap apiserver
    bootstrap controller-manager
    bootstrap scheduler
    Ansible
    Launch self-hosted cluster
    Manifest files via Kubernetes API
    

    View Slide

40. 4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    master
    Docker
    kubelet
    kube-proxy
    flannel
    node
    Docker
    kubelet
    kube-proxy
    flannel
    node
    bootstrap apiserver
    bootstrap controller-manager
    bootstrap scheduler
    Ansible
    Launch self-hosted cluster
    Manifest files via Kubernetes API
    NodeAffinityΛ࢖ͬͯmasterͷΈىಈ
    

    View Slide

41. 4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    scheduler
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    scheduler
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    master
    Docker
    kubelet
    kube-proxy
    flannel
    node
    Docker
    kubelet
    kube-proxy
    flannel
    node
    bootstrap apiserver
    bootstrap controller-manager
    bootstrap scheduler
    Ansible
    Launch self-hosted cluster
    Manifest files via Kubernetes API
    

    View Slide

42. 4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    scheduler
    controller-manager
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    scheduler
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    controller-manager
    master
    Docker
    kubelet
    kube-proxy
    flannel
    node
    Docker
    kubelet
    kube-proxy
    flannel
    node
    bootstrap apiserver
    bootstrap controller-manager
    bootstrap scheduler
    Ansible
    Launch self-hosted cluster
    Manifest files via Kubernetes API
    NodeAffinityΛ࢖ͬͯmasterͷΈىಈ
    

    View Slide

43. 4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    scheduler
    controller-manager
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    scheduler
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    controller-manager
    master
    Docker
    kubelet
    kube-proxy
    flannel
    node
    Docker
    kubelet
    kube-proxy
    flannel
    node
    bootstrap apiserver
    bootstrap controller-manager
    bootstrap scheduler
    • DaemonSetɺDeploymentͰىಈ
    Ansible
    Launch self-hosted cluster
    Manifest files via Kubernetes API
    

    View Slide

44. 4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    scheduler
    controller-manager
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    scheduler
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    controller-manager
    master
    Docker
    kubelet
    kube-proxy
    flannel
    node
    Docker
    kubelet
    kube-proxy
    flannel
    node
    bootstrap apiserver
    bootstrap controller-manager
    bootstrap scheduler
    • bootstrap apiserverͱapiserverͰ࢖༻ϙʔτ͕
    িಥͦ͠͏
    • ࣮ࡍɺ͜ͷapiserver͸ىಈʹࣦഊ͢Δ
    • ͔͠͠ɺ͜ͷޙແࣄʹىಈͯ͘͠Δ
    Ansible
    Launch self-hosted cluster
    Manifest files via Kubernetes API
    

    View Slide

45. #PPUTUSBQ$MVTUFS͔Β4FMG)PTUFE$MVTUFS΁੾Γସ͑Δ
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    scheduler
    controller-manager
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    scheduler
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    controller-manager
    master
    Docker
    kubelet
    kube-proxy
    flannel
    node
    Docker
    kubelet
    kube-proxy
    flannel
    node
    bootstrap apiserver
    bootstrap controller-manager
    bootstrap scheduler
    Ansible
    Delete bootstrap cluster
    Delete StaticPod manifest files
    

    View Slide

46. #PPUTUSBQ$MVTUFS͔Β4FMG)PTUFE$MVTUFS΁੾Γସ͑Δ
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    scheduler
    controller-manager
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    scheduler
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    controller-manager
    master
    Docker
    kubelet
    kube-proxy
    flannel
    node
    Docker
    kubelet
    kube-proxy
    flannel
    node
    Ansible
    Delete bootstrap cluster
    Delete StaticPod manifest files
    

    View Slide

47. #PPUTUSBQ$MVTUFS͔Β4FMG)PTUFE$MVTUFS΁੾Γସ͑Δ
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    scheduler
    controller-manager
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    scheduler
    master
    Docker
    kubelet
    etcd
    apiserver
    kube-proxy
    flannel
    controller-manager
    master
    Docker
    kubelet
    kube-proxy
    flannel
    node
    Docker
    kubelet
    kube-proxy
    flannel
    node
    • bootstrap apiserverͷϓϩηε͕͍ͳ͘ͳͬͨ
    ͷͰɺapiserver͸ىಈϦτϥΠʹΑΓىಈͯ͠
    ͘Δ
    Ansible
    Delete bootstrap cluster
    Delete StaticPod manifest files
    

    View Slide

48. !48
    )JHI"WBJMBCJMJUZ
    etcd StaticPodΛ࢖͍ɺmaster3୆ʙͰΫϥελԽ
    apiserver
    DaemonSetΛ࢖͍ɺmasterશ୆ͰՔಇ
    keepalived + haproxyͰ৑௕Խ
    (Ϋϥελ֎ʹLBaaSͰLoadBalancerΛઃஔ͢Δͷ΋Մʣ
    controller-manager
    scheduler
    DeploymentΛ࢖͍ɺmasterෳ਺୆ͰՔಇ
    

    View Slide

49. ಘΒΕͨ΋ͷɺࠓޙͷ՝୊
    

    View Slide

50. • ཧ۶Ͱ͸෼͔ͬͯ΋ɺखΛಈ͔ͯ͠ΈΔ͜ͱ͸େ੾
    • Bootstrap Cluster͔ΒSelf-Hosted Clusterʹ੾Γସ͑Δ͜ͱ͕Ͱ͖
    ͨ࣌͸ײಈͨ͠
    • Ͳ͜·Ͱ͸؆୯ʹ࣮૷Ͱ͖ͯɺͲ͔͜Β೉͘͠ͳΔͷ͔΋஌Δ͜ͱ
    ͕Ͱ͖ͨ
    !50
    ࣗ෼ࣗ਎Λ؅ཧ͢Δɺͱ͍͏ํ๏ʹ͍ͭͯͷཧղ
    

    View Slide

51. • Ϋϥελࣗମͷߏஙɾӡ༻
    • Կͷίϯϙʔωϯτ͕ඞཁ͔ɺԿͷ໾ׂΛ͍࣋ͬͯΔͷ͔
    • ֤ίϯϙʔωϯτͷΦϓγϣϯʹ͍ͭͯͷཧղʢΊͪΌଟ͍Ͱ͢ΑͶ…ʣ
    • Ϋϥελ্Ͱιϑτ΢ΣΞΛ͏·͘ಈ͔͢
    • controller-manager΍schedulerͳͲͷPodΛ࠷௿1ͭ͸ҡ͍࣋ͨ͠
    • PodͷೖΕସ͑΍NodeͷDrain
    !51
    Ϋϥελͷ্ɾԼ྆ํͷ஌ݟΛಉ࣌ʹಘΒΕͨ
    

    View Slide

52. • ݱࡏ͸Ansibleɿ1ϊʔυͣͭαʔϏεΞ΢τ&ߋ৽&αʔϏεΠϯ
    • ࠷ॳ͸Immutable Infrastructureͷߟ͑ํΛద༻Ͱ͖ͳ͍͔ߟ͑ͨ
    • ϕʔεΠϝʔδΛೖΕସ͑ͯߋ৽
    • TerraformͰ΍Ζ͏ͱ͢Δͱ݁ߏେม
    • masterϊʔυ΍etcdΛͲ͏͢Δʁ
    • ࠓޙ͸Custom ControllerΛ࢖ͬͯ؅ཧ͢Δ͜ͱΛݕ౼͍ͯ͠Δ
    !52
    %PDLFS΍LVCFMFUͷߋ৽ΛͲ͏͢Δ͔
    

    View Slide

53. • TerraformͰϊʔυͷ૿ݮ͸Մೳ
    • ݮΒ࣌͢ʹdrain͢Δඞཁ͕͋Δ
    • Kubernetes؅ཧԼʹஔ͚ΔͱΑΓΑ͘Ͱ͖ͦ͏
    • Kubernetes APIܦ༝ͰεέʔϧΠϯɾΞ΢τ
    • HPAͷΑ͏ͳΦʔτεέʔϧػߏͱ૊Έ߹ΘͤΔ͜ͱ΋Ͱ͖ͦ͏
    • ͪ͜Β΋Custom ControllerԽ͢Δ͜ͱΛݕ౼
    !53
    ϊʔυͷ૿ݮΛͲ͏͢Δ͔
    

    View Slide

54. ·ͱΊ
    

    View Slide

55. • ൚༻తͳπʔϧΛ࢖ͬͯSelf-Hosted KubernetesͷߏஙΛߦͬͨ
    • ͔ͳΓ͍͍ײ͡ʹಈ࡞͢ΔΫϥελʔΛ࡞Δ͜ͱ͕Ͱ͖ͨ
    • ͔͠͠ɺݶք΋ײͨ͡
    • ϊʔυͷೖΕସ͑
    • ϊʔυ಺ίϯϙʔωϯτͷߋ৽
    • ࠓޙ͸Custom ControllerΛ׆༻ͯ͠ΑΓޮ཰తͳ؅ཧΛ࣮ݱ͍ͨ͠
    !55
    ·ͱΊ
    

    View Slide

56. View Slide