Kuboで快適k8s運用

第35回 PaaS勉強会で発表した資料です。 Kubernetes + BOSHなKuboを紹介します。

※ なお、このスライドの作成直後にCFの依存なく単体でKubernetesのデプロイが可能になりました

1. KuboͰշద
 Kubernetesӡ༻
2. Kazuto Kusama @jacopen
3. ࠷ۙస৬͠·ͨ͠
5. ࢖ͬͯ·͔͢ʁ
6. Ͳ͏΍ͬͯӡ༻ͯ͠·͔͢ʁ • GKE͔ͭͬͯΔ • Azure Container Service͔ͭͬͯΔ • ࣗલͰߏஙͯ͠Δ
7. ࣗલͰߏஙɺ݁ߏେม͡Όͳ͍Ͱ͔͢ʁ • ͦ΋ͦ΋Ͳ͏΍ͬͯσϓϩΠ͢Δ͔ • kubeadm • Ansible • ͦͷଞʁ • ؂ࢹ͸ʁ • εέʔϧ͸ʁ • Ξοϓσʔτ͸ʁ
8. ࣗલͰ΍Γͨ͘ͳ͍
9. ͳΜͰͭΒ͍ͷ͔ • ཱࣗ෼ࢄܕͷΞʔΩςΫνϟ • ߏ੒͢Δίϯϙʔωϯτ͕ଟ͍ • σϓϩΠ͸·͍͍ͩ • ͲΕ͔ΒΞοϓσʔτ͢ΔʁͲ͏͍͏खॱͰΞοϓσʔτ͢Δʁ ԿΛόοΫΞοϓ͢Ε͹ྑ͍ʁ Ͳ͜Λ؂ࢹ͢Ε͹ྑ͍ʁ
10. ͳΜͰͭΒ͍ͷ͔ • ཱࣗ෼ࢄܕͷΞʔΩςΫνϟ • ߏ੒͢Δίϯϙʔωϯτ͕ଟ͍ • σϓϩΠ͸·͍͍ͩ • ͲΕ͔ΒΞοϓσʔτ͢ΔʁͲ͏͍͏खॱͰΞοϓσʔτ͢Δʁ ԿΛόοΫΞοϓ͢Ε͹ྑ͍ʁ Ͳ͜Λ؂ࢹ͢Ε͹ྑ͍ʁ k8sʹؔΘΒͣେن໛ͳج൫ͷӡ༻͸ͭΒ͍
11. ղܾࡦ͸ͳ͍ͷ͔
12. ͋Γ·͢ʂ
14. BOSH
15. BOSH
16. BOSHͱ͸ • ϦϦʔε؅ཧ/σϓϩΠ/ϥΠϑαΠΫϧϚωδϝϯτ/ϞχλϦ ϯάΛߦ͏ͨΊͷ࢓૊Έ • Cloud Foundry΍ɺपลαʔϏεͷσϓϩΠʹར༻͞Ε͍ͯΔ
17. Α͋͘ΔσϓϩΠπʔϧ VM OS VM OS VM OS IaaS app app app
18. VM OS VM OS VM OS IaaS app app app • σϓϩΠ͢Δઌ͸طʹଘࡏ͢Δલఏ • σϓϩΠͨ͠ΒͦΕͰऴΘΓ
19. VM OS VM OS VM OS IaaS kubelet docker kube-proxy kubelet docker kube-proxy api etcd τϥϒϧͰϓϩηε͕μ΢ϯͨ͠Βʁ
20. VM OS VM OS VM OS IaaS kubelet docker kube-proxy kubelet docker kube-proxy api etcd τϥϒϧͰVM͕μ΢ϯͨ͠Βʁ
21. VM OS VM OS VM OS IaaS kubelet docker kube-proxy kubelet docker kube-proxy api etcd ੬ऑੑ͕൑໌ͨ͠Βʁ
22. VM OS VM OS VM OS IaaS kubelet docker kube-proxy kubelet docker kube-proxy api etcd
24. Throw away the duct
 tape!
25. BOSH
26. BOSH
27. BOSH Stemcell Stemcell - VMͷΠϝʔδϑΝΠϧ
28. VM OS VM OS VM OS Stemcell
29. VM OS VM OS VM OS release deployment release - ΞϓϦέʔγϣϯͷόΠφϦ΍ίϯϑΟάҰࣜ deployment - release΍stemcellΛͲ͜ʹͲΕ͚ͩσϓϩΠ ͢Δ͔ͷࢦࣔॻ
30. VM OS VM OS VM OS release deployment kubelet docker kube-proxy kubelet docker kube-proxy api etcd
31. VM OS VM OS VM OS kubelet docker kube-proxy kubelet docker kube-proxy api etcd σϓϩΠޙ͸ɺbosh͕ϓϩηεɺVMͷ؂ࢹ͓Αͼ ϩάऩू
32. VM OS VM OS VM OS kubelet docker kube-proxy kubelet docker kube-proxy api etcd ΋͠VM͕ಥવࢮͯ͠΋ Σ
33. VM OS VM OS VM OS kubelet docker kube-proxy kubelet docker kube-proxy api etcd BOSH͕௚͢
34. VM OS VM OS VM OS release deployment VM VM ΋͠εέʔϧΞ΢τͨ͘͠ͳͬͯ΋
35. VM OS VM OS VM OS release deployment VM VM deploymentʹ૿΍͍ͨ͠෼͚ͩॻ͍ͯɺ BOSHʹ৯ΘͤΕ͹εέʔϧ׬ྃ
36. Day1 • ࠷ॳͷσϓϩΠ • AnsibleͰ΋ChefͰ΋PuppetͰ΋্ख͘΍ΕΔ ӡ༻ͷ2ϑΣʔζ Day2 • ϞχλϦϯά • Ξοϓσʔτ • ϥΠϑαΠΫϧϚωδϝϯτ • όοΫΞοϓ
37. Day1 • ࠷ॳͷσϓϩΠ • AnsibleͰ΋ChefͰ΋PuppetͰ΋্ख͘΍ΕΔ ӡ༻ͷ2ϑΣʔζ Day2 • ϞχλϦϯά • Ξοϓσʔτ • ϥΠϑαΠΫϧϚωδϝϯτ • όοΫΞοϓ ӡ༻͸Day2໋͕ͦ͜ɻ ͚ͩͲɺͳ͔ͥ͜͜ΛμΫτςʔϓʹ ͍ͯ͠Δέʔε͕ଟ͍
38. Day2Λҙࣝ͠ͳ͍ͱɾɾɾ • ӡ༻ίετ͕O(n)Ͱ૿େɹ( Լख͢Ε͹O(n^2)ͷέʔε΋ŋŋŋ ) • AnsibleͰ͍͍έʔε • খن໛ͳΞϓϦ • 1ճσϓϩΠͨ͠ΒҎ߱͋·ΓΞοϓσʔτ͠ͳ͍ΞϓϦ • Day2ͷҙ͕ࣝඞਢͳέʔε • େن໛ɺ෼ࢄܕ • ͭ·Γ͸Cloud Foundryͱ͔BOSHͱ͔
39. Day2ӡ༻͠·͠ΐ͏
40. ͔ͭͯCF΍ͬͯͨਓ Ͱ΋ɺBOSH΋େ֓ͭΒ͘ͳ͍ʁ ֶशίετ΍͹͍
41. ͔ͭͯͷBOSH
42. - name: doppler azs: - z1 instances: 1 vm_type: small stemcell: default networks: - name: default jobs: - name: consul_agent release: consul consumes: consul: {from: consul_link} consul_common: nil consul_server: nil consul_client: nil properties: consul: agent: services: doppler: name: doppler - name: doppler release: loggregator properties: doppler: etcd: client_cert: "((etcd_client.certificate))" client_key: "((etcd_client.private_key))" loggregator: tls: ca_cert: "((loggregator_tls_doppler.ca))" doppler: cert: "((loggregator_tls_doppler.certificate))" key: "((loggregator_tls_doppler.private_key))" etcd: require_ssl: true ca_cert: "((etcd_server.ca))" machines: - cf-etcd.service.cf.internal doppler_endpoint: shared_secret: "((dropsonde_shared_secret))" - name: syslog_drain_binder release: loggregator properties: loggregator: tls: key: "((loggregator_tls_syslogdrainbinder.private_key))" [437/1737] etcd: require_ssl: true ca_cert: "((etcd_server.ca))" machines: - cf-etcd.service.cf.internal syslog_drain_binder: etcd: client_cert: "((etcd_client.certificate))" client_key: "((etcd_client.private_key))" system_domain: "((system_domain))" cc: mutual_tls: ca_cert: "((loggregator_tls_syslogdrainbinder.ca))" srv_api_uri: https://api.((system_domain)) ssl: *ssl - name: metron_agent release: loggregator properties: *metron_agent_properties - name: log-api azs: - z1 instances: 1 vm_type: small stemcell: default update: max_in_flight: 1 serial: true networks: ntroller: ͔ͭͯͷBOSH YAML - name: default jobs: - name: consul_agent release: consul consumes: consul: {from: consul_link} consul_common: nil consul_server: nil consul_client: nil properties: consul: agent: services: loggregator_trafficcontroller: {} - name: loggregator_trafficcontroller release: loggregator properties: traffic_co properties: traffic_controller: etcd: client_cert: "((etcd_client.certificate))" client_key: "((etcd_client.private_key))" uaa: url: https://uaa.((system_domain)) loggregator: tls: ca_cert: "((loggregator_tls_tc.ca))" trafficcontroller: cert: "((loggregator_tls_tc.certificate))" key: "((loggregator_tls_tc.private_key))" etcd: require_ssl: true ca_cert: "((etcd_server.ca))" machines: - cf-etcd.service.cf.internal uaa: client_secret: "((uaa_clients_doppler_secret))" system_domain: "((system_domain))" ssl: *ssl cc: srv_api_uri: "http://cloud-controller-ng.service.cf.internal:9022" - name: route_registrar release: routing properties: route_registrar: routes: - name: loggregator port: 8080 registration_interval: 20s uris: - loggregator.((system_domain)) - name: doppler port: 8081 registration_interval: 20s uris: - doppler.((system_domain)) - "*.doppler.((system_domain))" - name: metron_agent release: loggregator properties: *metron_agent_properties variables: - name: blobstore_admin_users_password type: password - name: blobstore_secure_link_secret - name: doppler azs: - z1 instances: 1 vm_type: small stemcell: default networks: - name: default jobs: - name: consul_agent release: consul consumes: consul: {from: consul_link} consul_common: nil consul_server: nil consul_client: nil properties: consul: agent: services: doppler: name: doppler - name: doppler release: loggregator properties: doppler: etcd: client_cert: "((etcd_client.certificate))" client_key: "((etcd_client.private_key))" loggregator: tls: ca_cert: "((loggregator_tls_doppler.ca))" doppler: cert: "((loggregator_tls_doppler.certificate))" key: "((loggregator_tls_doppler.private_key))" etcd: require_ssl: true ca_cert: "((etcd_server.ca))" machines: - cf-etcd.service.cf.internal doppler_endpoint: shared_secret: "((dropsonde_shared_secret))" - name: syslog_drain_binder release: loggregator properties: loggregator: tls: key: "((loggregator_tls_syslogdrainbinder.private_key))" [437/1737] etcd: require_ssl: true ca_cert: "((etcd_server.ca))" machines: - cf-etcd.service.cf.internal syslog_drain_binder: etcd: client_cert: "((etcd_client.certificate))" client_key: "((etcd_client.private_key))" system_domain: "((system_domain))" cc: mutual_tls: ca_cert: "((loggregator_tls_syslogdrainbinder.ca))" srv_api_uri: https://api.((system_domain)) ssl: *ssl - name: metron_agent release: loggregator properties: *metron_agent_properties - name: log-api azs: - z1 instances: 1 vm_type: small stemcell: default update: max_in_flight: 1 serial: true networks: ntroller: - name: default jobs: - name: consul_agent release: consul consumes: consul: {from: consul_link} consul_common: nil consul_server: nil consul_client: nil properties: consul: agent: services: loggregator_trafficcontroller: {} - name: loggregator_trafficcontroller release: loggregator properties: traffic_co properties: traffic_controller: etcd: client_cert: "((etcd_client.certificate))" client_key: "((etcd_client.private_key))" uaa: url: https://uaa.((system_domain)) loggregator: tls: ca_cert: "((loggregator_tls_tc.ca))" trafficcontroller: cert: "((loggregator_tls_tc.certificate))" key: "((loggregator_tls_tc.private_key))" etcd: require_ssl: true ca_cert: "((etcd_server.ca))" machines: - cf-etcd.service.cf.internal uaa: client_secret: "((uaa_clients_doppler_secret))" system_domain: "((system_domain))" ssl: *ssl cc: srv_api_uri: "http://cloud-controller-ng.service.cf.internal:9022" - name: route_registrar release: routing properties: route_registrar: routes: - name: loggregator port: 8080 registration_interval: 20s uris: - loggregator.((system_domain)) - name: doppler port: 8081 registration_interval: 20s uris: - doppler.((system_domain)) - "*.doppler.((system_domain))" - name: metron_agent release: loggregator properties: *metron_agent_properties variables: - name: blobstore_admin_users_password type: password - name: blobstore_secure_link_secret
43. ࠓͷBOSH YAML YAML YAML
44. ࠓͷBOSH • Manifest v2ʹͳ͍ͬͯͩͿݟ௨͕͠ྑ͘ͳͬͨ • BOSH CLI v2 (Go) ͕ࠓ݄GAʹ • ൿ఻ͷλϨʹͳΓ͕ͪͩͬͨdeployment manifest͕ͦͦ͜͜ ݟ௨͠Α͘ɻ
45. BOSH
46. Kubo
47. Kubo Kubernetes + BOSH
48. https://pivotal.io/kubo
49. CFFʹ΋Proposal͕ग़͍ͯ·͢ https://docs.google.com/document/d/1ZOFD5nBQC_vh9CmKHOGT7ugtNaJQ1t03jkLVsyDOH6k/edit?usp=sharing
50. Function Systems Event-driven microservices. Developers use a variety of abstractions today. App-centric Systems Full-featured applications. Container Systems Deep control over app packaging and runtime behavior. Data Services On Demand Services via interfaces. Infrastructure On-premise and/or public clouds. BOSH Cloud NativeͳΞϓϦέʔγϣϯͷ ։ൃɾӡ༻͸CF͕ϕετ
51. Function Systems Event-driven microservices. Developers use a variety of abstractions today. App-centric Systems Full-featured applications. Container Systems Deep control over app packaging and runtime behavior. Data Services On Demand Services via interfaces. Infrastructure On-premise and/or public clouds. BOSH CFͰ͸ରԠ͖͠Εͳ͍ϨΨγʔΞϓϦ΍ StatefulͳΞϓϦέʔγϣϯ͸k8sͰ
52. Function Systems Event-driven microservices. Developers use a variety of abstractions today. App-centric Systems Full-featured applications. Container Systems Deep control over app packaging and runtime behavior. Data Services On Demand Services via interfaces. Infrastructure On-premise and/or public clouds. BOSH ͦΕΒΛBOSHͰ؅ཧ
53. k8s API͸CFͷTCP Routerܦ༝
54. api-route-registrar͕ϧʔτΛ޿ࠂ
55. k8s Pod΁ͷHTTPΞΫηε͸Gorouterܦ༝
56. route-sync͕NATSܦ༝ͰϧʔτΛ޿ࠂ
57. CFͷRouting͸ࠓ·Ͱ௨Γ
58. CFͷRouting Layerͱ k8s΁ͷRouting͕౷߹͞Ε͍ͯΔ
59. k8s StandaloneͰͷߏங΋ͦͷ͏ͪग़དྷΔΑ͏ʹͳΔͱ͔
60. DEMO
61. ݱࡏͷεςʔλε ·ͩ·ͩΞϧϑΝ
62. Product Roadmap 2017 • K8 Parity: cloud packages for LB & Volumes, LB deployment type • Networking: app routes externally accessible, replace powerDNS • Persistence: stateful workloads for COTS data services • High Availability: Single-AZ & Multi-AZ / failover • Core: Migration to Etcd v3 • Rolling upgrades: Cluster upgrades w/Zero-downtime • Multi-IAAS: extend support for all BOSH-supported IAAS
 https://docs.google.com/presentation/d/1z-qGCcHLlPpz5LtS0TOcvBZIK4hUQ4GhB-jjQyHEF3c/edit?usp=sharing
63. ·ͱΊ • େن໛෼ࢄγεςϜΛAnsible΍Chef+IaaSͰ؅ཧ͢Δͷ͸
 ແཧήʔ • Day2Λҙࣝͨ͠ӡ༻Λߟ͑Α͏ɻBOSHͰָ͠·͠ΐ͏ɻ • CF+KuboͰCloud NativeͳΞϓϦέʔγϣϯΛޮ཰Α͘ӡ༻
64. Resources • https://github.com/pivotal-cf-experimental/kubo-deployment • https://github.com/pivotal-cf-experimental/kubo-release • http://bosh.io/docs