Kuboで快適k8s運用

第35回 PaaS勉強会で発表した資料です。 Kubernetes + BOSHなKuboを紹介します。

※ なお、このスライドの作成直後にCFの依存なく単体でKubernetesのデプロイが可能になりました

Transcript

1. KuboͰշద

   Kubernetesӡ༻
   

   View Slide

2. Kazuto Kusama
   @jacopen
   

   View Slide

3. ࠷ۙస৬͠·ͨ͠
   

   View Slide

4. View Slide

5. ࢖ͬͯ·͔͢ʁ
   

   View Slide

6. Ͳ͏΍ͬͯӡ༻ͯ͠·͔͢ʁ
   • GKE͔ͭͬͯΔ
   • Azure Container Service͔ͭͬͯΔ
   • ࣗલͰߏஙͯ͠Δ
   

   View Slide

7. ࣗલͰߏஙɺ݁ߏେม͡Όͳ͍Ͱ͔͢ʁ
   • ͦ΋ͦ΋Ͳ͏΍ͬͯσϓϩΠ͢Δ͔
   • kubeadm
   • Ansible
   • ͦͷଞʁ
   • ؂ࢹ͸ʁ
   • εέʔϧ͸ʁ
   • Ξοϓσʔτ͸ʁ
   

   View Slide

8. ࣗલͰ΍Γͨ͘ͳ͍
   

   View Slide

9. ͳΜͰͭΒ͍ͷ͔
   • ཱࣗ෼ࢄܕͷΞʔΩςΫνϟ
   • ߏ੒͢Δίϯϙʔωϯτ͕ଟ͍
   • σϓϩΠ͸·͍͍ͩ
   • ͲΕ͔ΒΞοϓσʔτ͢ΔʁͲ͏͍͏खॱͰΞοϓσʔτ͢Δʁ
   ԿΛόοΫΞοϓ͢Ε͹ྑ͍ʁ Ͳ͜Λ؂ࢹ͢Ε͹ྑ͍ʁ
   

   View Slide

10. ͳΜͰͭΒ͍ͷ͔
    • ཱࣗ෼ࢄܕͷΞʔΩςΫνϟ
    • ߏ੒͢Δίϯϙʔωϯτ͕ଟ͍
    • σϓϩΠ͸·͍͍ͩ
    • ͲΕ͔ΒΞοϓσʔτ͢ΔʁͲ͏͍͏खॱͰΞοϓσʔτ͢Δʁ
    ԿΛόοΫΞοϓ͢Ε͹ྑ͍ʁ Ͳ͜Λ؂ࢹ͢Ε͹ྑ͍ʁ
    k8sʹؔΘΒͣେن໛ͳج൫ͷӡ༻͸ͭΒ͍
    

    View Slide

11. ղܾࡦ͸ͳ͍ͷ͔
    

    View Slide

12. ͋Γ·͢ʂ
    

    View Slide

13. View Slide

14. BOSH
    

    View Slide

15. BOSH
    

    View Slide

16. BOSHͱ͸
    • ϦϦʔε؅ཧ/σϓϩΠ/ϥΠϑαΠΫϧϚωδϝϯτ/ϞχλϦ
    ϯάΛߦ͏ͨΊͷ࢓૊Έ
    • Cloud Foundry΍ɺपลαʔϏεͷσϓϩΠʹར༻͞Ε͍ͯΔ
    

    View Slide

17. Α͋͘ΔσϓϩΠπʔϧ
    VM
    OS
    VM
    OS
    VM
    OS
    IaaS
    app app app
    

    View Slide

18. VM
    OS
    VM
    OS
    VM
    OS
    IaaS
    app app app
    • σϓϩΠ͢Δઌ͸طʹଘࡏ͢Δલఏ
    • σϓϩΠͨ͠ΒͦΕͰऴΘΓ
    

    View Slide

19. VM
    OS
    VM
    OS
    VM
    OS
    IaaS
    kubelet
    docker
    kube-proxy
    kubelet
    docker
    kube-proxy
    api
    etcd
    τϥϒϧͰϓϩηε͕μ΢ϯͨ͠Βʁ
    

    View Slide

20. VM
    OS
    VM
    OS
    VM
    OS
    IaaS
    kubelet
    docker
    kube-proxy
    kubelet
    docker
    kube-proxy
    api
    etcd
    τϥϒϧͰVM͕μ΢ϯͨ͠Βʁ
    

    View Slide

21. VM
    OS
    VM
    OS
    VM
    OS
    IaaS
    kubelet
    docker
    kube-proxy
    kubelet
    docker
    kube-proxy
    api
    etcd
    ੬ऑੑ͕൑໌ͨ͠Βʁ
    

    View Slide

22. VM
    OS
    VM
    OS
    VM
    OS
    IaaS
    kubelet
    docker
    kube-proxy
    kubelet
    docker
    kube-proxy
    api
    etcd
    

    View Slide

23. View Slide

24. Throw away
    the duct

    tape!
    

    View Slide

25. BOSH
    

    View Slide

26. BOSH
    

    View Slide

27. BOSH
    Stemcell
    Stemcell - VMͷΠϝʔδϑΝΠϧ
    

    View Slide

28. VM
    OS
    VM
    OS
    VM
    OS
    Stemcell
    

    View Slide

29. VM
    OS
    VM
    OS
    VM
    OS
    release
    deployment
    release - ΞϓϦέʔγϣϯͷόΠφϦ΍ίϯϑΟάҰࣜ
    deployment - release΍stemcellΛͲ͜ʹͲΕ͚ͩσϓϩΠ
    ͢Δ͔ͷࢦࣔॻ
    

    View Slide

30. VM
    OS
    VM
    OS
    VM
    OS
    release
    deployment
    kubelet
    docker
    kube-proxy
    kubelet
    docker
    kube-proxy
    api
    etcd
    

    View Slide

31. VM
    OS
    VM
    OS
    VM
    OS
    kubelet
    docker
    kube-proxy
    kubelet
    docker
    kube-proxy
    api
    etcd
    σϓϩΠޙ͸ɺbosh͕ϓϩηεɺVMͷ؂ࢹ͓Αͼ
    ϩάऩू
    

    View Slide

32. VM
    OS
    VM
    OS
    VM
    OS
    kubelet
    docker
    kube-proxy
    kubelet
    docker
    kube-proxy
    api
    etcd
    ΋͠VM͕ಥવࢮͯ͠΋
    Σ
    

    View Slide

33. VM
    OS
    VM
    OS
    VM
    OS
    kubelet
    docker
    kube-proxy
    kubelet
    docker
    kube-proxy
    api
    etcd
    BOSH͕௚͢
    

    View Slide

34. VM
    OS
    VM
    OS
    VM
    OS
    release
    deployment
    VM
    VM
    ΋͠εέʔϧΞ΢τͨ͘͠ͳͬͯ΋
    

    View Slide

35. VM
    OS
    VM
    OS
    VM
    OS
    release
    deployment
    VM
    VM
    deploymentʹ૿΍͍ͨ͠෼͚ͩॻ͍ͯɺ
    BOSHʹ৯ΘͤΕ͹εέʔϧ׬ྃ
    

    View Slide

36. Day1
    • ࠷ॳͷσϓϩΠ
    • AnsibleͰ΋ChefͰ΋PuppetͰ΋্ख͘΍ΕΔ
    ӡ༻ͷ2ϑΣʔζ
    Day2
    • ϞχλϦϯά
    • Ξοϓσʔτ
    • ϥΠϑαΠΫϧϚωδϝϯτ
    • όοΫΞοϓ
    

    View Slide

37. Day1
    • ࠷ॳͷσϓϩΠ
    • AnsibleͰ΋ChefͰ΋PuppetͰ΋্ख͘΍ΕΔ
    ӡ༻ͷ2ϑΣʔζ
    Day2
    • ϞχλϦϯά
    • Ξοϓσʔτ
    • ϥΠϑαΠΫϧϚωδϝϯτ
    • όοΫΞοϓ
    ӡ༻͸Day2໋͕ͦ͜ɻ
    ͚ͩͲɺͳ͔ͥ͜͜ΛμΫτςʔϓʹ
    ͍ͯ͠Δέʔε͕ଟ͍
    

    View Slide

38. Day2Λҙࣝ͠ͳ͍ͱɾɾɾ
    • ӡ༻ίετ͕O(n)Ͱ૿େɹ( Լख͢Ε͹O(n^2)ͷέʔε΋ŋŋŋ )
    • AnsibleͰ͍͍έʔε
    • খن໛ͳΞϓϦ
    • 1ճσϓϩΠͨ͠ΒҎ߱͋·ΓΞοϓσʔτ͠ͳ͍ΞϓϦ
    • Day2ͷҙ͕ࣝඞਢͳέʔε
    • େن໛ɺ෼ࢄܕ
    • ͭ·Γ͸Cloud Foundryͱ͔BOSHͱ͔
    

    View Slide

39. Day2ӡ༻͠·͠ΐ͏
    

    View Slide

40. ͔ͭͯCF΍ͬͯͨਓ
    Ͱ΋ɺBOSH΋େ֓ͭΒ͘ͳ͍ʁ
    ֶशίετ΍͹͍
    

    View Slide

41. ͔ͭͯͷBOSH
    

    View Slide

42. - name: doppler
    azs:
    - z1
    instances: 1
    vm_type: small
    stemcell: default
    networks:
    - name: default
    jobs:
    - name: consul_agent
    release: consul
    consumes:
    consul: {from: consul_link}
    consul_common: nil
    consul_server: nil
    consul_client: nil
    properties:
    consul:
    agent:
    services:
    doppler:
    name: doppler
    - name: doppler
    release: loggregator
    properties:
    doppler:
    etcd:
    client_cert: "((etcd_client.certificate))"
    client_key: "((etcd_client.private_key))"
    loggregator:
    tls:
    ca_cert: "((loggregator_tls_doppler.ca))"
    doppler:
    cert: "((loggregator_tls_doppler.certificate))"
    key: "((loggregator_tls_doppler.private_key))"
    etcd:
    require_ssl: true
    ca_cert: "((etcd_server.ca))"
    machines:
    - cf-etcd.service.cf.internal
    doppler_endpoint:
    shared_secret: "((dropsonde_shared_secret))"
    - name: syslog_drain_binder
    release: loggregator
    properties:
    loggregator:
    tls:
    key: "((loggregator_tls_syslogdrainbinder.private_key))" [437/1737]
    etcd:
    require_ssl: true
    ca_cert: "((etcd_server.ca))"
    machines:
    - cf-etcd.service.cf.internal
    syslog_drain_binder:
    etcd:
    client_cert: "((etcd_client.certificate))"
    client_key: "((etcd_client.private_key))"
    system_domain: "((system_domain))"
    cc:
    mutual_tls:
    ca_cert: "((loggregator_tls_syslogdrainbinder.ca))"
    srv_api_uri: https://api.((system_domain))
    ssl: *ssl
    - name: metron_agent
    release: loggregator
    properties: *metron_agent_properties
    - name: log-api
    azs:
    - z1
    instances: 1
    vm_type: small
    stemcell: default
    update:
    max_in_flight: 1
    serial: true
    networks:
    ntroller:
    ͔ͭͯͷBOSH
    YAML
    - name: default
    jobs:
    - name: consul_agent
    release: consul
    consumes:
    consul: {from: consul_link}
    consul_common: nil
    consul_server: nil
    consul_client: nil
    properties:
    consul:
    agent:
    services:
    loggregator_trafficcontroller: {}
    - name: loggregator_trafficcontroller
    release: loggregator
    properties:
    traffic_co
    properties:
    traffic_controller:
    etcd:
    client_cert: "((etcd_client.certificate))"
    client_key: "((etcd_client.private_key))"
    uaa:
    url: https://uaa.((system_domain))
    loggregator:
    tls:
    ca_cert: "((loggregator_tls_tc.ca))"
    trafficcontroller:
    cert: "((loggregator_tls_tc.certificate))"
    key: "((loggregator_tls_tc.private_key))"
    etcd:
    require_ssl: true
    ca_cert: "((etcd_server.ca))"
    machines:
    - cf-etcd.service.cf.internal
    uaa:
    client_secret: "((uaa_clients_doppler_secret))"
    system_domain: "((system_domain))"
    ssl: *ssl
    cc:
    srv_api_uri: "http://cloud-controller-ng.service.cf.internal:9022"
    - name: route_registrar
    release: routing
    properties:
    route_registrar:
    routes:
    - name: loggregator
    port: 8080
    registration_interval: 20s
    uris:
    - loggregator.((system_domain))
    - name: doppler
    port: 8081
    registration_interval: 20s
    uris:
    - doppler.((system_domain))
    - "*.doppler.((system_domain))"
    - name: metron_agent
    release: loggregator
    properties: *metron_agent_properties
    variables:
    - name: blobstore_admin_users_password
    type: password
    - name: blobstore_secure_link_secret
    - name: doppler
    azs:
    - z1
    instances: 1
    vm_type: small
    stemcell: default
    networks:
    - name: default
    jobs:
    - name: consul_agent
    release: consul
    consumes:
    consul: {from: consul_link}
    consul_common: nil
    consul_server: nil
    consul_client: nil
    properties:
    consul:
    agent:
    services:
    doppler:
    name: doppler
    - name: doppler
    release: loggregator
    properties:
    doppler:
    etcd:
    client_cert: "((etcd_client.certificate))"
    client_key: "((etcd_client.private_key))"
    loggregator:
    tls:
    ca_cert: "((loggregator_tls_doppler.ca))"
    doppler:
    cert: "((loggregator_tls_doppler.certificate))"
    key: "((loggregator_tls_doppler.private_key))"
    etcd:
    require_ssl: true
    ca_cert: "((etcd_server.ca))"
    machines:
    - cf-etcd.service.cf.internal
    doppler_endpoint:
    shared_secret: "((dropsonde_shared_secret))"
    - name: syslog_drain_binder
    release: loggregator
    properties:
    loggregator:
    tls:
    key: "((loggregator_tls_syslogdrainbinder.private_key))" [437/1737]
    etcd:
    require_ssl: true
    ca_cert: "((etcd_server.ca))"
    machines:
    - cf-etcd.service.cf.internal
    syslog_drain_binder:
    etcd:
    client_cert: "((etcd_client.certificate))"
    client_key: "((etcd_client.private_key))"
    system_domain: "((system_domain))"
    cc:
    mutual_tls:
    ca_cert: "((loggregator_tls_syslogdrainbinder.ca))"
    srv_api_uri: https://api.((system_domain))
    ssl: *ssl
    - name: metron_agent
    release: loggregator
    properties: *metron_agent_properties
    - name: log-api
    azs:
    - z1
    instances: 1
    vm_type: small
    stemcell: default
    update:
    max_in_flight: 1
    serial: true
    networks:
    ntroller:
    - name: default
    jobs:
    - name: consul_agent
    release: consul
    consumes:
    consul: {from: consul_link}
    consul_common: nil
    consul_server: nil
    consul_client: nil
    properties:
    consul:
    agent:
    services:
    loggregator_trafficcontroller: {}
    - name: loggregator_trafficcontroller
    release: loggregator
    properties:
    traffic_co
    properties:
    traffic_controller:
    etcd:
    client_cert: "((etcd_client.certificate))"
    client_key: "((etcd_client.private_key))"
    uaa:
    url: https://uaa.((system_domain))
    loggregator:
    tls:
    ca_cert: "((loggregator_tls_tc.ca))"
    trafficcontroller:
    cert: "((loggregator_tls_tc.certificate))"
    key: "((loggregator_tls_tc.private_key))"
    etcd:
    require_ssl: true
    ca_cert: "((etcd_server.ca))"
    machines:
    - cf-etcd.service.cf.internal
    uaa:
    client_secret: "((uaa_clients_doppler_secret))"
    system_domain: "((system_domain))"
    ssl: *ssl
    cc:
    srv_api_uri: "http://cloud-controller-ng.service.cf.internal:9022"
    - name: route_registrar
    release: routing
    properties:
    route_registrar:
    routes:
    - name: loggregator
    port: 8080
    registration_interval: 20s
    uris:
    - loggregator.((system_domain))
    - name: doppler
    port: 8081
    registration_interval: 20s
    uris:
    - doppler.((system_domain))
    - "*.doppler.((system_domain))"
    - name: metron_agent
    release: loggregator
    properties: *metron_agent_properties
    variables:
    - name: blobstore_admin_users_password
    type: password
    - name: blobstore_secure_link_secret
    

    View Slide

43. ࠓͷBOSH
    YAML
    YAML
    YAML
    

    View Slide

44. ࠓͷBOSH
    • Manifest v2ʹͳ͍ͬͯͩͿݟ௨͕͠ྑ͘ͳͬͨ
    • BOSH CLI v2 (Go) ͕ࠓ݄GAʹ
    • ൿ఻ͷλϨʹͳΓ͕ͪͩͬͨdeployment manifest͕ͦͦ͜͜
    ݟ௨͠Α͘ɻ
    

    View Slide

45. BOSH
    

    View Slide

46. Kubo
    

    View Slide

47. Kubo
    Kubernetes + BOSH
    

    View Slide

48. https://pivotal.io/kubo
    

    View Slide

49. CFFʹ΋Proposal͕ग़͍ͯ·͢
    https://docs.google.com/document/d/1ZOFD5nBQC_vh9CmKHOGT7ugtNaJQ1t03jkLVsyDOH6k/edit?usp=sharing
    

    View Slide

50. Function
    Systems
    Event-driven
    microservices.
    Developers use a variety of abstractions today.
    App-centric
    Systems
    Full-featured
    applications.
    Container
    Systems
    Deep control over
    app packaging and
    runtime behavior.
    Data
    Services
    On Demand
    Services via
    interfaces.
    Infrastructure
    On-premise and/or public clouds.
    BOSH
    Cloud NativeͳΞϓϦέʔγϣϯͷ
    ։ൃɾӡ༻͸CF͕ϕετ
    

    View Slide

51. Function
    Systems
    Event-driven
    microservices.
    Developers use a variety of abstractions today.
    App-centric
    Systems
    Full-featured
    applications.
    Container
    Systems
    Deep control over
    app packaging and
    runtime behavior.
    Data
    Services
    On Demand
    Services via
    interfaces.
    Infrastructure
    On-premise and/or public clouds.
    BOSH
    CFͰ͸ରԠ͖͠Εͳ͍ϨΨγʔΞϓϦ΍
    StatefulͳΞϓϦέʔγϣϯ͸k8sͰ
    

    View Slide

52. Function
    Systems
    Event-driven
    microservices.
    Developers use a variety of abstractions today.
    App-centric
    Systems
    Full-featured
    applications.
    Container
    Systems
    Deep control over
    app packaging and
    runtime behavior.
    Data
    Services
    On Demand
    Services via
    interfaces.
    Infrastructure
    On-premise and/or public clouds.
    BOSH
    ͦΕΒΛBOSHͰ؅ཧ
    

    View Slide

53. k8s API͸CFͷTCP Routerܦ༝
    

    View Slide

54. api-route-registrar͕ϧʔτΛ޿ࠂ
    

    View Slide

55. k8s Pod΁ͷHTTPΞΫηε͸Gorouterܦ༝
    

    View Slide

56. route-sync͕NATSܦ༝ͰϧʔτΛ޿ࠂ
    

    View Slide

57. CFͷRouting͸ࠓ·Ͱ௨Γ
    

    View Slide

58. CFͷRouting Layerͱ
    k8s΁ͷRouting͕౷߹͞Ε͍ͯΔ
    

    View Slide

59. k8s StandaloneͰͷߏங΋ͦͷ͏ͪग़དྷΔΑ͏ʹͳΔͱ͔
    

    View Slide

60. DEMO
    

    View Slide

61. ݱࡏͷεςʔλε
    ·ͩ·ͩΞϧϑΝ
    

    View Slide

62. Product Roadmap
    2017
    • K8 Parity: cloud packages for LB & Volumes, LB deployment type
    • Networking: app routes externally accessible, replace powerDNS
    • Persistence: stateful workloads for COTS data services
    • High Availability: Single-AZ & Multi-AZ / failover
    • Core: Migration to Etcd v3
    • Rolling upgrades: Cluster upgrades w/Zero-downtime
    • Multi-IAAS: extend support for all BOSH-supported IAAS

    https://docs.google.com/presentation/d/1z-qGCcHLlPpz5LtS0TOcvBZIK4hUQ4GhB-jjQyHEF3c/edit?usp=sharing
    

    View Slide

63. ·ͱΊ
    • େن໛෼ࢄγεςϜΛAnsible΍Chef+IaaSͰ؅ཧ͢Δͷ͸

    ແཧήʔ
    • Day2Λҙࣝͨ͠ӡ༻Λߟ͑Α͏ɻBOSHͰָ͠·͠ΐ͏ɻ
    • CF+KuboͰCloud NativeͳΞϓϦέʔγϣϯΛޮ཰Α͘ӡ༻
    

    View Slide

64. Resources
    • https://github.com/pivotal-cf-experimental/kubo-deployment
    • https://github.com/pivotal-cf-experimental/kubo-release
    • http://bosh.io/docs
    

    View Slide