CoreOSクラスタにDockerコンテナをデプロイ #dockerjp

CoreOSクラスタにDockerコンテナをデプロイ #dockerjp

Ecb3acc2d246962361a4f8b3f7a6dd12?s=128

taichi nakashima

January 16, 2015
Tweet

Transcript

  1. Deploy Docker Containers on CoreOS Cluster

  2. I’m Taichi Nakashima @deeeet tcnksm https://www.flickr.com/photos/unforgiven/9278027165

  3. None
  4. None
  5. None
  6. • Portability • Image distribution/sharing • Resource isolation Docker provides

  7. Docker does NOT provide • Orchestration • Service discovery •

    Monitoring • Docker host environment parity
  8. HOST A Story of infra growing We start with a

    single host
  9. HOST A If we use docker, we can see a

    host like it has many isolated rooms Story of infra growing
  10. HOST A A Container A Deploying first docker container Story

    of infra growing
  11. HOST A A B docker --link Within a single host,

    we can easily link 2 containers Story of infra growing
  12. HOST A A B C D Number of containers will

    increase as service growing Story of infra growing
  13. HOST A HOST B HOST C F A B C

    D So at some point, we would think about scale out servers Story of infra growing
  14. HOST A HOST B A B HOST C D C

    E F G And deploying docker containers on multiple host Story of infra growing
  15. HOST A HOST B A B HOST C D C

    E F G Problems will be happen … Story of infra growing
  16. HOST A HOST B A B HOST C D C

    E F G How to link two docker container between multiple hosts ? Orchestration
  17. HOST A HOST B A B HOST C D C

    E F G Where is a service F on container F ? Service discovery
  18. HOST A HOST B A B HOST C D C

    E F G H H H I J K L M N O Where should we deploy container H ? Scheduling (It seems that HostB and HostC already has many containers)
  19. HOST A HOST B A B HOST C D C

    E F G Container G is working ? Monitoring
  20. DEV PRODUTION A A B B Docker v1.4 Docker v1.3

    Do you use same version of tool on DEV and PROSDUCTION? Docker host environment parity (As I mentioned, we must use orchestration tool, how to ensure we use same version not only docker)
  21. None
  22. CoreOS is a new Linux distribution that provides scale with

    high resilience infrastructure like Google, Facebook and Twitter realize.
  23. #1 CoreOS features #2 CoreOS technologies #3 CoreOS solves… #4

    CoreOS operation
  24. #1 CoreOS features #2 CoreOS technologies #3 CoreOS solves… #4

    CoreOS operation {• Minimal • Docker • Update system • Clustering
  25. Minimal Traditional Linux distributions has always been about adding value

    by adding more. CoreOS creates value by giving you less. http://formtek.com/blog/coreos-linux-vendors-in-the-crosshairs-of-disruption/
  26. • No package manager • No language runtime Docker Every

    application will work on Docker container
  27. Update System Quick and reliable updating system Same as update

    Engine as Chrome (Omahaʣ Just Replacing root file system Easy to validate, easy to rollback
  28. Update System Chef, Ansible, Puppet… ?? • OS … Replace

    root FS • Application … Update docker image • Configuration … Update etcd value
  29. Clustering Clustered By Default using powerful and simple tool like

    etcd and fleet Data center as a Computer
  30. Data Center as a Computer Clustering multiple servers in DC

    and using it like it’s a single computer
  31. #1 CoreOS features #2 CoreOS technologies #3 CoreOS solves… #4

    CoreOS operation {• etcd • fleet • cloud-config
  32. etcd e.g., Port & IP of service A Leader election

    (Raft algorithm) Distributed key-value store Base of clustering Save/distribute config values
  33. $ etcdctl set /msg hello hello Setting and getting a

    key by etcdctl $ etcdctl get /msg hello
  34. $ curl -L http://127.0.0.1:4001/v2/keys/msg \ -X PUT -d value="hello" $

    curl -L http://127.0.0.1:4001/v2/keys/msg { "action":"get", "node":{ "key":"/msg", "value":"hello", "modifiedIndex":1157552, "createdIndex":1157552 } } Setting and getting a key by curl
  35. fleet Distributed init System Confirm parity of desired state and

    current state Place docker container on an appropriate machine based on unit file and services which are already worked on cluster Scheduling & deploy docker container Monitor & failover container service
  36. fleet Distributed init System

  37. [X-Fleet] ɾ.BDIJOF*% ɾ$POqJDUT Machine identified by the given string. ɾ.BDIJOF0G

    ɾ.BDIJOF.FUBEBUB ɾ(MPCBM Machine that hosts a specific unit. Machine with this specific metadata. Prevent a unit from being collocated with other units. Schedule this unit on all agents in the cluster.
  38. Example of unit file (web.service) [Unit] Description=docker meetup demo on

    port %i Requires=docker.service Requires=discovery@%i.service [Service] TimeoutStartSec=0 KillMode=none EnvironmentFile=/etc/environment ExecStartPre=-/usr/bin/docker kill web.%i ExecStartPre=-/usr/bin/docker rm web.%i ExecStartPre=/usr/bin/docker pull tcnksm/docker-meetup-demo-web:1 ExecStart=/usr/bin/docker run --name web.%i -p ${COREOS_PRIVATE_IPV4}:%i:80 tcnksm/docker-meetup-demo-web:1 ExecStop=/usr/bin/docker stop web.%i [X-Fleet] MachineMetadata=provider=digitalocean Conflicts=web@*.service
  39. [Unit] Description=docker meetup demo on port %i Requires=docker.service Requires=discovery@%i.service [Service]

    TimeoutStartSec=0 KillMode=none EnvironmentFile=/etc/environment ExecStartPre=-/usr/bin/docker kill web.%i ExecStartPre=-/usr/bin/docker rm web.%i ExecStartPre=/usr/bin/docker pull tcnksm/docker-meetup-demo-web:1 ExecStart=/usr/bin/docker run --name web.%i -p ${COREOS_PRIVATE_IPV4}:%i:80 tcnksm/docker-meetup-demo-web:1 ExecStop=/usr/bin/docker stop web.%i [X-Fleet] MachineMetadata=provider=digitalocean Conflicts=web@*.service Example of unit file (web.service)
  40. [Unit] Description=docker meetup demo on port %i Requires=docker.service Requires=discovery@%i.service [Service]

    TimeoutStartSec=0 KillMode=none EnvironmentFile=/etc/environment ExecStartPre=-/usr/bin/docker kill web.%i ExecStartPre=-/usr/bin/docker rm web.%i ExecStartPre=/usr/bin/docker pull tcnksm/docker-meetup-demo-web:1 ExecStart=/usr/bin/docker run --name web.%i -p ${COREOS_PRIVATE_IPV4}:%i:80 tcnksm/docker-meetup-demo-web:1 ExecStop=/usr/bin/docker stop web.%i [X-Fleet] MachineMetadata=provider=digitalocean Conflicts=web@*.service Example of unit file (web.service)
  41. [Unit] Description=docker meetup demo on port %i Requires=docker.service Requires=discovery@%i.service [Service]

    TimeoutStartSec=0 KillMode=none EnvironmentFile=/etc/environment ExecStartPre=-/usr/bin/docker kill web.%i ExecStartPre=-/usr/bin/docker rm web.%i ExecStartPre=/usr/bin/docker pull tcnksm/docker-meetup-demo-web:1 ExecStart=/usr/bin/docker run --name web.%i -p ${COREOS_PRIVATE_IPV4}:%i:80 tcnksm/docker-meetup-demo-web:1 ExecStop=/usr/bin/docker stop web.%i [X-Fleet] MachineMetadata=provider=digitalocean Conflicts=web@*.service Example of unit file (web.service)
  42. Deploy service by fleet $ fleetctl start web.service Unit web.service

    launched on 06664657.../172.20.20.103 $ fleetctl start instance/*
  43. cloud-config cloud-config.yaml Customize CoreOS machines Define service Add & configure

    user Write config file Define update policy ….
  44. Example of cloud-config #cloud-config coreos: etcd: discovery: https://discovery.etcd.io/<YOUR_TOKEN> addr: $private_ipv4:4001

    peer-addr: $private_ipv4:7001 fleet: public-ip: $private_ipv4 metadata: role=web,provider=digitalocean units: - name: etcd.service command: start - name: fleet.service command: start update: group: alpha reboot-strategy: best-effort
  45. #cloud-config coreos: etcd: discovery: https://discovery.etcd.io/<YOUR_TOKEN> addr: $private_ipv4:4001 peer-addr: $private_ipv4:7001 fleet:

    public-ip: $private_ipv4 metadata: role=web,provider=digitalocean units: - name: etcd.service command: start - name: fleet.service command: start update: group: alpha reboot-strategy: best-effort Example of cloud-config
  46. #cloud-config coreos: etcd: discovery: https://discovery.etcd.io/<YOUR_TOKEN> addr: $private_ipv4:4001 peer-addr: $private_ipv4:7001 fleet:

    public-ip: $private_ipv4 metadata: role=web,provider=digitalocean units: - name: etcd.service command: start - name: fleet.service command: start update: group: alpha reboot-strategy: best-effort Example of cloud-config
  47. #1 CoreOS features #2 CoreOS technologies #3 CoreOS solves… #4

    CoreOS operation
  48. Docker does NOT provide • Orchestration • Service discovery •

    Monitoring • Docker host environment parity
  49. HOST A HOST B A B HOST C D C

    E F G How to link two docker container between multiple hosts ? Orchestration
  50. HOST A HOST B A B HOST C D C

    E F G How to link two docker container between multiple hosts ? Orchestration etcd
  51. HOST A HOST B A B HOST C D C

    E F G Where is a service F on container F ? Service discovery
  52. HOST A HOST B A B HOST C D C

    E F G Where is a service F on container F ? Service discovery etcd, fleet
  53. Service discovery by fleet $ fleetctl list-units UNIT MACHINE ACTIVE

    SUB lb.service 4cadccf1.../172.20.20.101 active running web@8001service 4cadccf1.../172.20.20.101 active running web@8002service 4cadccf1.../172.20.20.102 active running web@8003service 4cadccf1.../172.20.20.104 active running grafana.service 06664657.../172.20.20.103 active running influx.service 06664657.../172.20.20.103 active running
  54. HOST A HOST B A B HOST C D C

    E F G H H H I J K L M N O Where should we deploy container H ? Scheduling (It seems that HostB and HostC already has many containers)
  55. HOST A HOST B A B HOST C D C

    E F G H H H I J K L M N O Where should we deploy container H ? Scheduling (It seems that HostB and HostC already has many containers) fleet
  56. HOST A HOST B A B HOST C D C

    E F G Container G is working ? Monitoring
  57. HOST A HOST B A B HOST C D C

    E F G Container G is working ? Monitoring fleet
  58. DEV PRODUTION A A B B Docker v1.4 Docker v1.3

    Do you use same version of tool on DEV and PROSDUCTION? Docker host environment parity (As I mentioned, we must use orchestration tool, how to ensure we use same version not only docker)
  59. DEV PRODUTION A A B B Docker v1.4 Docker v1.3

    Do you use same version of tool on DEV and PROSDUCTION? Docker host environment parity (As I mentioned, we must use orchestration tool, how to ensure we use same version not only docker) cloud-config
  60. PlayStation: Developing Applications on CoreOS https://www.youtube.com/watch?v=M9hBsRUeRdg

  61. #1 CoreOS features #2 CoreOS technologies #3 CoreOS solves… #4

    CoreOS operation
  62. Demo of CoreOS operation Building CoreOS cluster on DigitalOcean by

    terraform Build CoreOS cluster Using a demo application container which is just serving static HTML file Deploy demo application by fleet Deploying multiple demo application containers and Load Balancing it By using LB container with confd Scale out demo application
  63. Demo of CoreOS operation Building CoreOS cluster on DigitalOcean by

    terraform Build CoreOS cluster Using a demo application container which is just serving static HTML file Deploy demo application by fleet Deploying multiple demo application containers and Load Balancing it By using LB container with confd Scale out demo application
  64. Building CoreOS cluster CoreOS cluster on two different clouds or

    cloud + bare metal is supported and encouraged. CoreOS runs on almost any platform Cluster size will be changed based on growing service and request loads Cluster size will be changed
  65. None
  66. Infrastructure as Code Terraform is a tool for building, changing,

    and combining infrastructure safely and efficiently.
  67. + Easy to build multiple platform On DigitalOcean, on Amazon

    EC2 and combine Easy to manage cluster member Just writing .tf file
  68. resource "digitalocean_droplet" "web1" { image = "coreos-stable" private_networking = true

    region = "sgp1" size = "512mb" ssh_keys = ["${var.ssh_key_id}"] user_data = "${file("cloud-config-web.yml")}" } .tf file for building CoreOS cluster on DigitalOcean
  69. resource "digitalocean_droplet" "web1" { image = "coreos-stable" private_networking = true

    region = "sgp1" size = "512mb" ssh_keys = ["${var.ssh_key_id}"] user_data = "${file("cloud-config-web.yml")}" } .tf file for building CoreOS cluster on DigitalOcean
  70. resource "digitalocean_droplet" "web1" { image = "coreos-stable" private_networking = true

    region = "sgp1" size = "512mb" ssh_keys = ["${var.ssh_key_id}"] user_data = "${file("cloud-config-web.yml")}" } .tf file for building CoreOS cluster on DigitalOcean
  71. resource "digitalocean_droplet" "web1" { image = "coreos-stable" private_networking = true

    region = "sgp1" size = "512mb" ssh_keys = ["${var.ssh_key_id}"] user_data = "${file("cloud-config-web.yml")}" } .tf file for building CoreOS cluster on DigitalOcean
  72. resource "digitalocean_droplet" "web1" { image = "coreos-stable" private_networking = true

    region = "sgp1" size = "512mb" ssh_keys = ["${var.ssh_key_id}"] user_data = "${file("cloud-config-web.yml")}" } resource "digitalocean_droplet" "web2" { image = "coreos-stable" private_networking = true region = "sgp1" size = "512mb" ssh_keys = ["${var.ssh_key_id}"] user_data = "${file("cloud-config-web.yml")}" } .tf file for building CoreOS cluster on DigitalOcean
  73. Build infrastructure by terraform CLI $ terraform plan $ terraform

    apply
  74. Demo of CoreOS operation Building CoreOS cluster on DigitalOcean by

    terraform Build CoreOS cluster Using a demo application container which is just serving static HTML file Deploy demo application by fleet Deploying multiple demo application containers and Load Balancing it By using LB container with confd Scale out demo application
  75. Demo application container $ docker run -p 8888:80 tcnksm/docker-meetup-demo-web:1 $

    curl localhost:8888 <h1>DEMO container 1</h1><h3>at Docker meetup #4</h3>
  76. web.service to deploy demo application container [Unit] Description=docker meetup demo

    on port %i Requires=docker.service Requires=discovery@%i.service [Service] TimeoutStartSec=0 KillMode=none EnvironmentFile=/etc/environment ExecStartPre=-/usr/bin/docker kill web.%i ExecStartPre=-/usr/bin/docker rm web.%i ExecStartPre=/usr/bin/docker pull tcnksm/docker-meetup-demo-web:1 ExecStart=/usr/bin/docker run --name web.%i -p ${COREOS_PRIVATE_IPV4}:%i:80 tcnksm/docker-meetup-demo-web:1 ExecStop=/usr/bin/docker stop web.%i [X-Fleet] MachineMetadata=provider=digitalocean Conflicts=web@*.service
  77. Demo of CoreOS operation Building CoreOS cluster on DigitalOcean by

    terraform Build CoreOS cluster Using a demo application container which is just serving static HTML file Deploy demo application by fleet Deploying multiple demo application containers and Load Balancing it By using LB container with confd Scale out demo application
  78. F W1 CoreOS B :80 We start with a single

    demo application container.
  79. D W1 W2 CoreOS B CoreOS C We will scale

    out servers and deploy demo application container on each host.
  80. CoreOS A D W1 W2 LB CoreOS B CoreOS C

    :80 :8888 :9999 And load-balancing to them by LB container.
  81. CoreOS A D W1 W2 LB CoreOS B CoreOS C

    :80 :8888 :9999 How to do it? Assuming that the number of containers will be changed dynamically.
  82. CoreOS A D F W1 W2 LB CoreOS B CoreOS

    C :80 :8888 :9999 etcd The easy way is using etcd.
  83. CoreOS A D F W1 W2 LB CoreOS B CoreOS

    C :80 :8888 :9999 etcd D1 D2 1PSU*1 1PSU*1 Discovery service Registering Port & IP of demo app When deploying demo app service, we also deploy discovery service.
  84. [Service] EnvironmentFile=/etc/environment ExecStart=/bin/bash -c '\ while true; do \ curl

    -f ${COREOS_PRIVATE_IPV4}:%i; \ if [ $? -eq 0 ]; then \ etcdctl set /services/docker-meetup-demo-web/${COREOS_PRIVATE_IPV4}\ '${COREOS_PRIVATE_IPV4}:8888\' --ttl 30;\ else \ etcdctl rm /services/docker-meetup-demo-web/${COREOS_PRIVATE_IPV4}; fi; \ sleep 20; \ done' [X-Fleet] MachineOf=web@%i.service discovery.service to register demo app IP & Port to etcd (Excerpt)
  85. [Service] EnvironmentFile=/etc/environment ExecStart=/bin/bash -c '\ while true; do \ curl

    -f ${COREOS_PRIVATE_IPV4}:%i; \ if [ $? -eq 0 ]; then \ etcdctl set /services/docker-meetup-demo-web/${COREOS_PRIVATE_IPV4}\ '${COREOS_PRIVATE_IPV4}:8888\' --ttl 30;\ else \ etcdctl rm /services/docker-meetup-demo-web/${COREOS_PRIVATE_IPV4}; fi; \ sleep 20; \ done' [X-Fleet] MachineOf=web@%i.service discovery.service to register demo app IP & Port to etcd (Excerpt)
  86. CoreOS A D F W1 W2 LB CoreOS B CoreOS

    C :80 :8888 :9999 etcd W1’ W2’ 1PSU*1 1PSU*1 D1 D2 Discovery service Registering Port & IP of demo app { { How to deploy same service on same machine?
  87. CoreOS A D F W1 W2 LB CoreOS B CoreOS

    C :80 :8888 :9999 etcd W1’ W2’ 1PSU*1 1PSU*1 { fleet { 9'MFFU.BDIJOF0G D1 D2 Discovery service Registering Port & IP of demo app We can use fleet (X-Fleet’s MachineOf).
  88. [Service] EnvironmentFile=/etc/environment ExecStart=/bin/bash -c '\ while true; do \ curl

    -f ${COREOS_PRIVATE_IPV4}:%i; \ if [ $? -eq 0 ]; then \ etcdctl set /services/docker-meetup-demo-web/${COREOS_PRIVATE_IPV4}\ '${COREOS_PRIVATE_IPV4}:8888\' --ttl 30;\ else \ etcdctl rm /services/docker-meetup-demo-web/${COREOS_PRIVATE_IPV4}; fi; \ sleep 20; \ done' [X-Fleet] MachineOf=web@%i.service discovery.service to register demo app IP & Port to etcd (Excerpt)
  89. CoreOS A D F W1 W2 LB CoreOS B CoreOS

    C :80 :8888 :9999 etcd W1’ W2’ 1PSU*1 1PSU*1 { fleet { 9'MFFU.BDIJOF0G D1 D2 Discovery service Registering Port & IP of demo app How LB uses etcd value for its configuration ?
  90. CoreOS A D F W1 W2 LB CoreOS B CoreOS

    C :80 :8888 :9999 etcd W1’ W2’ 1PSU*1 1PSU*1 { fleet { 9'MFFU.BDIJOF0G D1 D2 Discovery service Registering Port & IP of demo app We can use confd and generate nginx .conf file dynamically. 1PSU*1 LFMTFZIJHIUPXFSDPOGE Manage local application configuration files using templates and data from etcd
  91. tcnksm/docker-meetup-4-demo

  92. @deeeet