CoreOSクラスタにDockerコンテナをデプロイ #dockerjp

1. Deploy Docker Containers on CoreOS Cluster
2. I’m Taichi Nakashima @deeeet tcnksm https://www.flickr.com/photos/unforgiven/9278027165
6. • Portability • Image distribution/sharing • Resource isolation Docker provides
7. Docker does NOT provide • Orchestration • Service discovery • Monitoring • Docker host environment parity
8. HOST A Story of infra growing We start with a single host
9. HOST A If we use docker, we can see a host like it has many isolated rooms Story of infra growing
10. HOST A A Container A Deploying first docker container Story of infra growing
11. HOST A A B docker --link Within a single host, we can easily link 2 containers Story of infra growing
12. HOST A A B C D Number of containers will increase as service growing Story of infra growing
13. HOST A HOST B HOST C F A B C D So at some point, we would think about scale out servers Story of infra growing
14. HOST A HOST B A B HOST C D C E F G And deploying docker containers on multiple host Story of infra growing
15. HOST A HOST B A B HOST C D C E F G Problems will be happen … Story of infra growing
16. HOST A HOST B A B HOST C D C E F G How to link two docker container between multiple hosts ? Orchestration
17. HOST A HOST B A B HOST C D C E F G Where is a service F on container F ? Service discovery
18. HOST A HOST B A B HOST C D C E F G H H H I J K L M N O Where should we deploy container H ? Scheduling (It seems that HostB and HostC already has many containers)
19. HOST A HOST B A B HOST C D C E F G Container G is working ? Monitoring
20. DEV PRODUTION A A B B Docker v1.4 Docker v1.3 Do you use same version of tool on DEV and PROSDUCTION? Docker host environment parity (As I mentioned, we must use orchestration tool, how to ensure we use same version not only docker)
22. CoreOS is a new Linux distribution that provides scale with high resilience infrastructure like Google, Facebook and Twitter realize.
23. #1 CoreOS features #2 CoreOS technologies #3 CoreOS solves… #4 CoreOS operation
24. #1 CoreOS features #2 CoreOS technologies #3 CoreOS solves… #4 CoreOS operation {• Minimal • Docker • Update system • Clustering
25. Minimal Traditional Linux distributions has always been about adding value by adding more. CoreOS creates value by giving you less. http://formtek.com/blog/coreos-linux-vendors-in-the-crosshairs-of-disruption/
26. • No package manager • No language runtime Docker Every application will work on Docker container
27. Update System Quick and reliable updating system Same as update Engine as Chrome (Omahaʣ Just Replacing root file system Easy to validate, easy to rollback
28. Update System Chef, Ansible, Puppet… ?? • OS … Replace root FS • Application … Update docker image • Configuration … Update etcd value
29. Clustering Clustered By Default using powerful and simple tool like etcd and fleet Data center as a Computer
30. Data Center as a Computer Clustering multiple servers in DC and using it like it’s a single computer
31. #1 CoreOS features #2 CoreOS technologies #3 CoreOS solves… #4 CoreOS operation {• etcd • fleet • cloud-config
32. etcd e.g., Port & IP of service A Leader election (Raft algorithm) Distributed key-value store Base of clustering Save/distribute config values
33. $ etcdctl set /msg hello hello Setting and getting a key by etcdctl $ etcdctl get /msg hello
34. $ curl -L http://127.0.0.1:4001/v2/keys/msg \ -X PUT -d value="hello" $ curl -L http://127.0.0.1:4001/v2/keys/msg { "action":"get", "node":{ "key":"/msg", "value":"hello", "modifiedIndex":1157552, "createdIndex":1157552 } } Setting and getting a key by curl
35. fleet Distributed init System Confirm parity of desired state and current state Place docker container on an appropriate machine based on unit file and services which are already worked on cluster Scheduling & deploy docker container Monitor & failover container service
36. fleet Distributed init System
37. [X-Fleet] ɾ.BDIJOF*% ɾ$POqJDUT Machine identified by the given string. ɾ.BDIJOF0G ɾ.BDIJOF.FUBEBUB ɾ(MPCBM Machine that hosts a specific unit. Machine with this specific metadata. Prevent a unit from being collocated with other units. Schedule this unit on all agents in the cluster.
38. Example of unit file (web.service) [Unit] Description=docker meetup demo on port %i Requires=docker.service Requires=discovery@%i.service [Service] TimeoutStartSec=0 KillMode=none EnvironmentFile=/etc/environment ExecStartPre=-/usr/bin/docker kill web.%i ExecStartPre=-/usr/bin/docker rm web.%i ExecStartPre=/usr/bin/docker pull tcnksm/docker-meetup-demo-web:1 ExecStart=/usr/bin/docker run --name web.%i -p ${COREOS_PRIVATE_IPV4}:%i:80 tcnksm/docker-meetup-demo-web:1 ExecStop=/usr/bin/docker stop web.%i [X-Fleet] MachineMetadata=provider=digitalocean Conflicts=web@*.service
39. [Unit] Description=docker meetup demo on port %i Requires=docker.service Requires=discovery@%i.service [Service] TimeoutStartSec=0 KillMode=none EnvironmentFile=/etc/environment ExecStartPre=-/usr/bin/docker kill web.%i ExecStartPre=-/usr/bin/docker rm web.%i ExecStartPre=/usr/bin/docker pull tcnksm/docker-meetup-demo-web:1 ExecStart=/usr/bin/docker run --name web.%i -p ${COREOS_PRIVATE_IPV4}:%i:80 tcnksm/docker-meetup-demo-web:1 ExecStop=/usr/bin/docker stop web.%i [X-Fleet] MachineMetadata=provider=digitalocean Conflicts=web@*.service Example of unit file (web.service)
40. [Unit] Description=docker meetup demo on port %i Requires=docker.service Requires=discovery@%i.service [Service] TimeoutStartSec=0 KillMode=none EnvironmentFile=/etc/environment ExecStartPre=-/usr/bin/docker kill web.%i ExecStartPre=-/usr/bin/docker rm web.%i ExecStartPre=/usr/bin/docker pull tcnksm/docker-meetup-demo-web:1 ExecStart=/usr/bin/docker run --name web.%i -p ${COREOS_PRIVATE_IPV4}:%i:80 tcnksm/docker-meetup-demo-web:1 ExecStop=/usr/bin/docker stop web.%i [X-Fleet] MachineMetadata=provider=digitalocean Conflicts=web@*.service Example of unit file (web.service)
41. [Unit] Description=docker meetup demo on port %i Requires=docker.service Requires=discovery@%i.service [Service] TimeoutStartSec=0 KillMode=none EnvironmentFile=/etc/environment ExecStartPre=-/usr/bin/docker kill web.%i ExecStartPre=-/usr/bin/docker rm web.%i ExecStartPre=/usr/bin/docker pull tcnksm/docker-meetup-demo-web:1 ExecStart=/usr/bin/docker run --name web.%i -p ${COREOS_PRIVATE_IPV4}:%i:80 tcnksm/docker-meetup-demo-web:1 ExecStop=/usr/bin/docker stop web.%i [X-Fleet] MachineMetadata=provider=digitalocean Conflicts=web@*.service Example of unit file (web.service)
42. Deploy service by fleet $ fleetctl start web.service Unit web.service launched on 06664657.../172.20.20.103 $ fleetctl start instance/*
43. cloud-config cloud-config.yaml Customize CoreOS machines Define service Add & configure user Write config file Define update policy ….
44. Example of cloud-config #cloud-config coreos: etcd: discovery: https://discovery.etcd.io/<YOUR_TOKEN> addr: $private_ipv4:4001 peer-addr: $private_ipv4:7001 fleet: public-ip: $private_ipv4 metadata: role=web,provider=digitalocean units: - name: etcd.service command: start - name: fleet.service command: start update: group: alpha reboot-strategy: best-effort
45. #cloud-config coreos: etcd: discovery: https://discovery.etcd.io/<YOUR_TOKEN> addr: $private_ipv4:4001 peer-addr: $private_ipv4:7001 fleet: public-ip: $private_ipv4 metadata: role=web,provider=digitalocean units: - name: etcd.service command: start - name: fleet.service command: start update: group: alpha reboot-strategy: best-effort Example of cloud-config
46. #cloud-config coreos: etcd: discovery: https://discovery.etcd.io/<YOUR_TOKEN> addr: $private_ipv4:4001 peer-addr: $private_ipv4:7001 fleet: public-ip: $private_ipv4 metadata: role=web,provider=digitalocean units: - name: etcd.service command: start - name: fleet.service command: start update: group: alpha reboot-strategy: best-effort Example of cloud-config
47. #1 CoreOS features #2 CoreOS technologies #3 CoreOS solves… #4 CoreOS operation
48. Docker does NOT provide • Orchestration • Service discovery • Monitoring • Docker host environment parity
49. HOST A HOST B A B HOST C D C E F G How to link two docker container between multiple hosts ? Orchestration
50. HOST A HOST B A B HOST C D C E F G How to link two docker container between multiple hosts ? Orchestration etcd
51. HOST A HOST B A B HOST C D C E F G Where is a service F on container F ? Service discovery
52. HOST A HOST B A B HOST C D C E F G Where is a service F on container F ? Service discovery etcd, fleet
53. Service discovery by fleet $ fleetctl list-units UNIT MACHINE ACTIVE SUB lb.service 4cadccf1.../172.20.20.101 active running web@8001service 4cadccf1.../172.20.20.101 active running web@8002service 4cadccf1.../172.20.20.102 active running web@8003service 4cadccf1.../172.20.20.104 active running grafana.service 06664657.../172.20.20.103 active running influx.service 06664657.../172.20.20.103 active running
54. HOST A HOST B A B HOST C D C E F G H H H I J K L M N O Where should we deploy container H ? Scheduling (It seems that HostB and HostC already has many containers)
55. HOST A HOST B A B HOST C D C E F G H H H I J K L M N O Where should we deploy container H ? Scheduling (It seems that HostB and HostC already has many containers) fleet
56. HOST A HOST B A B HOST C D C E F G Container G is working ? Monitoring
57. HOST A HOST B A B HOST C D C E F G Container G is working ? Monitoring fleet
58. DEV PRODUTION A A B B Docker v1.4 Docker v1.3 Do you use same version of tool on DEV and PROSDUCTION? Docker host environment parity (As I mentioned, we must use orchestration tool, how to ensure we use same version not only docker)
59. DEV PRODUTION A A B B Docker v1.4 Docker v1.3 Do you use same version of tool on DEV and PROSDUCTION? Docker host environment parity (As I mentioned, we must use orchestration tool, how to ensure we use same version not only docker) cloud-config
60. PlayStation: Developing Applications on CoreOS https://www.youtube.com/watch?v=M9hBsRUeRdg
61. #1 CoreOS features #2 CoreOS technologies #3 CoreOS solves… #4 CoreOS operation
62. Demo of CoreOS operation Building CoreOS cluster on DigitalOcean by terraform Build CoreOS cluster Using a demo application container which is just serving static HTML file Deploy demo application by fleet Deploying multiple demo application containers and Load Balancing it By using LB container with confd Scale out demo application
63. Demo of CoreOS operation Building CoreOS cluster on DigitalOcean by terraform Build CoreOS cluster Using a demo application container which is just serving static HTML file Deploy demo application by fleet Deploying multiple demo application containers and Load Balancing it By using LB container with confd Scale out demo application
64. Building CoreOS cluster CoreOS cluster on two different clouds or cloud + bare metal is supported and encouraged. CoreOS runs on almost any platform Cluster size will be changed based on growing service and request loads Cluster size will be changed
66. Infrastructure as Code Terraform is a tool for building, changing, and combining infrastructure safely and efficiently.
67. + Easy to build multiple platform On DigitalOcean, on Amazon EC2 and combine Easy to manage cluster member Just writing .tf file
68. resource "digitalocean_droplet" "web1" { image = "coreos-stable" private_networking = true region = "sgp1" size = "512mb" ssh_keys = ["${var.ssh_key_id}"] user_data = "${file("cloud-config-web.yml")}" } .tf file for building CoreOS cluster on DigitalOcean
69. resource "digitalocean_droplet" "web1" { image = "coreos-stable" private_networking = true region = "sgp1" size = "512mb" ssh_keys = ["${var.ssh_key_id}"] user_data = "${file("cloud-config-web.yml")}" } .tf file for building CoreOS cluster on DigitalOcean
70. resource "digitalocean_droplet" "web1" { image = "coreos-stable" private_networking = true region = "sgp1" size = "512mb" ssh_keys = ["${var.ssh_key_id}"] user_data = "${file("cloud-config-web.yml")}" } .tf file for building CoreOS cluster on DigitalOcean
71. resource "digitalocean_droplet" "web1" { image = "coreos-stable" private_networking = true region = "sgp1" size = "512mb" ssh_keys = ["${var.ssh_key_id}"] user_data = "${file("cloud-config-web.yml")}" } .tf file for building CoreOS cluster on DigitalOcean
72. resource "digitalocean_droplet" "web1" { image = "coreos-stable" private_networking = true region = "sgp1" size = "512mb" ssh_keys = ["${var.ssh_key_id}"] user_data = "${file("cloud-config-web.yml")}" } resource "digitalocean_droplet" "web2" { image = "coreos-stable" private_networking = true region = "sgp1" size = "512mb" ssh_keys = ["${var.ssh_key_id}"] user_data = "${file("cloud-config-web.yml")}" } .tf file for building CoreOS cluster on DigitalOcean
73. Build infrastructure by terraform CLI $ terraform plan $ terraform apply
74. Demo of CoreOS operation Building CoreOS cluster on DigitalOcean by terraform Build CoreOS cluster Using a demo application container which is just serving static HTML file Deploy demo application by fleet Deploying multiple demo application containers and Load Balancing it By using LB container with confd Scale out demo application
75. Demo application container $ docker run -p 8888:80 tcnksm/docker-meetup-demo-web:1 $ curl localhost:8888 <h1>DEMO container 1</h1><h3>at Docker meetup #4</h3>
76. web.service to deploy demo application container [Unit] Description=docker meetup demo on port %i Requires=docker.service Requires=discovery@%i.service [Service] TimeoutStartSec=0 KillMode=none EnvironmentFile=/etc/environment ExecStartPre=-/usr/bin/docker kill web.%i ExecStartPre=-/usr/bin/docker rm web.%i ExecStartPre=/usr/bin/docker pull tcnksm/docker-meetup-demo-web:1 ExecStart=/usr/bin/docker run --name web.%i -p ${COREOS_PRIVATE_IPV4}:%i:80 tcnksm/docker-meetup-demo-web:1 ExecStop=/usr/bin/docker stop web.%i [X-Fleet] MachineMetadata=provider=digitalocean Conflicts=web@*.service
77. Demo of CoreOS operation Building CoreOS cluster on DigitalOcean by terraform Build CoreOS cluster Using a demo application container which is just serving static HTML file Deploy demo application by fleet Deploying multiple demo application containers and Load Balancing it By using LB container with confd Scale out demo application
78. F W1 CoreOS B :80 We start with a single demo application container.
79. D W1 W2 CoreOS B CoreOS C We will scale out servers and deploy demo application container on each host.
80. CoreOS A D W1 W2 LB CoreOS B CoreOS C :80 :8888 :9999 And load-balancing to them by LB container.
81. CoreOS A D W1 W2 LB CoreOS B CoreOS C :80 :8888 :9999 How to do it? Assuming that the number of containers will be changed dynamically.
82. CoreOS A D F W1 W2 LB CoreOS B CoreOS C :80 :8888 :9999 etcd The easy way is using etcd.
83. CoreOS A D F W1 W2 LB CoreOS B CoreOS C :80 :8888 :9999 etcd D1 D2 1PSU*1 1PSU*1 Discovery service Registering Port & IP of demo app When deploying demo app service, we also deploy discovery service.
84. [Service] EnvironmentFile=/etc/environment ExecStart=/bin/bash -c '\ while true; do \ curl -f ${COREOS_PRIVATE_IPV4}:%i; \ if [ $? -eq 0 ]; then \ etcdctl set /services/docker-meetup-demo-web/${COREOS_PRIVATE_IPV4}\ '${COREOS_PRIVATE_IPV4}:8888\' --ttl 30;\ else \ etcdctl rm /services/docker-meetup-demo-web/${COREOS_PRIVATE_IPV4}; fi; \ sleep 20; \ done' [X-Fleet] MachineOf=web@%i.service discovery.service to register demo app IP & Port to etcd (Excerpt)
85. [Service] EnvironmentFile=/etc/environment ExecStart=/bin/bash -c '\ while true; do \ curl -f ${COREOS_PRIVATE_IPV4}:%i; \ if [ $? -eq 0 ]; then \ etcdctl set /services/docker-meetup-demo-web/${COREOS_PRIVATE_IPV4}\ '${COREOS_PRIVATE_IPV4}:8888\' --ttl 30;\ else \ etcdctl rm /services/docker-meetup-demo-web/${COREOS_PRIVATE_IPV4}; fi; \ sleep 20; \ done' [X-Fleet] MachineOf=web@%i.service discovery.service to register demo app IP & Port to etcd (Excerpt)
86. CoreOS A D F W1 W2 LB CoreOS B CoreOS C :80 :8888 :9999 etcd W1’ W2’ 1PSU*1 1PSU*1 D1 D2 Discovery service Registering Port & IP of demo app { { How to deploy same service on same machine?
87. CoreOS A D F W1 W2 LB CoreOS B CoreOS C :80 :8888 :9999 etcd W1’ W2’ 1PSU*1 1PSU*1 { fleet { 9'MFFU.BDIJOF0G D1 D2 Discovery service Registering Port & IP of demo app We can use fleet (X-Fleet’s MachineOf).
88. [Service] EnvironmentFile=/etc/environment ExecStart=/bin/bash -c '\ while true; do \ curl -f ${COREOS_PRIVATE_IPV4}:%i; \ if [ $? -eq 0 ]; then \ etcdctl set /services/docker-meetup-demo-web/${COREOS_PRIVATE_IPV4}\ '${COREOS_PRIVATE_IPV4}:8888\' --ttl 30;\ else \ etcdctl rm /services/docker-meetup-demo-web/${COREOS_PRIVATE_IPV4}; fi; \ sleep 20; \ done' [X-Fleet] MachineOf=web@%i.service discovery.service to register demo app IP & Port to etcd (Excerpt)
89. CoreOS A D F W1 W2 LB CoreOS B CoreOS C :80 :8888 :9999 etcd W1’ W2’ 1PSU*1 1PSU*1 { fleet { 9'MFFU.BDIJOF0G D1 D2 Discovery service Registering Port & IP of demo app How LB uses etcd value for its configuration ?
90. CoreOS A D F W1 W2 LB CoreOS B CoreOS C :80 :8888 :9999 etcd W1’ W2’ 1PSU*1 1PSU*1 { fleet { 9'MFFU.BDIJOF0G D1 D2 Discovery service Registering Port & IP of demo app We can use confd and generate nginx .conf file dynamically. 1PSU*1 LFMTFZIJHIUPXFSDPOGE Manage local application configuration files using templates and data from etcd
91. tcnksm/docker-meetup-4-demo
92. @deeeet