Tweet
Tweet

Slide 1

Slide 1 text

Deploy Docker Containers on CoreOS Cluster

Slide 2

Slide 2 text

I’m Taichi Nakashima @deeeet tcnksm https://www.flickr.com/photos/unforgiven/9278027165

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

• Portability • Image distribution/sharing • Resource isolation Docker provides

Slide 7

Slide 7 text

Docker does NOT provide • Orchestration • Service discovery • Monitoring • Docker host environment parity

Slide 8

Slide 8 text

HOST A Story of infra growing We start with a single host

Slide 9

Slide 9 text

HOST A If we use docker, we can see a host like it has many isolated rooms Story of infra growing

Slide 10

Slide 10 text

HOST A A Container A Deploying first docker container Story of infra growing

Slide 11

Slide 11 text

HOST A A B docker --link Within a single host, we can easily link 2 containers Story of infra growing

Slide 12

Slide 12 text

HOST A A B C D Number of containers will increase as service growing Story of infra growing

Slide 13

Slide 13 text

HOST A HOST B HOST C F A B C D So at some point, we would think about scale out servers Story of infra growing

Slide 14

Slide 14 text

HOST A HOST B A B HOST C D C E F G And deploying docker containers on multiple host Story of infra growing

Slide 15

Slide 15 text

HOST A HOST B A B HOST C D C E F G Problems will be happen … Story of infra growing

Slide 16

Slide 16 text

HOST A HOST B A B HOST C D C E F G How to link two docker container between multiple hosts ? Orchestration

Slide 17

Slide 17 text

HOST A HOST B A B HOST C D C E F G Where is a service F on container F ? Service discovery

Slide 18

Slide 18 text

HOST A HOST B A B HOST C D C E F G H H H I J K L M N O Where should we deploy container H ? Scheduling (It seems that HostB and HostC already has many containers)

Slide 19

Slide 19 text

HOST A HOST B A B HOST C D C E F G Container G is working ? Monitoring

Slide 20

Slide 20 text

DEV PRODUTION A A B B Docker v1.4 Docker v1.3 Do you use same version of tool on DEV and PROSDUCTION? Docker host environment parity (As I mentioned, we must use orchestration tool, how to ensure we use same version not only docker)

Slide 21

Slide 21 text

No content

Slide 22

Slide 22 text

CoreOS is a new Linux distribution that provides scale with high resilience infrastructure like Google, Facebook and Twitter realize.

Slide 23

Slide 23 text

#1 CoreOS features #2 CoreOS technologies #3 CoreOS solves… #4 CoreOS operation

Slide 24

Slide 24 text

#1 CoreOS features #2 CoreOS technologies #3 CoreOS solves… #4 CoreOS operation {• Minimal • Docker • Update system • Clustering

Slide 25

Slide 25 text

Minimal Traditional Linux distributions has always been about adding value by adding more. CoreOS creates value by giving you less. http://formtek.com/blog/coreos-linux-vendors-in-the-crosshairs-of-disruption/

Slide 26

Slide 26 text

• No package manager • No language runtime Docker Every application will work on Docker container

Slide 27

Slide 27 text

Update System Quick and reliable updating system Same as update Engine as Chrome (Omahaʣ Just Replacing root file system Easy to validate, easy to rollback

Slide 28

Slide 28 text

Update System Chef, Ansible, Puppet… ?? • OS … Replace root FS • Application … Update docker image • Configuration … Update etcd value

Slide 29

Slide 29 text

Clustering Clustered By Default using powerful and simple tool like etcd and fleet Data center as a Computer

Slide 30

Slide 30 text

Data Center as a Computer Clustering multiple servers in DC and using it like it’s a single computer

Slide 31

Slide 31 text

#1 CoreOS features #2 CoreOS technologies #3 CoreOS solves… #4 CoreOS operation {• etcd • fleet • cloud-config

Slide 32

Slide 32 text

etcd e.g., Port & IP of service A Leader election (Raft algorithm) Distributed key-value store Base of clustering Save/distribute config values

Slide 33

Slide 33 text

$ etcdctl set /msg hello hello Setting and getting a key by etcdctl $ etcdctl get /msg hello

Slide 34

Slide 34 text

$ curl -L http://127.0.0.1:4001/v2/keys/msg \ -X PUT -d value="hello" $ curl -L http://127.0.0.1:4001/v2/keys/msg { "action":"get", "node":{ "key":"/msg", "value":"hello", "modifiedIndex":1157552, "createdIndex":1157552 } } Setting and getting a key by curl

Slide 35

Slide 35 text

fleet Distributed init System Confirm parity of desired state and current state Place docker container on an appropriate machine based on unit file and services which are already worked on cluster Scheduling & deploy docker container Monitor & failover container service

Slide 36

Slide 36 text

fleet Distributed init System

Slide 37

Slide 37 text

[X-Fleet] ɾ.BDIJOF*% ɾ$POqJDUT Machine identified by the given string. ɾ.BDIJOF0G ɾ.BDIJOF.FUBEBUB ɾ(MPCBM Machine that hosts a specific unit. Machine with this specific metadata. Prevent a unit from being collocated with other units. Schedule this unit on all agents in the cluster.

Slide 38

Slide 38 text

Example of unit file (web.service) [Unit] Description=docker meetup demo on port %i Requires=docker.service Requires=discovery@%i.service [Service] TimeoutStartSec=0 KillMode=none EnvironmentFile=/etc/environment ExecStartPre=-/usr/bin/docker kill web.%i ExecStartPre=-/usr/bin/docker rm web.%i ExecStartPre=/usr/bin/docker pull tcnksm/docker-meetup-demo-web:1 ExecStart=/usr/bin/docker run --name web.%i -p ${COREOS_PRIVATE_IPV4}:%i:80 tcnksm/docker-meetup-demo-web:1 ExecStop=/usr/bin/docker stop web.%i [X-Fleet] MachineMetadata=provider=digitalocean Conflicts=web@*.service

Slide 39

Slide 39 text

[Unit] Description=docker meetup demo on port %i Requires=docker.service Requires=discovery@%i.service [Service] TimeoutStartSec=0 KillMode=none EnvironmentFile=/etc/environment ExecStartPre=-/usr/bin/docker kill web.%i ExecStartPre=-/usr/bin/docker rm web.%i ExecStartPre=/usr/bin/docker pull tcnksm/docker-meetup-demo-web:1 ExecStart=/usr/bin/docker run --name web.%i -p ${COREOS_PRIVATE_IPV4}:%i:80 tcnksm/docker-meetup-demo-web:1 ExecStop=/usr/bin/docker stop web.%i [X-Fleet] MachineMetadata=provider=digitalocean Conflicts=web@*.service Example of unit file (web.service)

Slide 40

Slide 40 text

[Unit] Description=docker meetup demo on port %i Requires=docker.service Requires=discovery@%i.service [Service] TimeoutStartSec=0 KillMode=none EnvironmentFile=/etc/environment ExecStartPre=-/usr/bin/docker kill web.%i ExecStartPre=-/usr/bin/docker rm web.%i ExecStartPre=/usr/bin/docker pull tcnksm/docker-meetup-demo-web:1 ExecStart=/usr/bin/docker run --name web.%i -p ${COREOS_PRIVATE_IPV4}:%i:80 tcnksm/docker-meetup-demo-web:1 ExecStop=/usr/bin/docker stop web.%i [X-Fleet] MachineMetadata=provider=digitalocean Conflicts=web@*.service Example of unit file (web.service)

Slide 41

Slide 41 text

[Unit] Description=docker meetup demo on port %i Requires=docker.service Requires=discovery@%i.service [Service] TimeoutStartSec=0 KillMode=none EnvironmentFile=/etc/environment ExecStartPre=-/usr/bin/docker kill web.%i ExecStartPre=-/usr/bin/docker rm web.%i ExecStartPre=/usr/bin/docker pull tcnksm/docker-meetup-demo-web:1 ExecStart=/usr/bin/docker run --name web.%i -p ${COREOS_PRIVATE_IPV4}:%i:80 tcnksm/docker-meetup-demo-web:1 ExecStop=/usr/bin/docker stop web.%i [X-Fleet] MachineMetadata=provider=digitalocean Conflicts=web@*.service Example of unit file (web.service)

Slide 42

Slide 42 text

Deploy service by fleet $ fleetctl start web.service Unit web.service launched on 06664657.../172.20.20.103 $ fleetctl start instance/*

Slide 43

Slide 43 text

cloud-config cloud-config.yaml Customize CoreOS machines Define service Add & configure user Write config file Define update policy ….

Slide 44

Slide 44 text

Example of cloud-config #cloud-config coreos: etcd: discovery: https://discovery.etcd.io/ addr: $private_ipv4:4001 peer-addr: $private_ipv4:7001 fleet: public-ip: $private_ipv4 metadata: role=web,provider=digitalocean units: - name: etcd.service command: start - name: fleet.service command: start update: group: alpha reboot-strategy: best-effort

Slide 45

Slide 45 text

#cloud-config coreos: etcd: discovery: https://discovery.etcd.io/ addr: $private_ipv4:4001 peer-addr: $private_ipv4:7001 fleet: public-ip: $private_ipv4 metadata: role=web,provider=digitalocean units: - name: etcd.service command: start - name: fleet.service command: start update: group: alpha reboot-strategy: best-effort Example of cloud-config

Slide 46

Slide 46 text

#cloud-config coreos: etcd: discovery: https://discovery.etcd.io/ addr: $private_ipv4:4001 peer-addr: $private_ipv4:7001 fleet: public-ip: $private_ipv4 metadata: role=web,provider=digitalocean units: - name: etcd.service command: start - name: fleet.service command: start update: group: alpha reboot-strategy: best-effort Example of cloud-config

Slide 47

Slide 47 text

#1 CoreOS features #2 CoreOS technologies #3 CoreOS solves… #4 CoreOS operation

Slide 48

Slide 48 text

Docker does NOT provide • Orchestration • Service discovery • Monitoring • Docker host environment parity

Slide 49

Slide 49 text

HOST A HOST B A B HOST C D C E F G How to link two docker container between multiple hosts ? Orchestration

Slide 50

Slide 50 text

HOST A HOST B A B HOST C D C E F G How to link two docker container between multiple hosts ? Orchestration etcd

Slide 51

Slide 51 text

HOST A HOST B A B HOST C D C E F G Where is a service F on container F ? Service discovery

Slide 52

Slide 52 text

HOST A HOST B A B HOST C D C E F G Where is a service F on container F ? Service discovery etcd, fleet

Slide 53

Slide 53 text

Service discovery by fleet $ fleetctl list-units UNIT MACHINE ACTIVE SUB lb.service 4cadccf1.../172.20.20.101 active running web@8001service 4cadccf1.../172.20.20.101 active running web@8002service 4cadccf1.../172.20.20.102 active running web@8003service 4cadccf1.../172.20.20.104 active running grafana.service 06664657.../172.20.20.103 active running influx.service 06664657.../172.20.20.103 active running

Slide 54

Slide 54 text

HOST A HOST B A B HOST C D C E F G H H H I J K L M N O Where should we deploy container H ? Scheduling (It seems that HostB and HostC already has many containers)

Slide 55

Slide 55 text

HOST A HOST B A B HOST C D C E F G H H H I J K L M N O Where should we deploy container H ? Scheduling (It seems that HostB and HostC already has many containers) fleet

Slide 56

Slide 56 text

HOST A HOST B A B HOST C D C E F G Container G is working ? Monitoring

Slide 57

Slide 57 text

HOST A HOST B A B HOST C D C E F G Container G is working ? Monitoring fleet

Slide 58

Slide 58 text

DEV PRODUTION A A B B Docker v1.4 Docker v1.3 Do you use same version of tool on DEV and PROSDUCTION? Docker host environment parity (As I mentioned, we must use orchestration tool, how to ensure we use same version not only docker)

Slide 59

Slide 59 text

DEV PRODUTION A A B B Docker v1.4 Docker v1.3 Do you use same version of tool on DEV and PROSDUCTION? Docker host environment parity (As I mentioned, we must use orchestration tool, how to ensure we use same version not only docker) cloud-config

Slide 60

Slide 60 text

PlayStation: Developing Applications on CoreOS https://www.youtube.com/watch?v=M9hBsRUeRdg

Slide 61

Slide 61 text

#1 CoreOS features #2 CoreOS technologies #3 CoreOS solves… #4 CoreOS operation

Slide 62

Slide 62 text

Demo of CoreOS operation Building CoreOS cluster on DigitalOcean by terraform Build CoreOS cluster Using a demo application container which is just serving static HTML file Deploy demo application by fleet Deploying multiple demo application containers and Load Balancing it By using LB container with confd Scale out demo application

Slide 63

Slide 63 text

Demo of CoreOS operation Building CoreOS cluster on DigitalOcean by terraform Build CoreOS cluster Using a demo application container which is just serving static HTML file Deploy demo application by fleet Deploying multiple demo application containers and Load Balancing it By using LB container with confd Scale out demo application

Slide 64

Slide 64 text

Building CoreOS cluster CoreOS cluster on two different clouds or cloud + bare metal is supported and encouraged. CoreOS runs on almost any platform Cluster size will be changed based on growing service and request loads Cluster size will be changed

Slide 65

Slide 65 text

No content

Slide 66

Slide 66 text

Infrastructure as Code Terraform is a tool for building, changing, and combining infrastructure safely and efficiently.

Slide 67

Slide 67 text

+ Easy to build multiple platform On DigitalOcean, on Amazon EC2 and combine Easy to manage cluster member Just writing .tf file

Slide 68

Slide 68 text

resource "digitalocean_droplet" "web1" { image = "coreos-stable" private_networking = true region = "sgp1" size = "512mb" ssh_keys = ["${var.ssh_key_id}"] user_data = "${file("cloud-config-web.yml")}" } .tf file for building CoreOS cluster on DigitalOcean

Slide 69

Slide 69 text

resource "digitalocean_droplet" "web1" { image = "coreos-stable" private_networking = true region = "sgp1" size = "512mb" ssh_keys = ["${var.ssh_key_id}"] user_data = "${file("cloud-config-web.yml")}" } .tf file for building CoreOS cluster on DigitalOcean

Slide 70

Slide 70 text

resource "digitalocean_droplet" "web1" { image = "coreos-stable" private_networking = true region = "sgp1" size = "512mb" ssh_keys = ["${var.ssh_key_id}"] user_data = "${file("cloud-config-web.yml")}" } .tf file for building CoreOS cluster on DigitalOcean

Slide 71

Slide 71 text

resource "digitalocean_droplet" "web1" { image = "coreos-stable" private_networking = true region = "sgp1" size = "512mb" ssh_keys = ["${var.ssh_key_id}"] user_data = "${file("cloud-config-web.yml")}" } .tf file for building CoreOS cluster on DigitalOcean

Slide 72

Slide 72 text

resource "digitalocean_droplet" "web1" { image = "coreos-stable" private_networking = true region = "sgp1" size = "512mb" ssh_keys = ["${var.ssh_key_id}"] user_data = "${file("cloud-config-web.yml")}" } resource "digitalocean_droplet" "web2" { image = "coreos-stable" private_networking = true region = "sgp1" size = "512mb" ssh_keys = ["${var.ssh_key_id}"] user_data = "${file("cloud-config-web.yml")}" } .tf file for building CoreOS cluster on DigitalOcean

Slide 73

Slide 73 text

Build infrastructure by terraform CLI $ terraform plan $ terraform apply

Slide 74

Slide 74 text

Demo of CoreOS operation Building CoreOS cluster on DigitalOcean by terraform Build CoreOS cluster Using a demo application container which is just serving static HTML file Deploy demo application by fleet Deploying multiple demo application containers and Load Balancing it By using LB container with confd Scale out demo application

Slide 75

Slide 75 text

Demo application container $ docker run -p 8888:80 tcnksm/docker-meetup-demo-web:1 $ curl localhost:8888

DEMO container 1

at Docker meetup #4

Slide 76

Slide 76 text

web.service to deploy demo application container [Unit] Description=docker meetup demo on port %i Requires=docker.service Requires=discovery@%i.service [Service] TimeoutStartSec=0 KillMode=none EnvironmentFile=/etc/environment ExecStartPre=-/usr/bin/docker kill web.%i ExecStartPre=-/usr/bin/docker rm web.%i ExecStartPre=/usr/bin/docker pull tcnksm/docker-meetup-demo-web:1 ExecStart=/usr/bin/docker run --name web.%i -p ${COREOS_PRIVATE_IPV4}:%i:80 tcnksm/docker-meetup-demo-web:1 ExecStop=/usr/bin/docker stop web.%i [X-Fleet] MachineMetadata=provider=digitalocean Conflicts=web@*.service

Slide 77

Slide 77 text

Demo of CoreOS operation Building CoreOS cluster on DigitalOcean by terraform Build CoreOS cluster Using a demo application container which is just serving static HTML file Deploy demo application by fleet Deploying multiple demo application containers and Load Balancing it By using LB container with confd Scale out demo application

Slide 78

Slide 78 text

F W1 CoreOS B :80 We start with a single demo application container.

Slide 79

Slide 79 text

D W1 W2 CoreOS B CoreOS C We will scale out servers and deploy demo application container on each host.

Slide 80

Slide 80 text

CoreOS A D W1 W2 LB CoreOS B CoreOS C :80 :8888 :9999 And load-balancing to them by LB container.

Slide 81

Slide 81 text

CoreOS A D W1 W2 LB CoreOS B CoreOS C :80 :8888 :9999 How to do it? Assuming that the number of containers will be changed dynamically.

Slide 82

Slide 82 text

CoreOS A D F W1 W2 LB CoreOS B CoreOS C :80 :8888 :9999 etcd The easy way is using etcd.

Slide 83

Slide 83 text

CoreOS A D F W1 W2 LB CoreOS B CoreOS C :80 :8888 :9999 etcd D1 D2 1PSU*1 1PSU*1 Discovery service Registering Port & IP of demo app When deploying demo app service, we also deploy discovery service.

Slide 84

Slide 84 text

[Service] EnvironmentFile=/etc/environment ExecStart=/bin/bash -c '\ while true; do \ curl -f ${COREOS_PRIVATE_IPV4}:%i; \ if [ $? -eq 0 ]; then \ etcdctl set /services/docker-meetup-demo-web/${COREOS_PRIVATE_IPV4}\ '${COREOS_PRIVATE_IPV4}:8888\' --ttl 30;\ else \ etcdctl rm /services/docker-meetup-demo-web/${COREOS_PRIVATE_IPV4}; fi; \ sleep 20; \ done' [X-Fleet] MachineOf=web@%i.service discovery.service to register demo app IP & Port to etcd (Excerpt)

Slide 85

Slide 85 text

[Service] EnvironmentFile=/etc/environment ExecStart=/bin/bash -c '\ while true; do \ curl -f ${COREOS_PRIVATE_IPV4}:%i; \ if [ $? -eq 0 ]; then \ etcdctl set /services/docker-meetup-demo-web/${COREOS_PRIVATE_IPV4}\ '${COREOS_PRIVATE_IPV4}:8888\' --ttl 30;\ else \ etcdctl rm /services/docker-meetup-demo-web/${COREOS_PRIVATE_IPV4}; fi; \ sleep 20; \ done' [X-Fleet] MachineOf=web@%i.service discovery.service to register demo app IP & Port to etcd (Excerpt)

Slide 86

Slide 86 text

CoreOS A D F W1 W2 LB CoreOS B CoreOS C :80 :8888 :9999 etcd W1’ W2’ 1PSU*1 1PSU*1 D1 D2 Discovery service Registering Port & IP of demo app { { How to deploy same service on same machine?

Slide 87

Slide 87 text

CoreOS A D F W1 W2 LB CoreOS B CoreOS C :80 :8888 :9999 etcd W1’ W2’ 1PSU*1 1PSU*1 { fleet { 9'MFFU.BDIJOF0G D1 D2 Discovery service Registering Port & IP of demo app We can use fleet (X-Fleet’s MachineOf).

Slide 88

Slide 88 text

[Service] EnvironmentFile=/etc/environment ExecStart=/bin/bash -c '\ while true; do \ curl -f ${COREOS_PRIVATE_IPV4}:%i; \ if [ $? -eq 0 ]; then \ etcdctl set /services/docker-meetup-demo-web/${COREOS_PRIVATE_IPV4}\ '${COREOS_PRIVATE_IPV4}:8888\' --ttl 30;\ else \ etcdctl rm /services/docker-meetup-demo-web/${COREOS_PRIVATE_IPV4}; fi; \ sleep 20; \ done' [X-Fleet] MachineOf=web@%i.service discovery.service to register demo app IP & Port to etcd (Excerpt)

Slide 89

Slide 89 text

CoreOS A D F W1 W2 LB CoreOS B CoreOS C :80 :8888 :9999 etcd W1’ W2’ 1PSU*1 1PSU*1 { fleet { 9'MFFU.BDIJOF0G D1 D2 Discovery service Registering Port & IP of demo app How LB uses etcd value for its configuration ?

Slide 90

Slide 90 text

CoreOS A D F W1 W2 LB CoreOS B CoreOS C :80 :8888 :9999 etcd W1’ W2’ 1PSU*1 1PSU*1 { fleet { 9'MFFU.BDIJOF0G D1 D2 Discovery service Registering Port & IP of demo app We can use confd and generate nginx .conf file dynamically. 1PSU*1 LFMTFZIJHIUPXFSDPOGE Manage local application configuration files using templates and data from etcd

Slide 91

Slide 91 text

tcnksm/docker-meetup-4-demo

Slide 92

Slide 92 text

@deeeet