Upgrade to Pro — share decks privately, control downloads, hide ads and more …

CoreOSクラスタにDockerコンテナをデプロイ #dockerjp

CoreOSクラスタにDockerコンテナをデプロイ #dockerjp

taichi nakashima

January 16, 2015
Tweet

More Decks by taichi nakashima

Other Decks in Technology

Transcript

  1. Deploy
    Docker Containers
    on CoreOS Cluster

    View full-size slide

  2. I’m Taichi Nakashima
    @deeeet tcnksm
    https://www.flickr.com/photos/unforgiven/9278027165

    View full-size slide

  3. • Portability
    • Image distribution/sharing
    • Resource isolation
    Docker provides

    View full-size slide

  4. Docker does NOT provide
    • Orchestration
    • Service discovery
    • Monitoring
    • Docker host environment parity

    View full-size slide

  5. HOST A
    Story of infra growing
    We start with a single host

    View full-size slide

  6. HOST A
    If we use docker, we can see a host like it has many isolated rooms
    Story of infra growing

    View full-size slide

  7. HOST A
    A
    Container A
    Deploying first docker container
    Story of infra growing

    View full-size slide

  8. HOST A
    A
    B
    docker --link
    Within a single host, we can easily link 2 containers
    Story of infra growing

    View full-size slide

  9. HOST A
    A
    B
    C D
    Number of containers will increase as service growing
    Story of infra growing

    View full-size slide

  10. HOST A HOST B HOST C
    F
    A
    B
    C D
    So at some point, we would think about scale out servers
    Story of infra growing

    View full-size slide

  11. HOST A HOST B
    A B
    HOST C
    D
    C
    E
    F G
    And deploying docker containers on multiple host
    Story of infra growing

    View full-size slide

  12. HOST A HOST B
    A B
    HOST C
    D
    C
    E
    F G
    Problems will be happen …
    Story of infra growing

    View full-size slide

  13. HOST A HOST B
    A B
    HOST C
    D
    C
    E
    F G
    How to link two docker container between multiple hosts ?
    Orchestration

    View full-size slide

  14. HOST A HOST B
    A
    B
    HOST C
    D
    C E
    F G
    Where is a service F on container F ?
    Service discovery

    View full-size slide

  15. HOST A HOST B
    A B
    HOST C
    D
    C E
    F G
    H H H
    I
    J
    K
    L
    M
    N
    O
    Where should we deploy container H ?
    Scheduling
    (It seems that HostB and HostC already has many containers)

    View full-size slide

  16. HOST A HOST B
    A B
    HOST C
    D
    C E
    F G
    Container G is working ?
    Monitoring

    View full-size slide

  17. DEV PRODUTION
    A A
    B
    B
    Docker
    v1.4
    Docker
    v1.3
    Do you use same version of tool on DEV and PROSDUCTION?
    Docker host environment parity
    (As I mentioned, we must use orchestration tool, how to ensure
    we use same version not only docker)

    View full-size slide

  18. CoreOS is a new Linux distribution that
    provides scale with high resilience
    infrastructure like Google, Facebook and
    Twitter realize.

    View full-size slide

  19. #1 CoreOS features
    #2 CoreOS technologies
    #3 CoreOS solves…
    #4 CoreOS operation

    View full-size slide

  20. #1 CoreOS features
    #2 CoreOS technologies
    #3 CoreOS solves…
    #4 CoreOS operation
    {• Minimal
    • Docker
    • Update system
    • Clustering

    View full-size slide

  21. Minimal
    Traditional Linux distributions has always
    been about adding value by adding more.
    CoreOS creates value by giving you less.
    http://formtek.com/blog/coreos-linux-vendors-in-the-crosshairs-of-disruption/

    View full-size slide

  22. • No package manager
    • No language runtime
    Docker
    Every application will work on
    Docker container

    View full-size slide

  23. Update System
    Quick and reliable updating system
    Same as update Engine as Chrome (Omahaʣ
    Just Replacing root file system
    Easy to validate, easy to rollback

    View full-size slide

  24. Update System
    Chef, Ansible, Puppet… ??
    • OS … Replace root FS
    • Application … Update docker image
    • Configuration … Update etcd value

    View full-size slide

  25. Clustering
    Clustered By Default
    using powerful and simple tool like etcd and fleet
    Data center as a Computer

    View full-size slide

  26. Data Center as a Computer
    Clustering multiple servers in DC
    and using it like it’s a single computer

    View full-size slide

  27. #1 CoreOS features
    #2 CoreOS technologies
    #3 CoreOS solves…
    #4 CoreOS operation
    {• etcd
    • fleet
    • cloud-config

    View full-size slide

  28. etcd
    e.g., Port & IP of service A
    Leader election (Raft algorithm)
    Distributed key-value store
    Base of clustering
    Save/distribute config values

    View full-size slide

  29. $ etcdctl set /msg hello
    hello
    Setting and getting a key by etcdctl
    $ etcdctl get /msg
    hello

    View full-size slide

  30. $ curl -L http://127.0.0.1:4001/v2/keys/msg \
    -X PUT -d value="hello"
    $ curl -L http://127.0.0.1:4001/v2/keys/msg
    {
    "action":"get",
    "node":{
    "key":"/msg",
    "value":"hello",
    "modifiedIndex":1157552,
    "createdIndex":1157552
    }
    }
    Setting and getting a key by curl

    View full-size slide

  31. fleet Distributed init System
    Confirm parity of desired state and current state
    Place docker container on an appropriate machine
    based on unit file and services which are already worked on cluster
    Scheduling & deploy docker container
    Monitor & failover container service

    View full-size slide

  32. fleet Distributed init System

    View full-size slide

  33. [X-Fleet]
    ɾ.BDIJOF*%
    ɾ$POqJDUT
    Machine identified by the given string.
    ɾ.BDIJOF0G
    ɾ.BDIJOF.FUBEBUB
    ɾ(MPCBM
    Machine that hosts a specific unit.
    Machine with this specific metadata.
    Prevent a unit from being collocated with other units.
    Schedule this unit on all agents in the cluster.

    View full-size slide

  34. Example of unit file (web.service)
    [Unit]
    Description=docker meetup demo on port %i
    Requires=docker.service
    Requires=discovery@%i.service
    [Service]
    TimeoutStartSec=0
    KillMode=none
    EnvironmentFile=/etc/environment
    ExecStartPre=-/usr/bin/docker kill web.%i
    ExecStartPre=-/usr/bin/docker rm web.%i
    ExecStartPre=/usr/bin/docker pull tcnksm/docker-meetup-demo-web:1
    ExecStart=/usr/bin/docker run --name web.%i -p ${COREOS_PRIVATE_IPV4}:%i:80
    tcnksm/docker-meetup-demo-web:1
    ExecStop=/usr/bin/docker stop web.%i
    [X-Fleet]
    MachineMetadata=provider=digitalocean
    Conflicts=web@*.service

    View full-size slide

  35. [Unit]
    Description=docker meetup demo on port %i
    Requires=docker.service
    Requires=discovery@%i.service
    [Service]
    TimeoutStartSec=0
    KillMode=none
    EnvironmentFile=/etc/environment
    ExecStartPre=-/usr/bin/docker kill web.%i
    ExecStartPre=-/usr/bin/docker rm web.%i
    ExecStartPre=/usr/bin/docker pull tcnksm/docker-meetup-demo-web:1
    ExecStart=/usr/bin/docker run --name web.%i -p ${COREOS_PRIVATE_IPV4}:%i:80
    tcnksm/docker-meetup-demo-web:1
    ExecStop=/usr/bin/docker stop web.%i
    [X-Fleet]
    MachineMetadata=provider=digitalocean
    Conflicts=web@*.service
    Example of unit file (web.service)

    View full-size slide

  36. [Unit]
    Description=docker meetup demo on port %i
    Requires=docker.service
    Requires=discovery@%i.service
    [Service]
    TimeoutStartSec=0
    KillMode=none
    EnvironmentFile=/etc/environment
    ExecStartPre=-/usr/bin/docker kill web.%i
    ExecStartPre=-/usr/bin/docker rm web.%i
    ExecStartPre=/usr/bin/docker pull tcnksm/docker-meetup-demo-web:1
    ExecStart=/usr/bin/docker run --name web.%i -p ${COREOS_PRIVATE_IPV4}:%i:80
    tcnksm/docker-meetup-demo-web:1
    ExecStop=/usr/bin/docker stop web.%i
    [X-Fleet]
    MachineMetadata=provider=digitalocean
    Conflicts=web@*.service
    Example of unit file (web.service)

    View full-size slide

  37. [Unit]
    Description=docker meetup demo on port %i
    Requires=docker.service
    Requires=discovery@%i.service
    [Service]
    TimeoutStartSec=0
    KillMode=none
    EnvironmentFile=/etc/environment
    ExecStartPre=-/usr/bin/docker kill web.%i
    ExecStartPre=-/usr/bin/docker rm web.%i
    ExecStartPre=/usr/bin/docker pull tcnksm/docker-meetup-demo-web:1
    ExecStart=/usr/bin/docker run --name web.%i -p ${COREOS_PRIVATE_IPV4}:%i:80
    tcnksm/docker-meetup-demo-web:1
    ExecStop=/usr/bin/docker stop web.%i
    [X-Fleet]
    MachineMetadata=provider=digitalocean
    Conflicts=web@*.service
    Example of unit file (web.service)

    View full-size slide

  38. Deploy service by fleet
    $ fleetctl start web.service
    Unit web.service launched on 06664657.../172.20.20.103
    $ fleetctl start instance/*

    View full-size slide

  39. cloud-config
    cloud-config.yaml
    Customize CoreOS machines
    Define service
    Add & configure user
    Write config file
    Define update policy
    ….

    View full-size slide

  40. Example of cloud-config
    #cloud-config
    coreos:
    etcd:
    discovery: https://discovery.etcd.io/
    addr: $private_ipv4:4001
    peer-addr: $private_ipv4:7001
    fleet:
    public-ip: $private_ipv4
    metadata: role=web,provider=digitalocean
    units:
    - name: etcd.service
    command: start
    - name: fleet.service
    command: start
    update:
    group: alpha
    reboot-strategy: best-effort

    View full-size slide

  41. #cloud-config
    coreos:
    etcd:
    discovery: https://discovery.etcd.io/
    addr: $private_ipv4:4001
    peer-addr: $private_ipv4:7001
    fleet:
    public-ip: $private_ipv4
    metadata: role=web,provider=digitalocean
    units:
    - name: etcd.service
    command: start
    - name: fleet.service
    command: start
    update:
    group: alpha
    reboot-strategy: best-effort
    Example of cloud-config

    View full-size slide

  42. #cloud-config
    coreos:
    etcd:
    discovery: https://discovery.etcd.io/
    addr: $private_ipv4:4001
    peer-addr: $private_ipv4:7001
    fleet:
    public-ip: $private_ipv4
    metadata: role=web,provider=digitalocean
    units:
    - name: etcd.service
    command: start
    - name: fleet.service
    command: start
    update:
    group: alpha
    reboot-strategy: best-effort
    Example of cloud-config

    View full-size slide

  43. #1 CoreOS features
    #2 CoreOS technologies
    #3 CoreOS solves…
    #4 CoreOS operation

    View full-size slide

  44. Docker does NOT provide
    • Orchestration
    • Service discovery
    • Monitoring
    • Docker host environment parity

    View full-size slide

  45. HOST A HOST B
    A B
    HOST C
    D
    C
    E
    F G
    How to link two docker container between multiple hosts ?
    Orchestration

    View full-size slide

  46. HOST A HOST B
    A B
    HOST C
    D
    C
    E
    F G
    How to link two docker container between multiple hosts ?
    Orchestration
    etcd

    View full-size slide

  47. HOST A HOST B
    A
    B
    HOST C
    D
    C E
    F G
    Where is a service F on container F ?
    Service discovery

    View full-size slide

  48. HOST A HOST B
    A
    B
    HOST C
    D
    C E
    F G
    Where is a service F on container F ?
    Service discovery
    etcd, fleet

    View full-size slide

  49. Service discovery by fleet
    $ fleetctl list-units
    UNIT MACHINE ACTIVE SUB
    lb.service 4cadccf1.../172.20.20.101 active running
    web@8001service 4cadccf1.../172.20.20.101 active running
    web@8002service 4cadccf1.../172.20.20.102 active running
    web@8003service 4cadccf1.../172.20.20.104 active running
    grafana.service 06664657.../172.20.20.103 active running
    influx.service 06664657.../172.20.20.103 active running

    View full-size slide

  50. HOST A HOST B
    A B
    HOST C
    D
    C E
    F G
    H H H
    I
    J
    K
    L
    M
    N
    O
    Where should we deploy container H ?
    Scheduling
    (It seems that HostB and HostC already has many containers)

    View full-size slide

  51. HOST A HOST B
    A B
    HOST C
    D
    C E
    F G
    H H H
    I
    J
    K
    L
    M
    N
    O
    Where should we deploy container H ?
    Scheduling
    (It seems that HostB and HostC already has many containers)
    fleet

    View full-size slide

  52. HOST A HOST B
    A B
    HOST C
    D
    C E
    F G
    Container G is working ?
    Monitoring

    View full-size slide

  53. HOST A HOST B
    A B
    HOST C
    D
    C E
    F G
    Container G is working ?
    Monitoring
    fleet

    View full-size slide

  54. DEV PRODUTION
    A A
    B
    B
    Docker
    v1.4
    Docker
    v1.3
    Do you use same version of tool on DEV and PROSDUCTION?
    Docker host environment parity
    (As I mentioned, we must use orchestration tool, how to ensure
    we use same version not only docker)

    View full-size slide

  55. DEV PRODUTION
    A A
    B
    B
    Docker
    v1.4
    Docker
    v1.3
    Do you use same version of tool on DEV and PROSDUCTION?
    Docker host environment parity
    (As I mentioned, we must use orchestration tool, how to ensure
    we use same version not only docker)
    cloud-config

    View full-size slide

  56. PlayStation: Developing Applications on CoreOS
    https://www.youtube.com/watch?v=M9hBsRUeRdg

    View full-size slide

  57. #1 CoreOS features
    #2 CoreOS technologies
    #3 CoreOS solves…
    #4 CoreOS operation

    View full-size slide

  58. Demo of CoreOS operation
    Building CoreOS cluster on DigitalOcean by terraform
    Build CoreOS cluster
    Using a demo application container which is just serving static HTML file
    Deploy demo application by fleet
    Deploying multiple demo application containers and Load Balancing it
    By using LB container with confd
    Scale out demo application

    View full-size slide

  59. Demo of CoreOS operation
    Building CoreOS cluster on DigitalOcean by terraform
    Build CoreOS cluster
    Using a demo application container which is just serving static HTML file
    Deploy demo application by fleet
    Deploying multiple demo application containers and Load Balancing it
    By using LB container with confd
    Scale out demo application

    View full-size slide

  60. Building CoreOS cluster
    CoreOS cluster on two different clouds or cloud + bare metal is supported and
    encouraged.
    CoreOS runs on almost any platform
    Cluster size will be changed based on growing service and request loads
    Cluster size will be changed

    View full-size slide

  61. Infrastructure as Code
    Terraform is a tool for building, changing,
    and combining infrastructure safely and
    efficiently.

    View full-size slide

  62. +
    Easy to build multiple platform
    On DigitalOcean, on Amazon EC2 and combine
    Easy to manage cluster member
    Just writing .tf file

    View full-size slide

  63. resource "digitalocean_droplet" "web1" {
    image = "coreos-stable"
    private_networking = true
    region = "sgp1"
    size = "512mb"
    ssh_keys = ["${var.ssh_key_id}"]
    user_data = "${file("cloud-config-web.yml")}"
    }
    .tf file for building CoreOS cluster on DigitalOcean

    View full-size slide

  64. resource "digitalocean_droplet" "web1" {
    image = "coreos-stable"
    private_networking = true
    region = "sgp1"
    size = "512mb"
    ssh_keys = ["${var.ssh_key_id}"]
    user_data = "${file("cloud-config-web.yml")}"
    }
    .tf file for building CoreOS cluster on DigitalOcean

    View full-size slide

  65. resource "digitalocean_droplet" "web1" {
    image = "coreos-stable"
    private_networking = true
    region = "sgp1"
    size = "512mb"
    ssh_keys = ["${var.ssh_key_id}"]
    user_data = "${file("cloud-config-web.yml")}"
    }
    .tf file for building CoreOS cluster on DigitalOcean

    View full-size slide

  66. resource "digitalocean_droplet" "web1" {
    image = "coreos-stable"
    private_networking = true
    region = "sgp1"
    size = "512mb"
    ssh_keys = ["${var.ssh_key_id}"]
    user_data = "${file("cloud-config-web.yml")}"
    }
    .tf file for building CoreOS cluster on DigitalOcean

    View full-size slide

  67. resource "digitalocean_droplet" "web1" {
    image = "coreos-stable"
    private_networking = true
    region = "sgp1"
    size = "512mb"
    ssh_keys = ["${var.ssh_key_id}"]
    user_data = "${file("cloud-config-web.yml")}"
    }
    resource "digitalocean_droplet" "web2" {
    image = "coreos-stable"
    private_networking = true
    region = "sgp1"
    size = "512mb"
    ssh_keys = ["${var.ssh_key_id}"]
    user_data = "${file("cloud-config-web.yml")}"
    }
    .tf file for building CoreOS cluster on DigitalOcean

    View full-size slide

  68. Build infrastructure by terraform CLI
    $ terraform plan
    $ terraform apply

    View full-size slide

  69. Demo of CoreOS operation
    Building CoreOS cluster on DigitalOcean by terraform
    Build CoreOS cluster
    Using a demo application container which is just serving static HTML file
    Deploy demo application by fleet
    Deploying multiple demo application containers and Load Balancing it
    By using LB container with confd
    Scale out demo application

    View full-size slide

  70. Demo application container
    $ docker run -p 8888:80 tcnksm/docker-meetup-demo-web:1
    $ curl localhost:8888
    DEMO container 1at Docker meetup #4

    View full-size slide

  71. web.service to deploy demo application container
    [Unit]
    Description=docker meetup demo on port %i
    Requires=docker.service
    Requires=discovery@%i.service
    [Service]
    TimeoutStartSec=0
    KillMode=none
    EnvironmentFile=/etc/environment
    ExecStartPre=-/usr/bin/docker kill web.%i
    ExecStartPre=-/usr/bin/docker rm web.%i
    ExecStartPre=/usr/bin/docker pull tcnksm/docker-meetup-demo-web:1
    ExecStart=/usr/bin/docker run --name web.%i -p ${COREOS_PRIVATE_IPV4}:%i:80
    tcnksm/docker-meetup-demo-web:1
    ExecStop=/usr/bin/docker stop web.%i
    [X-Fleet]
    MachineMetadata=provider=digitalocean
    Conflicts=web@*.service

    View full-size slide

  72. Demo of CoreOS operation
    Building CoreOS cluster on DigitalOcean by terraform
    Build CoreOS cluster
    Using a demo application container which is just serving static HTML file
    Deploy demo application by fleet
    Deploying multiple demo application containers and Load Balancing it
    By using LB container with confd
    Scale out demo application

    View full-size slide

  73. F
    W1
    CoreOS B
    :80
    We start with a single demo application container.

    View full-size slide

  74. D
    W1
    W2
    CoreOS B CoreOS C
    We will scale out servers and deploy demo application container on each host.

    View full-size slide

  75. CoreOS A
    D
    W1
    W2
    LB
    CoreOS B CoreOS C
    :80
    :8888
    :9999
    And load-balancing to them by LB container.

    View full-size slide

  76. CoreOS A
    D
    W1
    W2
    LB
    CoreOS B CoreOS C
    :80
    :8888
    :9999
    How to do it? Assuming that the number of containers will be changed dynamically.

    View full-size slide

  77. CoreOS A
    D
    F
    W1
    W2
    LB
    CoreOS B CoreOS C
    :80
    :8888
    :9999
    etcd
    The easy way is using etcd.

    View full-size slide

  78. CoreOS A
    D
    F
    W1
    W2
    LB
    CoreOS B CoreOS C
    :80
    :8888
    :9999
    etcd
    D1
    D2
    1PSU*1
    1PSU*1
    Discovery service
    Registering Port & IP of demo app
    When deploying demo app service, we also deploy discovery service.

    View full-size slide

  79. [Service]
    EnvironmentFile=/etc/environment
    ExecStart=/bin/bash -c '\
    while true; do \
    curl -f ${COREOS_PRIVATE_IPV4}:%i; \
    if [ $? -eq 0 ]; then \
    etcdctl set /services/docker-meetup-demo-web/${COREOS_PRIVATE_IPV4}\
    '${COREOS_PRIVATE_IPV4}:8888\' --ttl 30;\
    else \
    etcdctl rm /services/docker-meetup-demo-web/${COREOS_PRIVATE_IPV4};
    fi; \
    sleep 20; \
    done'
    [X-Fleet]
    MachineOf=web@%i.service
    discovery.service to register demo app IP & Port to etcd (Excerpt)

    View full-size slide

  80. [Service]
    EnvironmentFile=/etc/environment
    ExecStart=/bin/bash -c '\
    while true; do \
    curl -f ${COREOS_PRIVATE_IPV4}:%i; \
    if [ $? -eq 0 ]; then \
    etcdctl set /services/docker-meetup-demo-web/${COREOS_PRIVATE_IPV4}\
    '${COREOS_PRIVATE_IPV4}:8888\' --ttl 30;\
    else \
    etcdctl rm /services/docker-meetup-demo-web/${COREOS_PRIVATE_IPV4};
    fi; \
    sleep 20; \
    done'
    [X-Fleet]
    MachineOf=web@%i.service
    discovery.service to register demo app IP & Port to etcd (Excerpt)

    View full-size slide

  81. CoreOS A
    D
    F
    W1
    W2
    LB
    CoreOS B CoreOS C
    :80
    :8888
    :9999
    etcd
    W1’
    W2’
    1PSU*1
    1PSU*1
    D1
    D2
    Discovery service
    Registering Port & IP of demo app
    {
    {
    How to deploy same service on same machine?

    View full-size slide

  82. CoreOS A
    D
    F
    W1
    W2
    LB
    CoreOS B CoreOS C
    :80
    :8888
    :9999
    etcd
    W1’
    W2’
    1PSU*1
    1PSU*1
    {
    fleet
    {
    9'MFFU.BDIJOF0G
    D1
    D2
    Discovery service
    Registering Port & IP of demo app
    We can use fleet (X-Fleet’s MachineOf).

    View full-size slide

  83. [Service]
    EnvironmentFile=/etc/environment
    ExecStart=/bin/bash -c '\
    while true; do \
    curl -f ${COREOS_PRIVATE_IPV4}:%i; \
    if [ $? -eq 0 ]; then \
    etcdctl set /services/docker-meetup-demo-web/${COREOS_PRIVATE_IPV4}\
    '${COREOS_PRIVATE_IPV4}:8888\' --ttl 30;\
    else \
    etcdctl rm /services/docker-meetup-demo-web/${COREOS_PRIVATE_IPV4};
    fi; \
    sleep 20; \
    done'
    [X-Fleet]
    MachineOf=web@%i.service
    discovery.service to register demo app IP & Port to etcd (Excerpt)

    View full-size slide

  84. CoreOS A
    D
    F
    W1
    W2
    LB
    CoreOS B CoreOS C
    :80
    :8888
    :9999
    etcd
    W1’
    W2’
    1PSU*1
    1PSU*1
    {
    fleet
    {
    9'MFFU.BDIJOF0G
    D1
    D2
    Discovery service
    Registering Port & IP of demo app
    How LB uses etcd value for its configuration ?

    View full-size slide

  85. CoreOS A
    D
    F
    W1
    W2
    LB
    CoreOS B CoreOS C
    :80
    :8888
    :9999
    etcd
    W1’
    W2’
    1PSU*1
    1PSU*1
    {
    fleet
    {
    9'MFFU.BDIJOF0G
    D1
    D2
    Discovery service
    Registering Port & IP of demo app
    We can use confd and generate nginx .conf file dynamically.
    1PSU*1
    LFMTFZIJHIUPXFSDPOGE
    Manage local application configuration
    files using templates and data from etcd

    View full-size slide

  86. tcnksm/docker-meetup-4-demo

    View full-size slide