Tweet
Tweet

Slide 1

Slide 1 text

Monitor Docker Containers on CoreOS cluster

Slide 2

Slide 2 text

I’m Taichi Nakashima @deeeet tcnksm https://www.flickr.com/photos/unforgiven/9278027165

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

What is difference? Traditional stack vs. Containers

Slide 8

Slide 8 text

Physical (Machines) Virtual Virtual Service A Service B Service C Service D

Slide 9

Slide 9 text

Physical (Machines) Virtual Virtual Container Container Container Container Service A Service B Service C Service D

Slide 10

Slide 10 text

Monitoring by Yourself Have your own monitoring system Monitoring as a Service Use external service

Slide 11

Slide 11 text

Monitoring by Yourself Have your own monitoring system Monitoring as a Service Use external service

Slide 12

Slide 12 text

For a single host monitoring cAdvisor For a cluster scale monitoring Heapster Monitoring by Yourself Have your own monitoring system

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

Just running dockerized cAdvisor container Collect all container metrics on a host Just access to :8080 in your browser Provide Web UI cAdvisor

Slide 15

Slide 15 text

Run cAdvisor container $ docker run \ --volume=/:/rootfs:ro \ --volume=/var/run:/var/run:rw \ --volume=/sys:/sys:ro \ --volume=/var/lib/docker/:/var/lib/docker:ro \ --publish=8080:8080 \ --detach=true \ --name=cadvisor \ google/cadvisor:latest

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

Collect cAdvisor metrics from cluster member, it’s used in Kubernetes Enables cluster wide monitoring of containers Draw graph by Grafana Support InfluxDB backend Heapster

Slide 18

Slide 18 text

Container

Slide 19

Slide 19 text

Container

Slide 20

Slide 20 text

Container Heapster

Slide 21

Slide 21 text

Container Heapster 

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

Monitoring by Yourself Have your own monitoring system Monitoring as a Service Use external service

Slide 24

Slide 24 text

No content

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

But No README and No document… stanaka/mackerel-docker Mackerel Not Support container specific feature No container specific monitoring

Slide 28

Slide 28 text

New Relic Dockerized collector agent but only for host metrics not for containers, To monitor each container we need to install it on each our docker image johanneswuerbach/newrelic-sysmond Not Support container specific feature No container specific monitoring

Slide 29

Slide 29 text

DataDog Dockerized collector agent, just run docker container DataDog/docker-dd-agent Support container specific feature !! Container metrics, Tagging, Lifecycle of container, etc

Slide 30

Slide 30 text

DataDog Container feature What is good point and why ?

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

DataDog container feature Agent tags by docker container name and its image name (by default) Tagging All containers on a host If you run 1 dd-agent container, it monitors all containers in the host It collects each container’s CPU, memory, network I/O and disk I/O (General) Lifecycle monitoring Agent also monitor container create, start, stop, destroy events

Slide 33

Slide 33 text

DataDog container feature Agent tags by docker container name and its image name (by default) Tagging All containers on a host If you run 1 dd-agent container, it monitors all containers in the host It collects each container’s CPU, memory, network I/O and disk I/O (General) Lifecycle monitoring Agent also monitor container create, start, stop, destroy events Easy to start

Slide 34

Slide 34 text

Run dd-agent container $ docker run \ --privileged \ --name dd-agent \ -h `hostname` \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /proc/mounts:/host/proc/mounts:ro \ -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \ -e API_KEY={your_api_key_here} \ datadog/docker-dd-agent

Slide 35

Slide 35 text

Container dd-agent

Slide 36

Slide 36 text

To dd-agent container on CoreOS cluster [Unit] … [Service] TimeoutStartSec=0 ExecStartPre=-/usr/bin/docker kill dd-agent ExecStartPre=-/usr/bin/docker rm dd-agent ExecStartPre=/usr/bin/docker pull datadog/docker-dd-agent ExecStart=/usr/bin/bash -c \ "/usr/bin/docker run --privileged --name dd-agent -h `hostname` \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /proc/mounts:/host/proc/mounts:ro \ -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \ -e API_KEY=`YOUR_API_KEY` \ datadog/docker-dd-agent" [X-Fleet] Global=true

Slide 37

Slide 37 text

To dd-agent container on CoreOS cluster [Unit] … [Service] TimeoutStartSec=0 ExecStartPre=-/usr/bin/docker kill dd-agent ExecStartPre=-/usr/bin/docker rm dd-agent ExecStartPre=/usr/bin/docker pull datadog/docker-dd-agent ExecStart=/usr/bin/bash -c \ "/usr/bin/docker run --privileged --name dd-agent -h `hostname` \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /proc/mounts:/host/proc/mounts:ro \ -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \ -e API_KEY=`YOUR_API_KEY` \ datadog/docker-dd-agent" [X-Fleet] Global=true

Slide 38

Slide 38 text

To dd-agent container on CoreOS cluster [Unit] … [Service] TimeoutStartSec=0 ExecStartPre=-/usr/bin/docker kill dd-agent ExecStartPre=-/usr/bin/docker rm dd-agent ExecStartPre=/usr/bin/docker pull datadog/docker-dd-agent ExecStart=/usr/bin/bash -c \ "/usr/bin/docker run --privileged --name dd-agent -h `hostname` \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /proc/mounts:/host/proc/mounts:ro \ -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \ -e API_KEY=`YOUR_API_KEY` \ datadog/docker-dd-agent" [X-Fleet] Global=true

Slide 39

Slide 39 text

To dd-agent container on CoreOS cluster $ fleetctl start dd-agent.service Run dd-agent.service by fleet

Slide 40

Slide 40 text

DataDog container feature Agent tags by docker container name and its image name (by default) Tagging All containers on a host If you run 1 dd-agent container, it monitors all containers in the host It collects each container’s CPU, memory, network I/O and disk I/O (General) Lifecycle monitoring Agent also monitor container create, start, stop, destroy events Easy to explore what you want

Slide 41

Slide 41 text

DataDog container feature Agent tags by docker container name and its image name (by default) Tagging All containers on a host If you run 1 dd-agent container, it monitors all containers in the host It collects each container’s CPU, memory, network I/O and disk I/O (General) Lifecycle monitoring Agent also monitor container create, start, stop, destroy events Help understand unexpected value

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

No content

Slide 45

Slide 45 text

New containers are created

Slide 46

Slide 46 text

DataDog container feature Agent tags by docker container name and its image name (by default) Tagging All containers on a host If you run 1 dd-agent container, it monitors all containers in the host It collects each container’s CPU, memory, network I/O and disk I/O (General) Lifecycle monitoring Agent also monitor container create, start, stop, destroy events Easy to start Easy to explore what you want Help understand unexpected value

Slide 47

Slide 47 text

Requirement for container monitoring Common part of recent trend

Slide 48

Slide 48 text

Container Heapster 

Slide 49

Slide 49 text

Container dd-agent

Slide 50

Slide 50 text

Install agent each container is not good idea, keep container simple ! All containers in a host by 1 agent Requirement for container monitoring Containerized Agent Container only approach is Docker-way (CoreOS)

Slide 51

Slide 51 text

Manage secret values on distributed KVS Save API token on etcd/consul Extra edition

Slide 52

Slide 52 text

To dd-agent container on CoreOS cluster [Unit] … [Service] TimeoutStartSec=0 ExecStartPre=-/usr/bin/docker kill dd-agent ExecStartPre=-/usr/bin/docker rm dd-agent ExecStartPre=/usr/bin/docker pull datadog/docker-dd-agent ExecStart=/usr/bin/bash -c \ "/usr/bin/docker run --privileged --name dd-agent -h `hostname` \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /proc/mounts:/host/proc/mounts:ro \ -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \ -e API_KEY=`YOUR_API_KEY` \ datadog/docker-dd-agent" [X-Fleet] Global=true

Slide 53

Slide 53 text

xordataexchange/crypt

Slide 54

Slide 54 text

Generate pub and sub key $ gpg2 --gen-key # generate pub.gpg and secret.gpg

Slide 55

Slide 55 text

Save json value on etcd by crypt $ cat < config.json {"test": "passw0rd"} EOF $ crypt set -keyring=pub.gpg /app/config config.json

Slide 56

Slide 56 text

Get value withtout crypt and secret-key $ etcdctl get /app/config wcBMA0OL+oKDi4zdAQgAh7iKVASBZvvX6WiiLPYSZgAbhYDhZyVGqX +uK2Bc1plC/mYkqw/n3FXyL+ZC0ISdK9Hdqv6HpCthnMHmBCfhPAjV4 DsrXKWO7TP0AYTxUPMxX9sIiTzrLTJGb73134Z6l0z0Ocj2dEuhyAt5u 3cucKkQb3CWGyuhM7C02aTeJoPjIkqi3agAizQn0uwcurSONpmCkArq33 3579iHZv42Xnr+1Dq4CkcDG9OYPyKcoixOvvW9OpB1E

Slide 57

Slide 57 text

Get value from etcd by crypt and key $ crypt get -secret-keyring secret.gpg /app/config {"test":"passw0rd"}

Slide 58

Slide 58 text

To dd-agent container on CoreOS cluster [Unit] … [Service] TimeoutStartSec=0 ExecStartPre=-/usr/bin/docker kill dd-agent ExecStartPre=-/usr/bin/docker rm dd-agent ExecStartPre=/usr/bin/docker pull datadog/docker-dd-agent ExecStart=/usr/bin/bash -c \ "/usr/bin/docker run --privileged --name dd-agent -h `hostname` \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /proc/mounts:/host/proc/mounts:ro \ -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \ -e API_KEY=`crypt get -secret-keyring /etc/secret.gpg /ddapikey` \ datadog/docker-dd-agent" [X-Fleet] Global=true

Slide 59

Slide 59 text

@deeeet