Monitor Docker Containers on CoreOS cluster #monitoringcasual

Ecb3acc2d246962361a4f8b3f7a6dd12?s=47 taichi nakashima
January 30, 2015
940

Monitor Docker Containers on CoreOS cluster #monitoringcasual

Ecb3acc2d246962361a4f8b3f7a6dd12?s=128

taichi nakashima

January 30, 2015
Tweet

Transcript

  1. Monitor Docker Containers on CoreOS cluster

  2. I’m Taichi Nakashima @deeeet tcnksm https://www.flickr.com/photos/unforgiven/9278027165

  3. None
  4. None
  5. None
  6. None
  7. What is difference? Traditional stack vs. Containers

  8. Physical (Machines) Virtual Virtual Service A Service B Service C

    Service D
  9. Physical (Machines) Virtual Virtual Container Container Container Container Service A

    Service B Service C Service D
  10. Monitoring by Yourself Have your own monitoring system Monitoring as

    a Service Use external service
  11. Monitoring by Yourself Have your own monitoring system Monitoring as

    a Service Use external service
  12. For a single host monitoring cAdvisor For a cluster scale

    monitoring Heapster Monitoring by Yourself Have your own monitoring system
  13. None
  14. Just running dockerized cAdvisor container Collect all container metrics on

    a host Just access to :8080 in your browser Provide Web UI cAdvisor
  15. Run cAdvisor container $ docker run \ --volume=/:/rootfs:ro \ --volume=/var/run:/var/run:rw

    \ --volume=/sys:/sys:ro \ --volume=/var/lib/docker/:/var/lib/docker:ro \ --publish=8080:8080 \ --detach=true \ --name=cadvisor \ google/cadvisor:latest
  16. None
  17. Collect cAdvisor metrics from cluster member, it’s used in Kubernetes

    Enables cluster wide monitoring of containers Draw graph by Grafana Support InfluxDB backend Heapster
  18. Container

  19. Container

  20. Container Heapster

  21. Container Heapster 

  22. None
  23. Monitoring by Yourself Have your own monitoring system Monitoring as

    a Service Use external service
  24. None
  25. None
  26. None
  27. But No README and No document… stanaka/mackerel-docker Mackerel Not Support

    container specific feature No container specific monitoring
  28. New Relic Dockerized collector agent but only for host metrics

    not for containers, To monitor each container we need to install it on each our docker image johanneswuerbach/newrelic-sysmond Not Support container specific feature No container specific monitoring
  29. DataDog Dockerized collector agent, just run docker container DataDog/docker-dd-agent Support

    container specific feature !! Container metrics, Tagging, Lifecycle of container, etc
  30. DataDog Container feature What is good point and why ?

  31. None
  32. DataDog container feature Agent tags by docker container name and

    its image name (by default) Tagging All containers on a host If you run 1 dd-agent container, it monitors all containers in the host It collects each container’s CPU, memory, network I/O and disk I/O (General) Lifecycle monitoring Agent also monitor container create, start, stop, destroy events
  33. DataDog container feature Agent tags by docker container name and

    its image name (by default) Tagging All containers on a host If you run 1 dd-agent container, it monitors all containers in the host It collects each container’s CPU, memory, network I/O and disk I/O (General) Lifecycle monitoring Agent also monitor container create, start, stop, destroy events Easy to start
  34. Run dd-agent container $ docker run \ --privileged \ --name

    dd-agent \ -h `hostname` \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /proc/mounts:/host/proc/mounts:ro \ -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \ -e API_KEY={your_api_key_here} \ datadog/docker-dd-agent
  35. Container dd-agent

  36. To dd-agent container on CoreOS cluster [Unit] … [Service] TimeoutStartSec=0

    ExecStartPre=-/usr/bin/docker kill dd-agent ExecStartPre=-/usr/bin/docker rm dd-agent ExecStartPre=/usr/bin/docker pull datadog/docker-dd-agent ExecStart=/usr/bin/bash -c \ "/usr/bin/docker run --privileged --name dd-agent -h `hostname` \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /proc/mounts:/host/proc/mounts:ro \ -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \ -e API_KEY=`YOUR_API_KEY` \ datadog/docker-dd-agent" [X-Fleet] Global=true
  37. To dd-agent container on CoreOS cluster [Unit] … [Service] TimeoutStartSec=0

    ExecStartPre=-/usr/bin/docker kill dd-agent ExecStartPre=-/usr/bin/docker rm dd-agent ExecStartPre=/usr/bin/docker pull datadog/docker-dd-agent ExecStart=/usr/bin/bash -c \ "/usr/bin/docker run --privileged --name dd-agent -h `hostname` \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /proc/mounts:/host/proc/mounts:ro \ -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \ -e API_KEY=`YOUR_API_KEY` \ datadog/docker-dd-agent" [X-Fleet] Global=true
  38. To dd-agent container on CoreOS cluster [Unit] … [Service] TimeoutStartSec=0

    ExecStartPre=-/usr/bin/docker kill dd-agent ExecStartPre=-/usr/bin/docker rm dd-agent ExecStartPre=/usr/bin/docker pull datadog/docker-dd-agent ExecStart=/usr/bin/bash -c \ "/usr/bin/docker run --privileged --name dd-agent -h `hostname` \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /proc/mounts:/host/proc/mounts:ro \ -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \ -e API_KEY=`YOUR_API_KEY` \ datadog/docker-dd-agent" [X-Fleet] Global=true
  39. To dd-agent container on CoreOS cluster $ fleetctl start dd-agent.service

    Run dd-agent.service by fleet
  40. DataDog container feature Agent tags by docker container name and

    its image name (by default) Tagging All containers on a host If you run 1 dd-agent container, it monitors all containers in the host It collects each container’s CPU, memory, network I/O and disk I/O (General) Lifecycle monitoring Agent also monitor container create, start, stop, destroy events Easy to explore what you want
  41. DataDog container feature Agent tags by docker container name and

    its image name (by default) Tagging All containers on a host If you run 1 dd-agent container, it monitors all containers in the host It collects each container’s CPU, memory, network I/O and disk I/O (General) Lifecycle monitoring Agent also monitor container create, start, stop, destroy events Help understand unexpected value
  42. None
  43. None
  44. None
  45. New containers are created

  46. DataDog container feature Agent tags by docker container name and

    its image name (by default) Tagging All containers on a host If you run 1 dd-agent container, it monitors all containers in the host It collects each container’s CPU, memory, network I/O and disk I/O (General) Lifecycle monitoring Agent also monitor container create, start, stop, destroy events Easy to start Easy to explore what you want Help understand unexpected value
  47. Requirement for container monitoring Common part of recent trend

  48. Container Heapster 

  49. Container dd-agent

  50. Install agent each container is not good idea, keep container

    simple ! All containers in a host by 1 agent Requirement for container monitoring Containerized Agent Container only approach is Docker-way (CoreOS)
  51. Manage secret values on distributed KVS Save API token on

    etcd/consul Extra edition
  52. To dd-agent container on CoreOS cluster [Unit] … [Service] TimeoutStartSec=0

    ExecStartPre=-/usr/bin/docker kill dd-agent ExecStartPre=-/usr/bin/docker rm dd-agent ExecStartPre=/usr/bin/docker pull datadog/docker-dd-agent ExecStart=/usr/bin/bash -c \ "/usr/bin/docker run --privileged --name dd-agent -h `hostname` \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /proc/mounts:/host/proc/mounts:ro \ -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \ -e API_KEY=`YOUR_API_KEY` \ datadog/docker-dd-agent" [X-Fleet] Global=true
  53. xordataexchange/crypt

  54. Generate pub and sub key $ gpg2 --gen-key # generate

    pub.gpg and secret.gpg
  55. Save json value on etcd by crypt $ cat <<EOF

    > config.json {"test": "passw0rd"} EOF $ crypt set -keyring=pub.gpg /app/config config.json
  56. Get value withtout crypt and secret-key $ etcdctl get /app/config

    wcBMA0OL+oKDi4zdAQgAh7iKVASBZvvX6WiiLPYSZgAbhYDhZyVGqX +uK2Bc1plC/mYkqw/n3FXyL+ZC0ISdK9Hdqv6HpCthnMHmBCfhPAjV4 DsrXKWO7TP0AYTxUPMxX9sIiTzrLTJGb73134Z6l0z0Ocj2dEuhyAt5u 3cucKkQb3CWGyuhM7C02aTeJoPjIkqi3agAizQn0uwcurSONpmCkArq33 3579iHZv42Xnr+1Dq4CkcDG9OYPyKcoixOvvW9OpB1E
  57. Get value from etcd by crypt and key $ crypt

    get -secret-keyring secret.gpg /app/config {"test":"passw0rd"}
  58. To dd-agent container on CoreOS cluster [Unit] … [Service] TimeoutStartSec=0

    ExecStartPre=-/usr/bin/docker kill dd-agent ExecStartPre=-/usr/bin/docker rm dd-agent ExecStartPre=/usr/bin/docker pull datadog/docker-dd-agent ExecStart=/usr/bin/bash -c \ "/usr/bin/docker run --privileged --name dd-agent -h `hostname` \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /proc/mounts:/host/proc/mounts:ro \ -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \ -e API_KEY=`crypt get -secret-keyring /etc/secret.gpg /ddapikey` \ datadog/docker-dd-agent" [X-Fleet] Global=true
  59. @deeeet