Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Monitor Docker Containers on CoreOS cluster #monitoringcasual

taichi nakashima
January 30, 2015
970

Monitor Docker Containers on CoreOS cluster #monitoringcasual

taichi nakashima

January 30, 2015
Tweet

Transcript

  1. Monitor
    Docker Containers
    on CoreOS cluster

    View full-size slide

  2. I’m Taichi Nakashima
    @deeeet tcnksm
    https://www.flickr.com/photos/unforgiven/9278027165

    View full-size slide

  3. What is difference?
    Traditional stack vs. Containers

    View full-size slide

  4. Physical (Machines)
    Virtual Virtual
    Service A Service B Service C Service D

    View full-size slide

  5. Physical (Machines)
    Virtual Virtual
    Container Container Container Container
    Service A Service B Service C Service D

    View full-size slide

  6. Monitoring by Yourself
    Have your own monitoring system
    Monitoring as a Service
    Use external service

    View full-size slide

  7. Monitoring by Yourself
    Have your own monitoring system
    Monitoring as a Service
    Use external service

    View full-size slide

  8. For a single host monitoring
    cAdvisor
    For a cluster scale monitoring
    Heapster
    Monitoring by Yourself
    Have your own monitoring system

    View full-size slide

  9. Just running dockerized cAdvisor container
    Collect all container metrics on a host
    Just access to :8080 in your browser
    Provide Web UI
    cAdvisor

    View full-size slide

  10. Run cAdvisor container
    $ docker run \
    --volume=/:/rootfs:ro \
    --volume=/var/run:/var/run:rw \
    --volume=/sys:/sys:ro \
    --volume=/var/lib/docker/:/var/lib/docker:ro \
    --publish=8080:8080 \
    --detach=true \
    --name=cadvisor \
    google/cadvisor:latest

    View full-size slide

  11. Collect cAdvisor metrics from cluster member, it’s used in Kubernetes
    Enables cluster wide monitoring of containers
    Draw graph by Grafana
    Support InfluxDB backend
    Heapster

    View full-size slide

  12. Container
    Heapster

    View full-size slide

  13. Container
    Heapster

    View full-size slide

  14. Monitoring by Yourself
    Have your own monitoring system
    Monitoring as a Service
    Use external service

    View full-size slide

  15. But No README and No document…
    stanaka/mackerel-docker
    Mackerel
    Not Support container specific feature
    No container specific monitoring

    View full-size slide

  16. New Relic
    Dockerized collector agent but only for host metrics not for containers,
    To monitor each container we need to install it on each our docker image
    johanneswuerbach/newrelic-sysmond
    Not Support container specific feature
    No container specific monitoring

    View full-size slide

  17. DataDog
    Dockerized collector agent, just run docker container
    DataDog/docker-dd-agent
    Support container specific feature !!
    Container metrics, Tagging, Lifecycle of container, etc

    View full-size slide

  18. DataDog Container feature
    What is good point and why ?

    View full-size slide

  19. DataDog container feature
    Agent tags by docker container name and its image name (by default)
    Tagging
    All containers on a host
    If you run 1 dd-agent container, it monitors all containers in the host
    It collects each container’s CPU, memory, network I/O and disk I/O (General)
    Lifecycle monitoring
    Agent also monitor container create, start, stop, destroy events

    View full-size slide

  20. DataDog container feature
    Agent tags by docker container name and its image name (by default)
    Tagging
    All containers on a host
    If you run 1 dd-agent container, it monitors all containers in the host
    It collects each container’s CPU, memory, network I/O and disk I/O (General)
    Lifecycle monitoring
    Agent also monitor container create, start, stop, destroy events
    Easy to start

    View full-size slide

  21. Run dd-agent container
    $ docker run \
    --privileged \
    --name dd-agent \
    -h `hostname` \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /proc/mounts:/host/proc/mounts:ro \
    -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
    -e API_KEY={your_api_key_here} \
    datadog/docker-dd-agent

    View full-size slide

  22. Container dd-agent

    View full-size slide

  23. To dd-agent container on CoreOS cluster
    [Unit]

    [Service]
    TimeoutStartSec=0
    ExecStartPre=-/usr/bin/docker kill dd-agent
    ExecStartPre=-/usr/bin/docker rm dd-agent
    ExecStartPre=/usr/bin/docker pull datadog/docker-dd-agent
    ExecStart=/usr/bin/bash -c \
    "/usr/bin/docker run --privileged --name dd-agent -h `hostname` \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /proc/mounts:/host/proc/mounts:ro \
    -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
    -e API_KEY=`YOUR_API_KEY` \
    datadog/docker-dd-agent"
    [X-Fleet]
    Global=true

    View full-size slide

  24. To dd-agent container on CoreOS cluster
    [Unit]

    [Service]
    TimeoutStartSec=0
    ExecStartPre=-/usr/bin/docker kill dd-agent
    ExecStartPre=-/usr/bin/docker rm dd-agent
    ExecStartPre=/usr/bin/docker pull datadog/docker-dd-agent
    ExecStart=/usr/bin/bash -c \
    "/usr/bin/docker run --privileged --name dd-agent -h `hostname` \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /proc/mounts:/host/proc/mounts:ro \
    -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
    -e API_KEY=`YOUR_API_KEY` \
    datadog/docker-dd-agent"
    [X-Fleet]
    Global=true

    View full-size slide

  25. To dd-agent container on CoreOS cluster
    [Unit]

    [Service]
    TimeoutStartSec=0
    ExecStartPre=-/usr/bin/docker kill dd-agent
    ExecStartPre=-/usr/bin/docker rm dd-agent
    ExecStartPre=/usr/bin/docker pull datadog/docker-dd-agent
    ExecStart=/usr/bin/bash -c \
    "/usr/bin/docker run --privileged --name dd-agent -h `hostname` \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /proc/mounts:/host/proc/mounts:ro \
    -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
    -e API_KEY=`YOUR_API_KEY` \
    datadog/docker-dd-agent"
    [X-Fleet]
    Global=true

    View full-size slide

  26. To dd-agent container on CoreOS cluster
    $ fleetctl start dd-agent.service
    Run dd-agent.service by fleet

    View full-size slide

  27. DataDog container feature
    Agent tags by docker container name and its image name (by default)
    Tagging
    All containers on a host
    If you run 1 dd-agent container, it monitors all containers in the host
    It collects each container’s CPU, memory, network I/O and disk I/O (General)
    Lifecycle monitoring
    Agent also monitor container create, start, stop, destroy events
    Easy to explore what you want

    View full-size slide

  28. DataDog container feature
    Agent tags by docker container name and its image name (by default)
    Tagging
    All containers on a host
    If you run 1 dd-agent container, it monitors all containers in the host
    It collects each container’s CPU, memory, network I/O and disk I/O (General)
    Lifecycle monitoring
    Agent also monitor container create, start, stop, destroy events
    Help understand unexpected value

    View full-size slide

  29. New containers are created

    View full-size slide

  30. DataDog container feature
    Agent tags by docker container name and its image name (by default)
    Tagging
    All containers on a host
    If you run 1 dd-agent container, it monitors all containers in the host
    It collects each container’s CPU, memory, network I/O and disk I/O (General)
    Lifecycle monitoring
    Agent also monitor container create, start, stop, destroy events
    Easy to start
    Easy to explore what you want
    Help understand unexpected value

    View full-size slide

  31. Requirement for container monitoring
    Common part of recent trend

    View full-size slide

  32. Container
    Heapster

    View full-size slide

  33. Container dd-agent

    View full-size slide

  34. Install agent each container is not good idea, keep container simple !
    All containers in a host by 1 agent
    Requirement for container monitoring
    Containerized Agent
    Container only approach is Docker-way (CoreOS)

    View full-size slide

  35. Manage secret values on distributed KVS
    Save API token on etcd/consul
    Extra edition

    View full-size slide

  36. To dd-agent container on CoreOS cluster
    [Unit]

    [Service]
    TimeoutStartSec=0
    ExecStartPre=-/usr/bin/docker kill dd-agent
    ExecStartPre=-/usr/bin/docker rm dd-agent
    ExecStartPre=/usr/bin/docker pull datadog/docker-dd-agent
    ExecStart=/usr/bin/bash -c \
    "/usr/bin/docker run --privileged --name dd-agent -h `hostname` \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /proc/mounts:/host/proc/mounts:ro \
    -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
    -e API_KEY=`YOUR_API_KEY` \
    datadog/docker-dd-agent"
    [X-Fleet]
    Global=true

    View full-size slide

  37. xordataexchange/crypt

    View full-size slide

  38. Generate pub and sub key
    $ gpg2 --gen-key # generate pub.gpg and secret.gpg

    View full-size slide

  39. Save json value on etcd by crypt
    $ cat < config.json
    {"test": "passw0rd"}
    EOF
    $ crypt set -keyring=pub.gpg /app/config config.json

    View full-size slide

  40. Get value withtout crypt and secret-key
    $ etcdctl get /app/config
    wcBMA0OL+oKDi4zdAQgAh7iKVASBZvvX6WiiLPYSZgAbhYDhZyVGqX
    +uK2Bc1plC/mYkqw/n3FXyL+ZC0ISdK9Hdqv6HpCthnMHmBCfhPAjV4
    DsrXKWO7TP0AYTxUPMxX9sIiTzrLTJGb73134Z6l0z0Ocj2dEuhyAt5u
    3cucKkQb3CWGyuhM7C02aTeJoPjIkqi3agAizQn0uwcurSONpmCkArq33
    3579iHZv42Xnr+1Dq4CkcDG9OYPyKcoixOvvW9OpB1E

    View full-size slide

  41. Get value from etcd by crypt and key
    $ crypt get -secret-keyring secret.gpg /app/config
    {"test":"passw0rd"}

    View full-size slide

  42. To dd-agent container on CoreOS cluster
    [Unit]

    [Service]
    TimeoutStartSec=0
    ExecStartPre=-/usr/bin/docker kill dd-agent
    ExecStartPre=-/usr/bin/docker rm dd-agent
    ExecStartPre=/usr/bin/docker pull datadog/docker-dd-agent
    ExecStart=/usr/bin/bash -c \
    "/usr/bin/docker run --privileged --name dd-agent -h `hostname` \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /proc/mounts:/host/proc/mounts:ro \
    -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
    -e API_KEY=`crypt get -secret-keyring /etc/secret.gpg /ddapikey` \
    datadog/docker-dd-agent"
    [X-Fleet]
    Global=true

    View full-size slide