Datadog - Docker Usage Patterns

Datadog - Docker Usage Patterns

A look at the result of our latest large-scale study about Docker usage in real environment. Analyze and see the impact for operations and monitoring.

London Docker Meetup - February, 3rd 2016


Benjamin Fernandes

February 03, 2016


  1. Docker Usage Patterns No Hype, Just Data Docker Meetup London

    Feb 3, 2016 Benjamin Fernandes Software Engineer — Datadog @LotharSee

  2. About Me • MS, CS degree from Ecole Centrale Paris

    • Software Engineer @ Datadog
 • Joined Datadog 3 years ago
 • Worked on Datadog’s integration with Docker and its ecosystem Benjamin Fernandes
  3. Quick Overview of Datadog Datadog gathers performance data from all

    your application and infrastructure components. • Monitoring for modern applications 1. Dynamic Infrastructure 2. Containers (Docker, ECS, k8s, …) 3. Microservices • Time series storage of metrics and events • Trending, alerting and anomaly detection • We’re hiring! (New York, Paris, remote)
  4. Monitor Everything Datadog gathers performance data from all your application

  5. Prologue:
 Docker to ship Open-Source

  6. Avoid Dependency Hell

  7. What about in production? Docker Adoption Who is running Docker

  8. Adopter: the average number of containers running during the month

    was at least 50% the number of distinct hosts run, or there were at least as many distinct containers as distinct hosts run during the month. Dabbler: used Docker during the month, but did not reach the “adopter” threshold. Abandoner: a currently active company that used Docker in the past, but hasn't used it at all in the last month. Study based on data from 7000 companies. Docker Adoption
  9. Are you using Docker? Turns out you aren’t alone! Docker

    Adoption Source:
  10. Fact 1: Docker Adoption Up 5x in 1 Year

  11. Docker Adoption Growth We’ve see 5x increase of Docker adoption

    over the last year.
  12. Fact 2: Docker now runs on 6% of hosts we

  13. None
  14. Fact 3: Larger Companies Are the Early Adopters!

  15. None
  16. Fact 4: 2/3 of Companies That Try Docker Adopt It

  17. None
  18. Fact 5: Users 3x the Number of Containers They Use

    in 5 Months
  19. None
  20. Fact 6: Most Widely Used Images Are Registry, NGINX, and

  21. None
  22. Fact 7: Docker Hosts Often Run Four Containers at a

  23. Fact 8: VMs Live 4x Longer Than Containers

  24. None
  25. Bonus fact. Thanks The Onion.

  26. None
  27. Operational Complexity • Average containers per host: N (N=4, 10/2015)

    • N-times as many “hosts” to manage • Affects • provisioning: prep’ing & building containers • configuration: passing config to containers • orchestration: deciding where/when containers run • monitoring: making sure containers run properly
  28. Operational Complexity: Scale 100 instances 400 containers

  29. Operational Complexity: Scale 160 metrics per host 640 metrics per

    host Assuming 4 containers per host
  30. Operational Complexity: Scale 100 instances 64,000 metrics Assuming 4 containers

    per host
  31. Operational Complexity: Velocity

  32. So what does that mean for monitoring and management?

  33. Monitoring Needs and Pains • Avoid Static config files tracking

    dynamic infrastructure • Avoid a host centric view. Focus on service level. • Use tags, labels, etc on your hosts and metrics to form queries.
  34. Query Based Monitoring “Show me rate of HTTP 500 responses

    from nginx” “… in region us-east-1 across all availability zones” “… running my app version 2….” • Pull in labels from your infrastructure whether EC2, Docker or your scheduler. • Ask questions that will ring true regardless of your scale that day. • Know your underlying tech. In this case Docker and how to pull metrics from it.
  35. Collecting Docker Metrics

  36. Collecting Docker Metrics: stats • Continuous live stream of basic

    CPU, memory, & network • docker stats --no-stream $(docker ps -q)
  37. Collecting Docker Metrics: stats api • Stream of JSON, more

  38. Collecting Docker Metrics: Pseudo Files • If you want to

    do it manually • CPU/Mem metrics • Access via sysfs in /sys/fs/cgroup or /cgroup • By default do not require root access • Network metrics • /proc/$PID/net/dev • Disk IO metrics • /proc/$PID/io
  39. None
  40. Woof? Woof!