Docker containers monitoring

Presented first during the APIdays and APIStrat Berlin 2015

Linux cgroups and namespaces APIs how to get metrics from containers without Docker.

Transcript

1. Docker containers monitoring Berlin 2015
2. Léo Unbekandt @Soulou CTO at Scalingo
3. Deploy, Build, Scale, Repeat
4. Our customers want metrics to help them in their decision making
5. DOCKER AS A TOOL ?
6. DOCKER AS A TOOL • Deployment … OK
7. DOCKER AS A TOOL • Deployment … OK • Execution … OK
8. DOCKER AS A TOOL • Deployment … OK • Execution … OK • Monitoring … ?
9. - Memory usage - CPU percentage - Disk IO - Network bandwidth What metrics?
10. Docker Remote API GET /containers/:id/stats Stats API
11. Docker Remote API generates load
12. Docker Remote API is not configurable
13. Docker →is not enough So
14. Linux API Control groups Namespaces
15. Control Groups CPU, Memory, IO Namespaces Network
16. Memory usage? Memory cgroup /sys/fs/cgroup/memory/docker/:id
17. Memory cgroup Files memory.usage_in_bytes memory.limit_in_bytes memory.memsw.usage_in_bytes memory.memsw.limit_in_bytes Bytes Are Easy
18. CPU usage? cpuacct cgroup /sys/fs/cgroup/cpuacct/docker/:id
19. Cpuaccnt cgroup Files cpuacct.usage cpuacct.usage_percpu → number of nanoseconds allocated to the container since its start
20. Nanoseconds? Let's calculate!
21. How to get a CPU percentage? /proc/stat User nice system idle iowait irq softirq 7277 51882 500481 6104557 72847 217 168631 Sum of values = CPU usage since system boot (in 1/100th second)
22. Delta of container usage = 0-100% ———————————————————— Delta of system usage How to get a CPU percentage?
23. What are the IO? blkio cgroup /sys/fs/cgroup/blkio/docker/:id → Bytes read/write by device
24. blkio cgroup File blkio.io_service_bytes
25. Namespaces Networking
26. Not a control group unfortunately… Network namespace
27. Network namespace Monitoring agent in the namespace setns() Linux system call
28. Network namespace Don't worry github.com/Scalingo/go-netns
29. Interface Received bytes packets errs … eth0 12520750 102418 0 lo 7796 560 0 Interface Transmit bytes packets errs … eth0 641338 165423 0 lo 97796 250 0 Netstat in namespace
30. Enter “Acadock” Acadock monitoring Linux APIs Docker HTTP API Acadock HTTP API Data visualisation via websockets
31. Got what we want
32. Thank you Léo Unbekandt - @Soulou github.com/Scalingo/acadock-monitoring github.com/Scalingo/acadock github.com/Scalingo/go-netns github.com/Scalingo/go-netstat