Monitoring in Motion: Challenges of Monitoring Containers and Kuberntes

Transcript

1. Monitoring In Motion Challenges in Monitoring Kubernetes & Containers Cloud Native SF Meetup Feb 25, 2016 Ilan Rabinovitch Director, Community Datadog
2. About Me • Long time Datadog user. • Prior to Datadog built automation and monitoring tooling at Ooyala and Edmunds.com • SCALE and TXLF Co-Founder Ilan Rabinovitch Datadog ilan@datadoghq.com @irabinovitch
3. Agenda • Monitoring 101 - Crash Course • Challenges in Monitoring Dynamic Infrastructure • Demo Time • Questions?
4. Monitoring Everything
6. @honest_update on Twitter
7. Quick Overview of Datadog • Monitoring for modern applications. • Time series storage of metrics and events. • Trending, alerting and anomaly detection. • Hundreds of integrations out of the box.
8. Monitoring 101: Categorization More at: http://goo.gl/t1Rgcg
10. Monitoring 101: Focus on symptoms More at: http://goo.gl/t1Rgcg
11. Recurse until you find root cause. More at: http://goo.gl/t1Rgcg
12. Container Monitoring Challenges
13. https://www.datadoghq.com/docker-adoption/
16. Operational Complexity •Average containers per host: N (N=4, 10/2015) •N-times as many “hosts” to manage •Affects everything
17. Operational Complexity: Scale 100 instances 400 containers
18. Operational Complexity: Scale 160 metrics per host 640 metrics per host
19. Operational Complexity: Scale 100 instances 64,000 metrics
21. Host Centric vs Service Centric
22. Host Centric vs Service Centric
23. Query Based Monitoring … … …
24. •Use tags, labels, etc on your hosts and metrics. •Pull in existing labels from your infrastructure (Region, Docker Images, K8S Tags..) Query Based Monitoring By using tags, auto-adapt!
25. Where is my application running ? What’s the total throughput of App X ? What’s its response time per tag ? (pod, version, DC) What’s the distribution of 5xx from Nginx per pod ?
26. Auto Discovery
27. Docker API Kubelet API Monitoring Agent Container A O A O A O Application Container Off-The-Shelf Application (Redis, PostgreSQL, …) Containers List Metadata Additional Metadata (Pod names, RC, …) Config Backend Integration Configurations Host Level Metrics
28. Some Pictures Dashboards and Metrics Alerts Sharing
29. Demo time