Monitoring OpenStack at Lithium (OpenStack Summit Austin 2016)

Transcript

1. Monitoring OpenStack At Lithium
2. How is OpenStack is used at Lithium? • Production public-facing communities to major brands (SaaS) ◦ Redis, Java, Node,js, MySQL, Elasticsearch, Cassandra • Infrastructure services ◦ Kubernetes + Docker ◦ Chef ◦ DNS ◦ Consul • Development environments • R&D services • Random VMs because it’s easy to use
3. • Kilo (EU) & Icehouse (US) ◦ Keystone, Nova, Cinder, Swift, Glance, Neutron, Heat, Horizon • ~60 hypervisors • ~1000 instances • ~10 TB RAM (~6 TB used) • Contrail for SDN • Ceph & SolidFire (provisioned iOps) for Cinder • Managed by a team of 3 engineers OpenStack @ Lithium
4. Tools are available out of the box • Horizon ◦ Difficult to create cross-tenant and/or rollup reports ◦ Multi-region compounds the cross-tenant issues ◦ Not very user friendly for all teams involved (Ops, DevOps, Dev, Finance, Mgmt, etc.) ◦ Doesn’t have time-based metrics to show usage over time • Nova APIs ◦ Rolling your own? Who really has time for that? ◦ Need to run graphite (or similar) to represent the data ◦ Push metrics into statsd or similar service using Python We used a combination of both before using Datadog’s integration Ugh… : (
6. • Our users want graphs & dashboards • Time series graphs tell the real story • Incredibly easy to implement • Easy to add/extend functionality ◦ Open Source code on GitHub ◦ Able to extend with our own custom enhancements • Open source is important to us • Able to see OpenStack metrics side-by-side with our application metrics Why we went with Datadog Photo credits: Google Images - The Indian Government encourages adoption of OSS http://news.softpedia.com/news/Use-of-Open-Source-Software-Is-Now- Mandatory-In-Indian-Government-Offices-477052.shtml
7. •SaaS based infrastructure and app monitoring •Open Source Agent •Time series data (metrics and events) •Processing nearly a trillion data points per day •Intelligent Alerting •We’re hiring! (www.datadoghq.com/careers/) Datadog Overview
9. Collecting data is cheap; not having it when you need it can be expensive
11. Instrument all the things!
12. Operational Complexity Increases with.. • Number of things to measure • Velocity of change
13. How much to measure measure? 1 node • 30 metrics from OpenStack 1 operating system (e.g., Linux) • 100 metrics per instance Custom Applications • 50~ metrics
14. How much to measure measure? 1 VM • 30~ metrics from OpenStack 1 operating system (e.g., Linux) • 100 metrics per instance Custom Applications • 50~ metrics Containers 150*N
15. Operational Complexity 100 instances 400 containers
16. Operational Complexity: Scale 180 metrics per host 630 metrics per host
17. Operational Complexity: Scale 100 instances 63,000 metrics
18. How much to measure measure? 1 VM • 30~ metrics from OpenStack 1 operating system (e.g., Linux) • 100 metrics per instance Custom Applications • 50~ metrics Containers 150*N Metrics Overload!
20. Source: http://bit.ly/1qFylWK
22. Tooling Needs: Static vs Dynamic
24. Tooling Needs: Combining Multiple Data Sources
25. Outlier Detection
26. Tooling Needs: Avoiding Host Centric Views
27. Tooling Needs: Host Centric vs Service Centric
29. How to show what it all means?
30. Visualization is key
31. Perspective Matters By Mysid - Self-made in Inkscape, contours from en:Image:Perspective-foreshortening.png., Public Domain, https://commons.wikimedia.org/w/index.php?curid=2562161 By Katri - Flickr: On the road, CC BY-SA 2.0, https://commons.wikimedia.org/w/index.php?curid=15967705
32. • You are no longer monitoring an infrastructure stack ◦ It’s a set of applications that provide your infrastructure • You need to start monitoring not just server stats (cpu, memory, disk) but also how the applications work together • Servers may look fine even if the services are not responding properly • Probably have > 1 network providing the network to the running instances Important concepts to remember https://en.wikipedia.org/wiki/Wikipedia OpenStack service
33. How do you know what to compare? By Sandy Austin - originally posted to Flickr as Snack time again, CC BY 2.0, https://commons.wikimedia.org/w/index.php?curid=10003672
34. Why didn’t my instance didn’t launch properly?
35. Why does elasticsearch lock up & then crash?
36. Monitoring 101: TL;DR Edition
37. Finding Signal - Categorizing Your Metrics
42. Examples: Nova - Metrics Work Metrics: • hdd_read_req • running_vms • Resource Metrics: • hypervisor_up • vcpus available • free_disk_gb • free_ram_mb
43. Examples: Nova - Events • Configuration Change • Code Deployment • Service Started / Stopped • Instance Migrations • Instance Creation • Adding / Removing Nodes
44. Examples: Events
45. When to let a sleeping engineer lie?
46. When to alert?
50. Recurse until you find root cause
51. What’s still missing? http://www.amazon.com/Missing-Piece-Shel-Silverstein/dp/0060256710/
52. • Mike Tougeron - @mtougeron • Ilan Rabinovitch - @irabinovitch