Aphorisms of Monitoring

Transcript

1. Aphorisms of Monitoring Chris Christensen @imetchrischris KS Linux Fest - March 22, 2015
2. Landscape Limelight Networks • Top tier CDN. We have a video platform & object storage on top • 10 tbps egress with our own international backbone • Global footprint with 100+ data centers • Five figure system magnitude Environment • Technology company, 24/7 engineering presence • Diverse, talented teams with variety of skills - systems & software ◦ Massive FreeBSD Edge ◦ Several flavors of *nix’s
3. Aphorisms aph·o·rism - noun 1. a pithy observation that contains a general truth, such as, “if it ain't broke, don't fix it”. 2. a concise statement of a scientific principle, typically by an ancient classical author. • Monitoring is metrics. • Numbers. • Monitoring Sucks. ◦ #monitoringsucks ◦ #monitoringlove • It would be awesome if...
4. Metrics + Monitoring A metric: 0.02
5. Metrics + Monitoring A metric: Free Disk Space: 0.02
6. Metrics + Monitoring A metric: web123.example.com: Free Disk Space (in %) on / ⇒ 2%
7. Metrics + Monitoring Time series - History
8. Anscombe's quartet All four sets are identical when examined using simple summary statistics, but vary considerably when graphed … in defense of time-series graphs
9. Metrics + Monitoring Threshold (Trigger)
10. Metrics + Monitoring A metric: web123.example.com: Free Disk Space (in %) on / ⇒ 2% Types: • Gauge • Counter
11. Metrics "API" key: value @ time • Graphite: metric_path value timestamp ◦ foo.bar.baz 42 74857843 • OpenTSDB: metric timestamp value tagk1=tagv1 ◦ sys.cpu.user 1356998400 42.5 host=webserver01 cpu=0 • Zabbix: metric timestamp value ◦ hw.serial.number 1287872261 SQ4321ASDF
12. Metrics "API" Metadata • Datatype • Context • Units • Description • Eagerness (safe polling interval) • ... Storage • MIB • DB Schema (e.g. Zabbix) • In-band?
13. Understanding where computation is happening "Back of the envelope" • Number of values • Frequency (per second) • Size (integer, float, text) Looks like: Analysis of algorithms → O(n)
14. Thresholds / Alerting • Analyzing metrics ◦ in real time… ◦ in report / summary time... Notable writeups: • Netflix: Atlas • Airbnb / Linkedin ◦ Kafka ◦ Samza
15. Building on the shoulders of ... • Work with upstream ◦ "Waiting for Godot" • Precedence • Valuing schemas
16. Thoughts and takeaways • Take some time to learn from others ◦ Why? "They're just graph servers and stuff" • Monitoring software often has NIH syndrome
17. Scale ...Can mean more than just raw processing / machinery • Teams / collaboration • Situational awareness • The right tool for the job
18. Q/A
19. Tools Interesting looking: • Bosun • Sensu • OpenNMS • OMD • ELK Stack • Librato • Circonus Experience / can provide feedback: • Zabbix • OpenTSDB • Grafana • Splunk • Jut
20. Thank you. Chris Christensen @imetchrischris p.s. we’re hiring...