Graphite & StatsD Workshop

Graphite & StatsD workshop for DevopsDays Tel Aviv 2013

Installation of graphite and statsd. General terminology about monitoring and its importance. Dashboards galore! And understanding StatsD - all in one.

Transcript

1. Graphite & StatsD Workshop
2. Evgeny Zislis about.me/kesor @kesor6 / #devopsdays tlv facebook.com/groups/ops.il www.devops.co.il
3. TOC • Install (might take time) • Monitoring & Graphite • Dashboards • StatsD
4. Installation • Prerequisuites ◦ VirtualBox ◦ Vagrant • Packages ◦ nodejs ◦ collectd ◦ python virtualenv ◦ libcairo dev headers bit.ly/dodtlv2013-graphite-code
5. Installation (techie!) $ git clone \ https://gist.github.com/61229dbce7f96505649e.git \ dodtlv-graphite $ cd dodtlv-graphite $ vagrant up
6. (more) Installation • source / git ◦ graphite.wikidot.com/installation • chef ◦ community.opscode.com/cookbooks/graphite • fabric ◦ github.com/gingerlime/graphite-fabric • puppet ◦ forge.puppetlabs.com/modules?q=graphite
7. Monitoring • Collecting • Persistence (history storage) • Graphing • Alerting
8. RRD Tool - Round Robin Database oss.oetiker.ch/rrdtool/
9. WTF? is Graphite • Python • Carbon • Whisper/Ceres • Web UI + API
10. Hosted SaaS www.hostedgraphite.com
11. graphite.readthedocs.org/en/latest/feeding-carbon.html echo "local.random.diceroll 4 `date +%s`" | \ nc graphite.your.org 2003; • carbon-cache ◦ accept metrics, write to disk • carbon-relay ◦ replication, sharding • carbon-aggregator ◦ buffer metrics, aggregate, send to carbon-cache Carbon
12. graphite.readthedocs.org/en/latest/feeding-carbon.html bash echo "local.random.diceroll 4 $(date +%s)" | \ nc -u graphite.your.org 2003 Python import socket sock = socket.socket(socket.AF_INET, socket. SOCK_DGRAM)) sock.connect( ('127.0.0.1', 2003) ) sock.sendall( 'foo.bar.baz 42 %d\n' % int(time.time()) ) sock.close() Send (app) metrics to Carbon
13. • collectd >= 5.1.x LoadPlugin "write_graphite" <Plugin "write_graphite"> <Carbon> Host "localhost" Port "2003" </Carbon> </Plugin> Tools sending data to Carbon
14. • Sensu # /etc/sensu/conf.d/graphite_tcp.json { "graphite": { "server":"graphite.example.com", "port":"2003" } } # /etc/sensu/conf.d/handler_graphite.json { "handlers": { "graphite": { "type": "pipe", "command": "/etc/sensu/handlers/graphite_tcp.rb" } } } Tools sending data to Carbon
15. • Backstop github.com/obfuscurity/backstop RestClient.post("https://backstop.example.com/publish/custom", [{:metric => key, :value => value, :measure_time => Time.now.to_i}].to_json) Tools sending data to Carbon
16. • Ganglia • Diamond • Rocksteady • Shinken graphite.rtfd.org/en/latest/tools.html Tools sending data to Carbon
17. Whisper • Round-robin (based on rrd concepts) ◦ Fixed size ◦ New data pushes older data away • Multiple precisions • Everything is a float • { timestamp: value }
18. Schemas for Storage 1 year of minutely data = 6mb per metric • more metrics → i/o & space • how long is your data useful for? • what is the precision you really need?
19. storage-schemas.conf cat <<-EOF > $graphite_root/conf/storage-schemas.conf [default_1min_for_1day] pattern = .* retentions = 60s:1d [collectd] pattern = ^collectd\. retentions = 10s:30d [stats] priority = 110 pattern = ^stats.* retentions = 10s:6h,1m:7d,10m:1y EOF
20. [min] pattern = \.lower$ xFilesFactor = 0.5 aggregationMethod = min [sum] pattern = \.sum$ xFilesFactor = 0.5 aggregationMethod = sum [default_average] pattern = .* xFilesFactor = 0.6 aggregationMethod = average storage-aggregation.conf
21. Notice the default xFilesFactor value < 1 xFilesFactor is the ratio of precision data points your archive must contain to be rolled up accurately. More than half nulls would aggregate to a null archive. xFilesFactor
22. Whisper Gotchas Avoid this - [default] pattern = .* retentions = 1s:1h, 10s:6h, 60s:1w, 5m:30d, 1h:1y Keep xFilesFactor >= 0.5 to avoid excess I/O Fetch returns data from single archive (only!)
23. Aggregation • Averaging ◦ great for trending ◦ applies to most data ◦ calculate aggregate sum if number of samples is known • Types: ◦ Average, Sum, Min, Max, Latest
24. Per Second rates statsd / collectd (StoreRates true) / … • Store as averages • Multiple by precision on query • scale(<metric>, 60)
25. github.com/tmm1/graphite/tree/master/whisper/bin $ whisper-info.py /data/whisper/collectd/63694/swap/used.wsp maxRetention: 31536000 xFilesFactor: 0.5 aggregationMethod: average fileSize: 534580 Archive 0 retention: 86400 secondsPerPoint: 60 points: 1440 size: 17280 offset: 52 Whisper CLI
26. • graphite.rtfd.org/en/latest/render_api.html • graphite.rtfd.org/en/latest/functions.html Graphite WEB UI / API
27. DEMO Graphite WEB UI / API
28. Dashboards / Tools codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/ • Etsy’s StatsD ◦ github.com/etsy/statsd • Etsy’s check_graphite_data ◦ github.com/etsy/nagios_tools • Etsy’s Dashboard ◦ github.com/etsy/dashboard
29. • GitHub OctoStatus (iOS app) ◦ not open source, but some source at - ◦ speakerdeck.com/wfarr/graphs-for-the-fucking-win • GitHub Descartes ◦ github.com/obfuscurity/descartes • Tasseo ◦ github.com/obfuscurity/tasseo Jason Dixon - obfuscurity.com Dashboards / Tools
30. dashboarddude.com/blog/2013/01/23/dashboards-for-graphite • Cubism.js square.github.io/cubism • Graphene jondot.github.io/graphene • Giraffe github.com/kenhub/giraffe • Seyren github.com/scobal/seyren • Graphitus github.com/erezmazor/graphitus • Graphiti github.com/paperlesspost/graphiti • GDash github.com/ripienaar/gdash • Dashing shopify.github.io/dashing • Graph Explorer vimeo.github.io/graph-explorer More Dashboards
31. StatsD code.flickr.net/2008/10/27/counting-timing • Front-end proxy for the Graphite/Carbon metrics server. • Aggregates into 10s buckets (default)
32. StatsD types statsd.rtfd.org/en/latest/types.html • Counters - statsd.incr('some.event') • Timers - statsd.timing('slept',10000) # int millis • Gauges - statsd.gauge('foo', 70)
33. StatsD Backends ◦ amqp ◦ ganglia ◦ librato ◦ socket.io ◦ statsd ◦ mongo ◦ datadog ◦ monitis ◦ zabbix ◦ mysql
34. StatsD implementation(s) github.com/etsy/statsd/wiki BatsD - github.com/noahhl/batsd
35. bit.ly/graphite-statsd-workshop ?