Stupid Graphite Tips

My Ignite talk from DevOpsDays Portland on November 4, 2013.

1. Stupid Graphite Tips DevOpsDays Portland 2013
2. What is Graphite? carbon whisper metrics
3. require 'socket' require ‘time’ conn = TCPSocket.new 'graphite.example.com', 2003 conn.puts "foo #{value} #{Time.now.to_i}\n" conn.close metric key value timestamp $ echo "test.foo.bar 1 `date +%s`" | nc graphite.example.com 2003 metric key value timestamp Sending metrics to Carbon
4. Organizing metrics • Prefix with your collector/agent/app source • Keep test/development stuff in check • Automatically purge test & release metrics • Avoid the urge to include your username
5. and Tracking deployments drawAsInfinite(color(custom.instances.*.killed,"white")) summarize(sumSeries(group(custom.instances.*.killed)), "1d")
6. mostDeviant(5, runtime.*.load.1m) Finding the Outliers p.s. Don’t forget about sortByMaxima, sortByMinima and limit !
7. Adjusting Rate scale(nonNegativeDerivative(snmp.IF-MIB::ifInOctets.7),0.133333333) doesn’t scale! ... because as soon as you hit a new resolution, your math is wrong. scaleToSeconds(nonNegativeDerivative(snmp.IF-MIB::ifInOctets.7),1)
8. Time Shifting sumSeries(mail.hosts.mta-*.messages.in) timeShift(sumSeries(mail.hosts.mta-*.messages.in),"6mon")
9. Counting metrics sumSeries(offset(scale(collectd.*.load.load.longterm,0),1))
10. Hybrid Graphs mail.hosts.mta-*.messages.in alpha(stacked(sumSeries(mail.hosts.mta-*.messages.in)),0.6)
11. Renaming Keys on-the-fly Ever wanted to redefine your metric key in the middle of a query because the wildcard wasn’t quite in the right spot? aliasSub(hosts.web-01-pdx-prod-example- com.metric,”-(\w+)-prod”, ”.\1.prod”)
12. Graph Bookmarklet javascript:url=prompt("Enter Url");if (url) {content.Composer.loadMyGraph("temp", decodeURIComponent(url));};
13. Listing metrics $ curl -s http://graphite/metrics/index.json | json_pp | tail "graphite-1.processes.ps_state-blocked.value", "graphite-1.processes.ps_state-paging.value", "graphite-1.processes.ps_state-running.value", "graphite-1.processes.ps_state-sleeping.value", "graphite-1.processes.ps_state-stopped.value", "graphite-1.processes.ps_state-zombies.value", "graphite-1.users.users.users", "test.bar", "test.foo" ]
14. Grokking Whisper $ whisper-info.py foo.wsp maxRetention: 31536000 xFilesFactor: 0.0 aggregationMethod: average fileSize: 1262932 Archive 0 retention: 86400 secondsPerPoint: 1 points: 86400 size: 1036800 offset: 52 $ whisper-fetch.py foo.wsp | tail 1383585660! None 1383585720! 29.529000 1383585780! 87.016000 1383585840! 33.416000 1383585900! 29.012000 1383585960! 31.164000 1383586020! None 1383586080! 29.702000 1383586140! 29.299000 1383586200! None
15. Repairing Data $ whisper-fetch.py foo.wsp | grep -v None 1376066700 1.000000 1376066760 226.000000 1376066820 215.000000 1376066880 266.000000 1376066940 237.000000 1376072760 5.000000 1376074560 1.000000 1376075280 2.000000 $ for i in 1376066760 1376066820 1376066880 1376066940; \ do whisper-update.py foo.wsp ${i}:0"; done
16. Know your Carbon sumSeries(group(carbon.agents.*.updateOperations)) sumSeries(group(carbon.agents.*.metricsReceived)) sumSeries(group(carbon.agents.*.committedPoints)) secondYAxis(sumSeries(group(carbon.agents.*.pointsPerUpdate))) secondYAxis(averageSeries(group(carbon.agents.*.cpuUsage))) secondYAxis(sumSeries(group(carbon.agents.*.creates))) p.s. Check out @scott_sanders’ collectd-tail examples https://gist.github.com/jssjr/35b592b4852625449c27
17. Tagged Events $ curl -X POST http://graphite/events/ \ -d '{"what": "Something Interesting", "tags" : "wtf", "data" : "foo bar"}' events(“wtf”) events(“wtf”,”zomg”) events(“*”)
18. Dat Spark $ brew install spark $ curl -H 'Accept: application/json' -s "http://graphite/ render/?target=foo&format=json&from=-30min" | json_pp | grep ',' | grep -v '\]' | grep -v target | spark ▂▂▁▂▂▂▂▂▁▃▁▂▂▂▄▃▂▂█▇▃█▂▃▂▃▁▂▄▂
19. Scaling dat Carbon Open Spaces, doge. omg multicore hip fanout replication me much io vintage metrics
20. Thank You