Stupid Graphite Tips

Transcript

1. Stupid Graphite Tips DevOpsDays Portland 2013

2. What is Graphite? carbon whisper metrics

3. require 'socket' require ‘time’ conn = TCPSocket.new 'graphite.example.com', 2003 conn.puts

   "foo #{value} #{Time.now.to_i}\n" conn.close metric key value timestamp $ echo "test.foo.bar 1 `date +%s`" | nc graphite.example.com 2003 metric key value timestamp Sending metrics to Carbon

4. Organizing metrics • Prefix with your collector/agent/app source • Keep

   test/development stuff in check • Automatically purge test & release metrics • Avoid the urge to include your username

5. and Tracking deployments drawAsInfinite(color(custom.instances.*.killed,"white")) summarize(sumSeries(group(custom.instances.*.killed)), "1d")

6. mostDeviant(5, runtime.*.load.1m) Finding the Outliers p.s. Don’t forget about sortByMaxima,

   sortByMinima and limit !

7. Adjusting Rate scale(nonNegativeDerivative(snmp.IF-MIB::ifInOctets.7),0.133333333) doesn’t scale! ... because as soon as

   you hit a new resolution, your math is wrong. scaleToSeconds(nonNegativeDerivative(snmp.IF-MIB::ifInOctets.7),1)

8. Time Shifting sumSeries(mail.hosts.mta-*.messages.in) timeShift(sumSeries(mail.hosts.mta-*.messages.in),"6mon")

9. Counting metrics sumSeries(offset(scale(collectd.*.load.load.longterm,0),1))

10. Hybrid Graphs mail.hosts.mta-*.messages.in alpha(stacked(sumSeries(mail.hosts.mta-*.messages.in)),0.6)

11. Renaming Keys on-the-fly Ever wanted to redefine your metric key

    in the middle of a query because the wildcard wasn’t quite in the right spot? aliasSub(hosts.web-01-pdx-prod-example- com.metric,”-(\w+)-prod”, ”.\1.prod”)

12. Graph Bookmarklet javascript:url=prompt("Enter Url");if (url) {content.Composer.loadMyGraph("temp", decodeURIComponent(url));};

13. Listing metrics $ curl -s http://graphite/metrics/index.json | json_pp | tail

    "graphite-1.processes.ps_state-blocked.value", "graphite-1.processes.ps_state-paging.value", "graphite-1.processes.ps_state-running.value", "graphite-1.processes.ps_state-sleeping.value", "graphite-1.processes.ps_state-stopped.value", "graphite-1.processes.ps_state-zombies.value", "graphite-1.users.users.users", "test.bar", "test.foo" ]

14. Grokking Whisper $ whisper-info.py foo.wsp maxRetention: 31536000 xFilesFactor: 0.0 aggregationMethod:

    average fileSize: 1262932 Archive 0 retention: 86400 secondsPerPoint: 1 points: 86400 size: 1036800 offset: 52 $ whisper-fetch.py foo.wsp | tail 1383585660! None 1383585720! 29.529000 1383585780! 87.016000 1383585840! 33.416000 1383585900! 29.012000 1383585960! 31.164000 1383586020! None 1383586080! 29.702000 1383586140! 29.299000 1383586200! None

15. Repairing Data $ whisper-fetch.py foo.wsp | grep -v None 1376066700

    1.000000 1376066760 226.000000 1376066820 215.000000 1376066880 266.000000 1376066940 237.000000 1376072760 5.000000 1376074560 1.000000 1376075280 2.000000 $ for i in 1376066760 1376066820 1376066880 1376066940; \ do whisper-update.py foo.wsp ${i}:0"; done

16. Know your Carbon sumSeries(group(carbon.agents.*.updateOperations)) sumSeries(group(carbon.agents.*.metricsReceived)) sumSeries(group(carbon.agents.*.committedPoints)) secondYAxis(sumSeries(group(carbon.agents.*.pointsPerUpdate))) secondYAxis(averageSeries(group(carbon.agents.*.cpuUsage))) secondYAxis(sumSeries(group(carbon.agents.*.creates))) p.s.

    Check out @scott_sanders’ collectd-tail examples https://gist.github.com/jssjr/35b592b4852625449c27

17. Tagged Events $ curl -X POST http://graphite/events/ \ -d '{"what":

    "Something Interesting", "tags" : "wtf", "data" : "foo bar"}' events(“wtf”) events(“wtf”,”zomg”) events(“*”)

18. Dat Spark $ brew install spark $ curl -H 'Accept:

    application/json' -s "http://graphite/ render/?target=foo&format=json&from=-30min" | json_pp | grep ',' | grep -v '\]' | grep -v target | spark ▂▂▁▂▂▂▂▂▁▃▁▂▂▂▄▃▂▂█▇▃█▂▃▂▃▁▂▄▂

19. Scaling dat Carbon Open Spaces, doge. omg multicore hip fanout

    replication me much io vintage metrics

20. Thank You