Stupid Graphite Tips DevOpsDays Portland 2013

What is Graphite? carbon whisper metrics

require 'socket' require ‘time’ conn = TCPSocket.new 'graphite.example.com', 2003 conn.puts "foo #{value} #{Time.now.to_i}\n" conn.close metric key value timestamp $ echo "test.foo.bar 1 `date +%s`" | nc graphite.example.com 2003 metric key value timestamp Sending metrics to Carbon

Organizing metrics • Prefix with your collector/agent/app source • Keep test/development stuff in check • Automatically purge test & release metrics • Avoid the urge to include your username

and Tracking deployments drawAsInfinite(color(custom.instances.*.killed,"white")) summarize(sumSeries(group(custom.instances.*.killed)), "1d")

mostDeviant(5, runtime.*.load.1m) Finding the Outliers p.s. Don’t forget about sortByMaxima, sortByMinima and limit !

Adjusting Rate scale(nonNegativeDerivative(snmp.IF-MIB::ifInOctets.7),0.133333333) doesn’t scale! ... because as soon as you hit a new resolution, your math is wrong. scaleToSeconds(nonNegativeDerivative(snmp.IF-MIB::ifInOctets.7),1)

Time Shifting sumSeries(mail.hosts.mta-*.messages.in) timeShift(sumSeries(mail.hosts.mta-*.messages.in),"6mon")

Counting metrics sumSeries(offset(scale(collectd.*.load.load.longterm,0),1))

Hybrid Graphs mail.hosts.mta-*.messages.in alpha(stacked(sumSeries(mail.hosts.mta-*.messages.in)),0.6)

Renaming Keys on-the-fly Ever wanted to redefine your metric key in the middle of a query because the wildcard wasn’t quite in the right spot? aliasSub(hosts.web-01-pdx-prod-example- com.metric,”-(\w+)-prod”, ”.\1.prod”)

Graph Bookmarklet javascript:url=prompt("Enter Url");if (url) {content.Composer.loadMyGraph("temp", decodeURIComponent(url));};

Listing metrics $ curl -s http://graphite/metrics/index.json | json_pp | tail "graphite-1.processes.ps_state-blocked.value", "graphite-1.processes.ps_state-paging.value", "graphite-1.processes.ps_state-running.value", "graphite-1.processes.ps_state-sleeping.value", "graphite-1.processes.ps_state-stopped.value", "graphite-1.processes.ps_state-zombies.value", "graphite-1.users.users.users", "test.bar", "test.foo" ]

Grokking Whisper $ whisper-info.py foo.wsp maxRetention: 31536000 xFilesFactor: 0.0 aggregationMethod: average fileSize: 1262932 Archive 0 retention: 86400 secondsPerPoint: 1 points: 86400 size: 1036800 offset: 52 $ whisper-fetch.py foo.wsp | tail 1383585660! None 1383585720! 29.529000 1383585780! 87.016000 1383585840! 33.416000 1383585900! 29.012000 1383585960! 31.164000 1383586020! None 1383586080! 29.702000 1383586140! 29.299000 1383586200! None

Repairing Data $ whisper-fetch.py foo.wsp | grep -v None 1376066700 1.000000 1376066760 226.000000 1376066820 215.000000 1376066880 266.000000 1376066940 237.000000 1376072760 5.000000 1376074560 1.000000 1376075280 2.000000 $ for i in 1376066760 1376066820 1376066880 1376066940; \ do whisper-update.py foo.wsp ${i}:0"; done

Know your Carbon sumSeries(group(carbon.agents.*.updateOperations)) sumSeries(group(carbon.agents.*.metricsReceived)) sumSeries(group(carbon.agents.*.committedPoints)) secondYAxis(sumSeries(group(carbon.agents.*.pointsPerUpdate))) secondYAxis(averageSeries(group(carbon.agents.*.cpuUsage))) secondYAxis(sumSeries(group(carbon.agents.*.creates))) p.s. Check out @scott_sanders’ collectd-tail examples https://gist.github.com/jssjr/35b592b4852625449c27

Tagged Events $ curl -X POST http://graphite/events/ \ -d '{"what": "Something Interesting", "tags" : "wtf", "data" : "foo bar"}' events(“wtf”) events(“wtf”,”zomg”) events(“*”)

Dat Spark $ brew install spark $ curl -H 'Accept: application/json' -s "http://graphite/ render/?target=foo&format=json&from=-30min" | json_pp | grep ',' | grep -v '\]' | grep -v target | spark ▂▂▁▂▂▂▂▂▁▃▁▂▂▂▄▃▂▂█▇▃█▂▃▂▃▁▂▄▂

Scaling dat Carbon Open Spaces, doge. omg multicore hip fanout replication me much io vintage metrics

Thank You