Stupid Graphite Tips

My Ignite talk from DevOpsDays Portland on November 4, 2013.

Transcript

1. Stupid Graphite Tips
   DevOpsDays Portland 2013
   

   View Slide

2. What is Graphite?
   carbon whisper
   metrics
   

   View Slide

3. require 'socket'
   require ‘time’
   conn = TCPSocket.new 'graphite.example.com', 2003
   conn.puts "foo #{value} #{Time.now.to_i}\n"
   conn.close
   metric key value timestamp
   $ echo "test.foo.bar 1 `date +%s`" | nc graphite.example.com 2003
   metric key value timestamp
   Sending metrics to Carbon
   

   View Slide

4. Organizing metrics
   • Prefix with your collector/agent/app source
   • Keep test/development stuff in check
   • Automatically purge test & release metrics
   • Avoid the urge to include your username
   

   View Slide

5. and
   Tracking deployments
   drawAsInfinite(color(custom.instances.*.killed,"white"))
   summarize(sumSeries(group(custom.instances.*.killed)), "1d")
   

   View Slide

6. mostDeviant(5, runtime.*.load.1m)
   Finding the Outliers
   p.s. Don’t forget about sortByMaxima, sortByMinima and limit !
   

   View Slide

7. Adjusting Rate
   scale(nonNegativeDerivative(snmp.IF-MIB::ifInOctets.7),0.133333333)
   doesn’t scale!
   ... because as soon as you hit a new
   resolution, your math is wrong.
   scaleToSeconds(nonNegativeDerivative(snmp.IF-MIB::ifInOctets.7),1)
   

   View Slide

8. Time Shifting
   sumSeries(mail.hosts.mta-*.messages.in)
   timeShift(sumSeries(mail.hosts.mta-*.messages.in),"6mon")
   

   View Slide

9. Counting metrics
   sumSeries(offset(scale(collectd.*.load.load.longterm,0),1))
   

   View Slide

10. Hybrid Graphs
    mail.hosts.mta-*.messages.in
    alpha(stacked(sumSeries(mail.hosts.mta-*.messages.in)),0.6)
    

    View Slide

11. Renaming Keys on-the-fly
    Ever wanted to redefine your metric key
    in the middle of a query because the
    wildcard wasn’t quite in the right spot?
    aliasSub(hosts.web-01-pdx-prod-example-
    com.metric,”-(\w+)-prod”, ”.\1.prod”)
    

    View Slide

12. Graph Bookmarklet
    javascript:url=prompt("Enter Url");if (url)
    {content.Composer.loadMyGraph("temp",
    decodeURIComponent(url));};
    

    View Slide

13. Listing metrics
    $ curl -s http://graphite/metrics/index.json | json_pp | tail
    "graphite-1.processes.ps_state-blocked.value",
    "graphite-1.processes.ps_state-paging.value",
    "graphite-1.processes.ps_state-running.value",
    "graphite-1.processes.ps_state-sleeping.value",
    "graphite-1.processes.ps_state-stopped.value",
    "graphite-1.processes.ps_state-zombies.value",
    "graphite-1.users.users.users",
    "test.bar",
    "test.foo"
    ]
    

    View Slide

14. Grokking Whisper
    $ whisper-info.py foo.wsp
    maxRetention: 31536000
    xFilesFactor: 0.0
    aggregationMethod: average
    fileSize: 1262932
    Archive 0
    retention: 86400
    secondsPerPoint: 1
    points: 86400
    size: 1036800
    offset: 52
    $ whisper-fetch.py foo.wsp | tail
    1383585660! None
    1383585720! 29.529000
    1383585780! 87.016000
    1383585840! 33.416000
    1383585900! 29.012000
    1383585960! 31.164000
    1383586020! None
    1383586080! 29.702000
    1383586140! 29.299000
    1383586200! None
    

    View Slide

15. Repairing Data
    $ whisper-fetch.py foo.wsp | grep -v None
    1376066700 1.000000
    1376066760 226.000000
    1376066820 215.000000
    1376066880 266.000000
    1376066940 237.000000
    1376072760 5.000000
    1376074560 1.000000
    1376075280 2.000000
    $ for i in 1376066760 1376066820 1376066880 1376066940; \
    do whisper-update.py foo.wsp ${i}:0"; done
    

    View Slide

16. Know your Carbon
    sumSeries(group(carbon.agents.*.updateOperations))
    sumSeries(group(carbon.agents.*.metricsReceived))
    sumSeries(group(carbon.agents.*.committedPoints))
    secondYAxis(sumSeries(group(carbon.agents.*.pointsPerUpdate)))
    secondYAxis(averageSeries(group(carbon.agents.*.cpuUsage)))
    secondYAxis(sumSeries(group(carbon.agents.*.creates)))
    p.s. Check out @scott_sanders’ collectd-tail examples
    https://gist.github.com/jssjr/35b592b4852625449c27
    

    View Slide

17. Tagged Events
    $ curl -X POST http://graphite/events/ \
    -d '{"what": "Something Interesting",
    "tags" : "wtf",
    "data" : "foo bar"}'
    events(“wtf”)
    events(“wtf”,”zomg”)
    events(“*”)
    

    View Slide

18. Dat Spark
    $ brew install spark
    $ curl -H 'Accept: application/json' -s "http://graphite/
    render/?target=foo&format=json&from=-30min" | json_pp | grep
    ',' | grep -v '\]' | grep -v target | spark
    ▂▂▁▂▂▂▂▂▁▃▁▂▂▂▄▃▂▂█▇▃█▂▃▂▃▁▂▄▂
    

    View Slide

19. Scaling dat Carbon
    Open Spaces, doge.
    omg multicore
    hip fanout
    replication me
    much io
    vintage metrics
    

    View Slide

20. Thank You
    

    View Slide