Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Stupid Graphite Tips

Jason Dixon
November 04, 2013

Stupid Graphite Tips

My Ignite talk from DevOpsDays Portland on November 4, 2013.

Jason Dixon

November 04, 2013
Tweet

More Decks by Jason Dixon

Other Decks in Technology

Transcript

  1. Stupid Graphite Tips
    DevOpsDays Portland 2013

    View full-size slide

  2. What is Graphite?
    carbon whisper
    metrics

    View full-size slide

  3. require 'socket'
    require ‘time’
    conn = TCPSocket.new 'graphite.example.com', 2003
    conn.puts "foo #{value} #{Time.now.to_i}\n"
    conn.close
    metric key value timestamp
    $ echo "test.foo.bar 1 `date +%s`" | nc graphite.example.com 2003
    metric key value timestamp
    Sending metrics to Carbon

    View full-size slide

  4. Organizing metrics
    • Prefix with your collector/agent/app source
    • Keep test/development stuff in check
    • Automatically purge test & release metrics
    • Avoid the urge to include your username

    View full-size slide

  5. and
    Tracking deployments
    drawAsInfinite(color(custom.instances.*.killed,"white"))
    summarize(sumSeries(group(custom.instances.*.killed)), "1d")

    View full-size slide

  6. mostDeviant(5, runtime.*.load.1m)
    Finding the Outliers
    p.s. Don’t forget about sortByMaxima, sortByMinima and limit !

    View full-size slide

  7. Adjusting Rate
    scale(nonNegativeDerivative(snmp.IF-MIB::ifInOctets.7),0.133333333)
    doesn’t scale!
    ... because as soon as you hit a new
    resolution, your math is wrong.
    scaleToSeconds(nonNegativeDerivative(snmp.IF-MIB::ifInOctets.7),1)

    View full-size slide

  8. Time Shifting
    sumSeries(mail.hosts.mta-*.messages.in)
    timeShift(sumSeries(mail.hosts.mta-*.messages.in),"6mon")

    View full-size slide

  9. Counting metrics
    sumSeries(offset(scale(collectd.*.load.load.longterm,0),1))

    View full-size slide

  10. Hybrid Graphs
    mail.hosts.mta-*.messages.in
    alpha(stacked(sumSeries(mail.hosts.mta-*.messages.in)),0.6)

    View full-size slide

  11. Renaming Keys on-the-fly
    Ever wanted to redefine your metric key
    in the middle of a query because the
    wildcard wasn’t quite in the right spot?
    aliasSub(hosts.web-01-pdx-prod-example-
    com.metric,”-(\w+)-prod”, ”.\1.prod”)

    View full-size slide

  12. Graph Bookmarklet
    javascript:url=prompt("Enter Url");if (url)
    {content.Composer.loadMyGraph("temp",
    decodeURIComponent(url));};

    View full-size slide

  13. Listing metrics
    $ curl -s http://graphite/metrics/index.json | json_pp | tail
    "graphite-1.processes.ps_state-blocked.value",
    "graphite-1.processes.ps_state-paging.value",
    "graphite-1.processes.ps_state-running.value",
    "graphite-1.processes.ps_state-sleeping.value",
    "graphite-1.processes.ps_state-stopped.value",
    "graphite-1.processes.ps_state-zombies.value",
    "graphite-1.users.users.users",
    "test.bar",
    "test.foo"
    ]

    View full-size slide

  14. Grokking Whisper
    $ whisper-info.py foo.wsp
    maxRetention: 31536000
    xFilesFactor: 0.0
    aggregationMethod: average
    fileSize: 1262932
    Archive 0
    retention: 86400
    secondsPerPoint: 1
    points: 86400
    size: 1036800
    offset: 52
    $ whisper-fetch.py foo.wsp | tail
    1383585660! None
    1383585720! 29.529000
    1383585780! 87.016000
    1383585840! 33.416000
    1383585900! 29.012000
    1383585960! 31.164000
    1383586020! None
    1383586080! 29.702000
    1383586140! 29.299000
    1383586200! None

    View full-size slide

  15. Repairing Data
    $ whisper-fetch.py foo.wsp | grep -v None
    1376066700 1.000000
    1376066760 226.000000
    1376066820 215.000000
    1376066880 266.000000
    1376066940 237.000000
    1376072760 5.000000
    1376074560 1.000000
    1376075280 2.000000
    $ for i in 1376066760 1376066820 1376066880 1376066940; \
    do whisper-update.py foo.wsp ${i}:0"; done

    View full-size slide

  16. Know your Carbon
    sumSeries(group(carbon.agents.*.updateOperations))
    sumSeries(group(carbon.agents.*.metricsReceived))
    sumSeries(group(carbon.agents.*.committedPoints))
    secondYAxis(sumSeries(group(carbon.agents.*.pointsPerUpdate)))
    secondYAxis(averageSeries(group(carbon.agents.*.cpuUsage)))
    secondYAxis(sumSeries(group(carbon.agents.*.creates)))
    p.s. Check out @scott_sanders’ collectd-tail examples
    https://gist.github.com/jssjr/35b592b4852625449c27

    View full-size slide

  17. Tagged Events
    $ curl -X POST http://graphite/events/ \
    -d '{"what": "Something Interesting",
    "tags" : "wtf",
    "data" : "foo bar"}'
    events(“wtf”)
    events(“wtf”,”zomg”)
    events(“*”)

    View full-size slide

  18. Dat Spark
    $ brew install spark
    $ curl -H 'Accept: application/json' -s "http://graphite/
    render/?target=foo&format=json&from=-30min" | json_pp | grep
    ',' | grep -v '\]' | grep -v target | spark
    ▂▂▁▂▂▂▂▂▁▃▁▂▂▂▄▃▂▂█▇▃█▂▃▂▃▁▂▄▂

    View full-size slide

  19. Scaling dat Carbon
    Open Spaces, doge.
    omg multicore
    hip fanout
    replication me
    much io
    vintage metrics

    View full-size slide