Upgrade to Pro — share decks privately, control downloads, hide ads and more …

StatsD Workshop Monitorama 2013

StatsD Workshop Monitorama 2013

Daniel Schauenberg

March 29, 2013
Tweet

More Decks by Daniel Schauenberg

Other Decks in Technology

Transcript

  1. StatsD workshop @
    Monitorama 2013
    Daniel Schauenberg
    Software Engineer, Etsy Infrastructure
    @mrtazz
    Friday, March 29, 13

    View Slide

  2. • What is this StatsD?
    • Metrics types
    • Backends
    • Graphite and operating StatsD
    • Send all the metrics
    • Write all the backends
    Friday, March 29, 13

    View Slide

  3. Friday, March 29, 13

    View Slide

  4. Receiver
    loop
    Metrics
    processing
    module
    Backends
    Friday, March 29, 13

    View Slide

  5. legacyNamespace: false
    Friday, March 29, 13

    View Slide

  6. Counters
    foo.bar:1|c
    stats.counters.foo.bar.rate
    stats.counters.foo.bar.count
    Friday, March 29, 13

    View Slide

  7. Gauges
    foo.gar:30|g
    foo.gar:-5|g
    foo.gar:+5|g
    stats.gauges.foo.gar
    Friday, March 29, 13

    View Slide

  8. Sets
    foo.sar:30|s
    foo.sar:50|s
    stats.sets.foo.sar
    Friday, March 29, 13

    View Slide

  9. Timers
    foo.tar:30|ms
    stats.timers.foo.tar.mean_90
    stats.timers.foo.tar.std
    stats.timers.foo.tar.sum_90
    Friday, March 29, 13

    View Slide

  10. Histograms
    histogram: [{metric: "foo.tar", bins:
    [10,20,30,50]}]
    stats.timers.foo.tar.histogram.bin_10
    stats.timers.foo.tar.histogram.bin_20
    Friday, March 29, 13

    View Slide

  11. Friday, March 29, 13

    View Slide

  12. Pluggable backends
    function MyBackend(startupTime, config, emitter){
    var self = this;
    this.lastFlush = startupTime;
    this.lastException = startupTime;
    this.config = config.mybackendconfig || {};
    // what to do with metrics
    emitter.on('flush', function(timestamp, metrics) {});
    // update your status
    emitter.on('status', function(callback) {});
    // handle raw packet
    emitter.on('packet', function(packet, rinfo) {});
    };
    Friday, March 29, 13

    View Slide

  13. Management Interface
    > telnet 127.0.0.1 8126
    > stats
    > counters
    > delcounters
    > help
    Friday, March 29, 13

    View Slide

  14. Graphite + StatsD
    • default backend
    • whisper archives should match flush
    interval
    • sum aggregation for raw counters
    • docs/graphite.md
    Friday, March 29, 13

    View Slide

  15. The single-thread
    dilemma
    Friday, March 29, 13

    View Slide

  16. Tuning UDP
    # increase default core memory sizes
    net.core.rmem_default = 16777216
    net.core.wmem_default = 16777216
    net.ipv4.udp_wmem_min = 67108864
    net.ipv4.udp_rmem_min = 67108864
    net.ipv4.udp_mem = 4648512 6198016 9297024
    Friday, March 29, 13

    View Slide

  17. Sampling
    • default sampling of 0.1 on all metrics
    • most sent keys (log file) are sampled 0.01
    Friday, March 29, 13

    View Slide

  18. Quis monitoret ipsos
    monitores?
    Friday, March 29, 13

    View Slide

  19. Friday, March 29, 13

    View Slide

  20. Friday, March 29, 13

    View Slide

  21. Friday, March 29, 13

    View Slide

  22. Friday, March 29, 13

    View Slide

  23. Let’s do this
    Friday, March 29, 13

    View Slide

  24. Basic setup
    • git clone https://github.com/etsy/statsd.git
    • curl https://gist.github.com/mrtazz/5270376
    > test_config.js
    • nc -k -l 2003 > output.log
    • node stats.js test_config.js
    Friday, March 29, 13

    View Slide

  25. Send some stats
    > echo -n "foo.bar:1|c" | nc -w0 -u 127.0.0.1 8125
    > echo -n "foo.lerl:3|g" | nc -w0 -u 127.0.0.1 8125
    > echo -n "foo.lol:33|ms" | nc -w0 -u 127.0.0.1 8125
    > echo -n "foo.lol:43|ms" | nc -w0 -u 127.0.0.1 8125
    > echo -n "foo.lol:53|ms" | nc -w0 -u 127.0.0.1 8125
    Friday, March 29, 13

    View Slide

  26. Some actual stats
    Disk space:
    df -m | grep "/var" | awk '{print "diskspace.var:"$4"|g"}'
    Apache HTTPD processes:
    echo -n "httpd.procs:$(ps aux | grep [h]ttpd | wc -l)|g"
    Logfile size:
    du -s /var/log/chef.log | awk '{print "cheflog:"$1"|g"}'
    Friday, March 29, 13

    View Slide

  27. Is this real life?
    • Set up a web page
    • Have it send stats to StatsD
    • page load time
    • counters when page was accessed
    • look in the examples/ directory for code
    Friday, March 29, 13

    View Slide

  28. Histograms
    • You should have timers now
    • Configure histograms for them
    • { graphite: { histograms: [] }
    Friday, March 29, 13

    View Slide

  29. Level up!
    Friday, March 29, 13

    View Slide

  30. Write a new backend
    • Have it HTTP POST to a web server
    • write metrics to a file
    • send a txt message
    • repeat all UDP packets to TCP
    Friday, March 29, 13

    View Slide

  31. Hackathon ideas
    • Metrics path validation in backend
    • Pluggable receiver frontends
    • Forked backends
    • Configurable timer metrics
    Friday, March 29, 13

    View Slide

  32. • #statsd on Freenode IRC
    [email protected]
    • Ask me
    Friday, March 29, 13

    View Slide

  33. Basic setup
    • git clone https://github.com/etsy/statsd.git
    • curl https://gist.github.com/mrtazz/5270376
    > test_config.js
    • nc -k -l 2003 > output.log
    • node stats.js test_config.js
    Friday, March 29, 13

    View Slide

  34. StatsD workshop @
    Monitorama 2013
    Daniel Schauenberg
    Software Engineer, Etsy Infrastructure
    @mrtazz
    Friday, March 29, 13

    View Slide