StatsD Workshop Monitorama 2013

1. StatsD workshop @ Monitorama 2013 Daniel Schauenberg Software Engineer, Etsy Infrastructure @mrtazz Friday, March 29, 13
2. • What is this StatsD? • Metrics types • Backends • Graphite and operating StatsD • Send all the metrics • Write all the backends Friday, March 29, 13
3. Friday, March 29, 13
4. Receiver loop Metrics processing module Backends Friday, March 29, 13
5. legacyNamespace: false Friday, March 29, 13
6. Counters foo.bar:1|c stats.counters.foo.bar.rate stats.counters.foo.bar.count Friday, March 29, 13
7. Gauges foo.gar:30|g foo.gar:-5|g foo.gar:+5|g stats.gauges.foo.gar Friday, March 29, 13
8. Sets foo.sar:30|s foo.sar:50|s stats.sets.foo.sar Friday, March 29, 13
9. Timers foo.tar:30|ms stats.timers.foo.tar.mean_90 stats.timers.foo.tar.std stats.timers.foo.tar.sum_90 Friday, March 29, 13
10. Histograms histogram: [{metric: "foo.tar", bins: [10,20,30,50]}] stats.timers.foo.tar.histogram.bin_10 stats.timers.foo.tar.histogram.bin_20 Friday, March 29, 13
11. Friday, March 29, 13
12. Pluggable backends function MyBackend(startupTime, config, emitter){ var self = this; this.lastFlush = startupTime; this.lastException = startupTime; this.config = config.mybackendconfig || {}; // what to do with metrics emitter.on('flush', function(timestamp, metrics) {}); // update your status emitter.on('status', function(callback) {}); // handle raw packet emitter.on('packet', function(packet, rinfo) {}); }; Friday, March 29, 13
13. Management Interface > telnet 127.0.0.1 8126 > stats > counters > delcounters > help Friday, March 29, 13
14. Graphite + StatsD • default backend • whisper archives should match flush interval • sum aggregation for raw counters • docs/graphite.md Friday, March 29, 13
15. The single-thread dilemma Friday, March 29, 13
16. Tuning UDP # increase default core memory sizes net.core.rmem_default = 16777216 net.core.wmem_default = 16777216 net.ipv4.udp_wmem_min = 67108864 net.ipv4.udp_rmem_min = 67108864 net.ipv4.udp_mem = 4648512 6198016 9297024 Friday, March 29, 13
17. Sampling • default sampling of 0.1 on all metrics • most sent keys (log file) are sampled 0.01 Friday, March 29, 13
18. Quis monitoret ipsos monitores? Friday, March 29, 13
19. Friday, March 29, 13
20. Friday, March 29, 13
21. Friday, March 29, 13
22. Friday, March 29, 13
23. Let’s do this Friday, March 29, 13
24. Basic setup • git clone https://github.com/etsy/statsd.git • curl https://gist.github.com/mrtazz/5270376 > test_config.js • nc -k -l 2003 > output.log • node stats.js test_config.js Friday, March 29, 13
25. Send some stats > echo -n "foo.bar:1|c" | nc -w0 -u 127.0.0.1 8125 > echo -n "foo.lerl:3|g" | nc -w0 -u 127.0.0.1 8125 > echo -n "foo.lol:33|ms" | nc -w0 -u 127.0.0.1 8125 > echo -n "foo.lol:43|ms" | nc -w0 -u 127.0.0.1 8125 > echo -n "foo.lol:53|ms" | nc -w0 -u 127.0.0.1 8125 Friday, March 29, 13
26. Some actual stats Disk space: df -m | grep "/var" | awk '{print "diskspace.var:"$4"|g"}' Apache HTTPD processes: echo -n "httpd.procs:$(ps aux | grep [h]ttpd | wc -l)|g" Logfile size: du -s /var/log/chef.log | awk '{print "cheflog:"$1"|g"}' Friday, March 29, 13
27. Is this real life? • Set up a web page • Have it send stats to StatsD • page load time • counters when page was accessed • look in the examples/ directory for code Friday, March 29, 13
28. Histograms • You should have timers now • Configure histograms for them • { graphite: { histograms: [] } Friday, March 29, 13
29. Level up! Friday, March 29, 13
30. Write a new backend • Have it HTTP POST to a web server • write metrics to a file • send a txt message • repeat all UDP packets to TCP Friday, March 29, 13
31. Hackathon ideas • Metrics path validation in backend • Pluggable receiver frontends • Forked backends • Configurable timer metrics Friday, March 29, 13
32. • #statsd on Freenode IRC • statsd@librelist.com • Ask me Friday, March 29, 13
33. Basic setup • git clone https://github.com/etsy/statsd.git • curl https://gist.github.com/mrtazz/5270376 > test_config.js • nc -k -l 2003 > output.log • node stats.js test_config.js Friday, March 29, 13
34. StatsD workshop @ Monitorama 2013 Daniel Schauenberg Software Engineer, Etsy Infrastructure @mrtazz Friday, March 29, 13