Tweet
Tweet

Slide 1

Slide 1 text

StatsD workshop @ Monitorama 2013 Daniel Schauenberg Software Engineer, Etsy Infrastructure @mrtazz Friday, March 29, 13

Slide 2

Slide 2 text

• What is this StatsD? • Metrics types • Backends • Graphite and operating StatsD • Send all the metrics • Write all the backends Friday, March 29, 13

Slide 3

Slide 3 text

Friday, March 29, 13

Slide 4

Slide 4 text

Receiver loop Metrics processing module Backends Friday, March 29, 13

Slide 5

Slide 5 text

legacyNamespace: false Friday, March 29, 13

Slide 6

Slide 6 text

Counters foo.bar:1|c stats.counters.foo.bar.rate stats.counters.foo.bar.count Friday, March 29, 13

Slide 7

Slide 7 text

Gauges foo.gar:30|g foo.gar:-5|g foo.gar:+5|g stats.gauges.foo.gar Friday, March 29, 13

Slide 8

Slide 8 text

Sets foo.sar:30|s foo.sar:50|s stats.sets.foo.sar Friday, March 29, 13

Slide 9

Slide 9 text

Timers foo.tar:30|ms stats.timers.foo.tar.mean_90 stats.timers.foo.tar.std stats.timers.foo.tar.sum_90 Friday, March 29, 13

Slide 10

Slide 10 text

Histograms histogram: [{metric: "foo.tar", bins: [10,20,30,50]}] stats.timers.foo.tar.histogram.bin_10 stats.timers.foo.tar.histogram.bin_20 Friday, March 29, 13

Slide 11

Slide 11 text

Friday, March 29, 13

Slide 12

Slide 12 text

Pluggable backends function MyBackend(startupTime, config, emitter){ var self = this; this.lastFlush = startupTime; this.lastException = startupTime; this.config = config.mybackendconfig || {}; // what to do with metrics emitter.on('flush', function(timestamp, metrics) {}); // update your status emitter.on('status', function(callback) {}); // handle raw packet emitter.on('packet', function(packet, rinfo) {}); }; Friday, March 29, 13

Slide 13

Slide 13 text

Management Interface > telnet 127.0.0.1 8126 > stats > counters > delcounters > help Friday, March 29, 13

Slide 14

Slide 14 text

Graphite + StatsD • default backend • whisper archives should match flush interval • sum aggregation for raw counters • docs/graphite.md Friday, March 29, 13

Slide 15

Slide 15 text

The single-thread dilemma Friday, March 29, 13

Slide 16

Slide 16 text

Tuning UDP # increase default core memory sizes net.core.rmem_default = 16777216 net.core.wmem_default = 16777216 net.ipv4.udp_wmem_min = 67108864 net.ipv4.udp_rmem_min = 67108864 net.ipv4.udp_mem = 4648512 6198016 9297024 Friday, March 29, 13

Slide 17

Slide 17 text

Sampling • default sampling of 0.1 on all metrics • most sent keys (log file) are sampled 0.01 Friday, March 29, 13

Slide 18

Slide 18 text

Quis monitoret ipsos monitores? Friday, March 29, 13

Slide 19

Slide 19 text

Friday, March 29, 13

Slide 20

Slide 20 text

Friday, March 29, 13

Slide 21

Slide 21 text

Friday, March 29, 13

Slide 22

Slide 22 text

Friday, March 29, 13

Slide 23

Slide 23 text

Let’s do this Friday, March 29, 13

Slide 24

Slide 24 text

Basic setup • git clone https://github.com/etsy/statsd.git • curl https://gist.github.com/mrtazz/5270376 > test_config.js • nc -k -l 2003 > output.log • node stats.js test_config.js Friday, March 29, 13

Slide 25

Slide 25 text

Send some stats > echo -n "foo.bar:1|c" | nc -w0 -u 127.0.0.1 8125 > echo -n "foo.lerl:3|g" | nc -w0 -u 127.0.0.1 8125 > echo -n "foo.lol:33|ms" | nc -w0 -u 127.0.0.1 8125 > echo -n "foo.lol:43|ms" | nc -w0 -u 127.0.0.1 8125 > echo -n "foo.lol:53|ms" | nc -w0 -u 127.0.0.1 8125 Friday, March 29, 13

Slide 26

Slide 26 text

Some actual stats Disk space: df -m | grep "/var" | awk '{print "diskspace.var:"$4"|g"}' Apache HTTPD processes: echo -n "httpd.procs:$(ps aux | grep [h]ttpd | wc -l)|g" Logfile size: du -s /var/log/chef.log | awk '{print "cheflog:"$1"|g"}' Friday, March 29, 13

Slide 27

Slide 27 text

Is this real life? • Set up a web page • Have it send stats to StatsD • page load time • counters when page was accessed • look in the examples/ directory for code Friday, March 29, 13

Slide 28

Slide 28 text

Histograms • You should have timers now • Configure histograms for them • { graphite: { histograms: [] } Friday, March 29, 13

Slide 29

Slide 29 text

Level up! Friday, March 29, 13

Slide 30

Slide 30 text

Write a new backend • Have it HTTP POST to a web server • write metrics to a file • send a txt message • repeat all UDP packets to TCP Friday, March 29, 13

Slide 31

Slide 31 text

Hackathon ideas • Metrics path validation in backend • Pluggable receiver frontends • Forked backends • Configurable timer metrics Friday, March 29, 13

Slide 32

Slide 32 text

• #statsd on Freenode IRC • [email protected] • Ask me Friday, March 29, 13

Slide 33

Slide 33 text

Basic setup • git clone https://github.com/etsy/statsd.git • curl https://gist.github.com/mrtazz/5270376 > test_config.js • nc -k -l 2003 > output.log • node stats.js test_config.js Friday, March 29, 13

Slide 34

Slide 34 text

StatsD workshop @ Monitorama 2013 Daniel Schauenberg Software Engineer, Etsy Infrastructure @mrtazz Friday, March 29, 13