StatsD Workshop Monitorama 2013

Transcript

1. StatsD workshop @
   Monitorama 2013
   Daniel Schauenberg
   Software Engineer, Etsy Infrastructure
   @mrtazz
   Friday, March 29, 13
   

   View Slide

2. • What is this StatsD?
   • Metrics types
   • Backends
   • Graphite and operating StatsD
   • Send all the metrics
   • Write all the backends
   Friday, March 29, 13
   

   View Slide

3. Friday, March 29, 13
   

   View Slide

4. Receiver
   loop
   Metrics
   processing
   module
   Backends
   Friday, March 29, 13
   

   View Slide

5. legacyNamespace: false
   Friday, March 29, 13
   

   View Slide

6. Counters
   foo.bar:1|c
   stats.counters.foo.bar.rate
   stats.counters.foo.bar.count
   Friday, March 29, 13
   

   View Slide

7. Gauges
   foo.gar:30|g
   foo.gar:-5|g
   foo.gar:+5|g
   stats.gauges.foo.gar
   Friday, March 29, 13
   

   View Slide

8. Sets
   foo.sar:30|s
   foo.sar:50|s
   stats.sets.foo.sar
   Friday, March 29, 13
   

   View Slide

9. Timers
   foo.tar:30|ms
   stats.timers.foo.tar.mean_90
   stats.timers.foo.tar.std
   stats.timers.foo.tar.sum_90
   Friday, March 29, 13
   

   View Slide

10. Histograms
    histogram: [{metric: "foo.tar", bins:
    [10,20,30,50]}]
    stats.timers.foo.tar.histogram.bin_10
    stats.timers.foo.tar.histogram.bin_20
    Friday, March 29, 13
    

    View Slide

11. Friday, March 29, 13
    

    View Slide

12. Pluggable backends
    function MyBackend(startupTime, config, emitter){
    var self = this;
    this.lastFlush = startupTime;
    this.lastException = startupTime;
    this.config = config.mybackendconfig || {};
    // what to do with metrics
    emitter.on('flush', function(timestamp, metrics) {});
    // update your status
    emitter.on('status', function(callback) {});
    // handle raw packet
    emitter.on('packet', function(packet, rinfo) {});
    };
    Friday, March 29, 13
    

    View Slide

13. Management Interface
    > telnet 127.0.0.1 8126
    > stats
    > counters
    > delcounters
    > help
    Friday, March 29, 13
    

    View Slide

14. Graphite + StatsD
    • default backend
    • whisper archives should match flush
    interval
    • sum aggregation for raw counters
    • docs/graphite.md
    Friday, March 29, 13
    

    View Slide

15. The single-thread
    dilemma
    Friday, March 29, 13
    

    View Slide

16. Tuning UDP
    # increase default core memory sizes
    net.core.rmem_default = 16777216
    net.core.wmem_default = 16777216
    net.ipv4.udp_wmem_min = 67108864
    net.ipv4.udp_rmem_min = 67108864
    net.ipv4.udp_mem = 4648512 6198016 9297024
    Friday, March 29, 13
    

    View Slide

17. Sampling
    • default sampling of 0.1 on all metrics
    • most sent keys (log file) are sampled 0.01
    Friday, March 29, 13
    

    View Slide

18. Quis monitoret ipsos
    monitores?
    Friday, March 29, 13
    

    View Slide

19. Friday, March 29, 13
    

    View Slide

20. Friday, March 29, 13
    

    View Slide

21. Friday, March 29, 13
    

    View Slide

22. Friday, March 29, 13
    

    View Slide

23. Let’s do this
    Friday, March 29, 13
    

    View Slide

24. Basic setup
    • git clone https://github.com/etsy/statsd.git
    • curl https://gist.github.com/mrtazz/5270376
    > test_config.js
    • nc -k -l 2003 > output.log
    • node stats.js test_config.js
    Friday, March 29, 13
    

    View Slide

25. Send some stats
    > echo -n "foo.bar:1|c" | nc -w0 -u 127.0.0.1 8125
    > echo -n "foo.lerl:3|g" | nc -w0 -u 127.0.0.1 8125
    > echo -n "foo.lol:33|ms" | nc -w0 -u 127.0.0.1 8125
    > echo -n "foo.lol:43|ms" | nc -w0 -u 127.0.0.1 8125
    > echo -n "foo.lol:53|ms" | nc -w0 -u 127.0.0.1 8125
    Friday, March 29, 13
    

    View Slide

26. Some actual stats
    Disk space:
    df -m | grep "/var" | awk '{print "diskspace.var:"$4"|g"}'
    Apache HTTPD processes:
    echo -n "httpd.procs:$(ps aux | grep [h]ttpd | wc -l)|g"
    Logfile size:
    du -s /var/log/chef.log | awk '{print "cheflog:"$1"|g"}'
    Friday, March 29, 13
    

    View Slide

27. Is this real life?
    • Set up a web page
    • Have it send stats to StatsD
    • page load time
    • counters when page was accessed
    • look in the examples/ directory for code
    Friday, March 29, 13
    

    View Slide

28. Histograms
    • You should have timers now
    • Configure histograms for them
    • { graphite: { histograms: [] }
    Friday, March 29, 13
    

    View Slide

29. Level up!
    Friday, March 29, 13
    

    View Slide

30. Write a new backend
    • Have it HTTP POST to a web server
    • write metrics to a file
    • send a txt message
    • repeat all UDP packets to TCP
    Friday, March 29, 13
    

    View Slide

31. Hackathon ideas
    • Metrics path validation in backend
    • Pluggable receiver frontends
    • Forked backends
    • Configurable timer metrics
    Friday, March 29, 13
    

    View Slide

32. • #statsd on Freenode IRC
    • statsd@librelist.com
    • Ask me
    Friday, March 29, 13
    

    View Slide

33. Basic setup
    • git clone https://github.com/etsy/statsd.git
    • curl https://gist.github.com/mrtazz/5270376
    > test_config.js
    • nc -k -l 2003 > output.log
    • node stats.js test_config.js
    Friday, March 29, 13
    

    View Slide

34. StatsD workshop @
    Monitorama 2013
    Daniel Schauenberg
    Software Engineer, Etsy Infrastructure
    @mrtazz
    Friday, March 29, 13
    

    View Slide