StatsD: Measure Anything / Measure Everything Dan Rowe April 30, 2013 Wednesday, May 1, 13

Who am I? • Dan Rowe • DevTools Engineer at Etsy • drowe@etsy.com • @draco2002 • https://github.com/draco2003 Wednesday, May 1, 13

Who is Etsy? • 1.57 billion page views • $101.7 million worth of goods sold • 4,534,479 items sold • 22+ million members Etsy is the marketplace we make together March 2013 Stats: http://www.etsy.com/blog/news/2013/etsy-statistics-march-2013-weather-report/ Wednesday, May 1, 13

What is StatsD? • A network daemon that listens for statistics, like counters and timers, and sends aggregates to one or more pluggable backend services. https://github.com/etsy/statsd Wednesday, May 1, 13

Why StatsD? • Real-time metrics at high concurrency, without introducing user facing latency. • Users shouldn't have to wait while you measure everything. • You shouldn't have to limit what you measure to avoid the user noticing. Wednesday, May 1, 13

Why UDP? • Fire and Forget • No waiting • Best Effort Wednesday, May 1, 13

What can I send? • Simple text based protocol messages • Accepts Counters,Timers, Gauges, and Sets • Calculates additional metrics for you as well. Wednesday, May 1, 13

Counters • stats.counters.foo.car.rate • stats.counters.foo.car.count foo.car:1|c Wednesday, May 1, 13

Gauges • stats.gauges.foo.gar foo.gar:30|g foo.gar:-5|g foo.gar:+5|g Wednesday, May 1, 13

Sets • stats.sets.foo.sar foo.sar:30|s foo.sar:50|s Wednesday, May 1, 13

Timers • stats.timers.foo.tar.mean_90 • stats.timers.foo.tar.std • stats.timers.foo.tar.sum_90 foo.tar:30|ms Wednesday, May 1, 13

Histograms • stats.timers.foo.tar.histogram.bin_10 • stats.timers.foo.tar.histogram.bin_20 histogram:[{metric:”foo.tar”, bins:[10,20,30,50]}] Wednesday, May 1, 13

How to send Metrics? • echo -n “foo.car:1|c” | nc -w0 -u 127.0.0.1 8125 • https://github.com/etsy/statsd/tree/master/ examples Wednesday, May 1, 13

What does StatsD do? • Aggregates metrics for a set interval, and then flushes them to the backend. • This buffers the backend from the user/application. Wednesday, May 1, 13

Where do metrics go? • StatsD has a pluggable backend system. • By default it comes with these backends: • Graphite • Console • Repeater Wednesday, May 1, 13

Pluggable Backends • Additional backend modules can be installed simply as additional npm modules. • cd statsd/ npm install statsd-socket.io Wednesday, May 1, 13

TCP Admin Interface • Defaults to listening on port 8126 • Commands: stats, counters, timers, gauges, delcounters, deltimers, delgauges, health, quit Wednesday, May 1, 13

Scaling StatsD • A single core only scales so far. Wednesday, May 1, 13

Scaling StatsD Metric Sampling • Reduces the volume of metrics sent to StatsD. • statsd-timer-metric-counts.sh • keyFlush setting and frequent Keys log. Wednesday, May 1, 13

Scaling StatsD deleteIdleStats: true • By Default StatsD "remembers" all metrics that it has recieved. • If it hasn't received a data point for the current flushInterval it'll send a zero. • This forces StatsD to processs old/stale metrics until the next restart or by manually deleting them via the tcp interface. Wednesday, May 1, 13

Scaling StatsD UDP Tuning • # increase default core memory sizes • net.core.rmem_default = 16777216 • net.core.wmem_default = 16777216 • net.ipv4.udp_wmem_min = 67108864 • net.ipv4.udp_rmem_min = 67108864 • net.ipv4.udp_mem = 4648512 6198016 9297024 Wednesday, May 1, 13

Scaling StatsD StatsD Cluster Proxy Wednesday, May 1, 13

How does it work? • Uses a hashring based on the metric name to assign which StatsD instance to send to. • Monitors StatsD instances for existance/health. • Recalculates hashring based on StatsD instance health. • deleteIdleStats must be turned on. Wednesday, May 1, 13

Configuring the Proxy #Example config { nodes: [ {host: '127.0.0.1', port: 8127, adminport: 8128}, {host: '127.0.0.1', port: 8129, adminport: 8130}, {host: '127.0.0.1', port: 8131, adminport: 8132} ], udp_version: 'udp4', host: '0.0.0.0', port: '8125', checkInterval: 1000, cacheSize: 10000 Configure two or more more StatsD Instances, and add them to the Proxy Configuration. Wednesday, May 1, 13

Why a seperate app? • Allows StatsD to scale beyond one machine • Keeps StatsD core as simple as possible. • If a StatsD instance or server goes down, than the hashring is recalculated and metrics still flow. Wednesday, May 1, 13

Demo Time • Show SocketIO backend • Show Multiple StatsD Instances • Show failover functionality Wednesday, May 1, 13

Hiring • Etsy is always looking for great Engineers. • Come help build and improve the tools that let Etsy Monitor, Develop and Deploy at scale. • http://etsy.com/careers Wednesday, May 1, 13

Questions? • Feel free to ask now, or later via • drowe@etsy.com • @draco2002 • https://github.com/draco2003 Wednesday, May 1, 13