Introduction to StatsD

WebCamp KL August 2013

Transcript

1. Introduction to StatsD
2. Information • Improving a system requires resources • Increase return on investment by correct assignment of resources • Decisions are made by analyzing information
3. Data Collection • Accurate data helps better decisions • Data from different aspects provide a better understanding • Data collection should have least affect on performance of the running system
4. Measurement • Measurement from outside simulates users better, but is limited to the surface layer (interface) • Measurement from inside provides accurate results from all layers of the system
5. StatsD • A solution to send data from within the system with non blocking behavior • StatsD was proposed by etsy, There is no standard (yet). Implemented in Node.js • System sends metrics over UDP to StatsD server • StatsD server aggregates and flushes data to a persistent backend
6. StatsD • Protocol is simple: Metrics have name, value and type sent as text over UDP. • Client libraries available for many languages, easy to implement. • Compatible servers implemented in other languages (Python, Ruby, Perl, C, …) • A very common backend is Graphite
7. Metric Types • Counter: Anything you count • Timer: Duration of a task • Gauge: Any arbitrary value • Set: Number of unique values in each sampling period
8. Bucky • Bucky is a StatsD server written in Python $ pip install bukcy $ pip install bukcy $ bucky bucky.conf metricsd_enabled = False collectd_enabled = False statsd_enabled = True statsd_ip = "" statsd_port = 8125 statsd_flush_time = 1.0 graphite_ip = "127.0.0.1" graphite_port = 2003 • Sample Bucky configuration file
9. Sample Code #!/usr/bin/env python # - install statsd module: pip install statsd # - we sleep for a random time (max 5 secs) to simulate a # real action # - statsd timers are in microseconds import time import random import statsd client = statsd.StatsClient('localhost', 8125) start_t = time.time() client.incr('action') time.sleep(random.random() * 5) client.timing('action', (time.time() - start_t) * 1000 * 1000)
10. Graphite • A solution to collect, store and visualize data over time • Contains 3 components – Carbon: Network service to receive data – Whisper: Time series database file – Graphite: Web interface – Simple protocol: Text overt TCP with a name, value and a timestamp separated by space.
11. Graphite
12. Farzad Ghanei www.ghanei.net