No content

Collecting metrics with Graphite and StatsD Víctor Martínez @knoopx http://github.com/knoopx http://knoopx.net

What is Graphite?

What is Graphite? An all-in-one solution for storing and visualizing realtime time-series data in an efficient manner. Key features: Efficient storage and ultra-fast retrieval. http://graphite.wikidot.com/

Graphite Architecture Graphite Web The frontend of Graphite. It provides a dashboard for retrieval and visualization of our metrics and a powerful plotting API. Carbon The core of Graphite. It consists of set of three processes that take care of receiving incoming data, aggregating it and persisting it to disk. Whisper The current (stable) data persistence storage; an efficient time-series based database. * There's a new persistence layer called Ceres in the works

Feeding in your data PORT=2003 SERVER=graphite.your.org echo "stats.app.logins.count 1 `date +%s`" | nc ${SERVER} ${PORT}; PORT=2003 SERVER=graphite.your.org echo "stats.app.logins.count 1 `date +%s`" | nc ${SERVER} ${PORT}; PORT=2003 SERVER=graphite.your.org echo "stats.app.logins.count 1 `date +%s`" | nc ${SERVER} ${PORT}; schema.application_name.namespace.metric_name.metric_type Value Name Timestamp

Understanding how Carbon works storage-schemas.conf Allows you to define how your data should be persisted (retention rules) and in which order (priority). storage-aggregation.conf Allows you to define how your metrics should be aggregated. Key configuration files

Understanding how Carbon works storage-schemas.conf What does retention mean? In a timespan of X time, in how much granularity I want to store my data

Understanding how Carbon works storage-schemas.conf ● For a timespan of 6 hours I want to store my data in intervals of 10 seconds. ● For a timespan of 7 days I want to store my data in intervals of 1 minute. ● For a timespan of 1 year I want to store my data in intervals of 1 hour. ● For a timespan of 5 year I want to store my data in intervals of 24 hours.

Understanding how Carbon works storage-schemas.conf

Understanding how Carbon works storage-aggregation.conf Some examples: stats.app.users.logins.count stats.app.emails.sent.count stats.app.orders.amount.max stats.app.orders.amount.min stats.app.response.time

Understanding how Carbon works storage-aggregation.conf

Understanding how Carbon works Carbon limitations What happens if I send two metrics at the same timespan (1h)? 1 new user – 6 minutes ago 2 new users – 5 minutes ago

Understanding how Carbon works Carbon limitations Carbon discards everything but the last one 2 users

Understanding how Carbon works Carbon limitations ● Increase schema retention: 1s:12h,[...] and never, ever, push metrics older than 12 hours/push more than once per second ● Pre-aggregate your data before pushing ● Use StatsD Possible Solutions:

Understanding how Carbon works Pre-aggregate your data https://gist.github.com/knoopx/4738209

Understanding how Carbon works Use StatsD What is StatdD? A network daemon that runs on the Node.js platform and listens for statistics, like counters and timers, sent over UDP and sends aggregates to one or more pluggable backend services (e.g., Graphite). Key features: Aggregates the data for us, fire and forget communication protocol (UDP) https://github.com/etsy/statsd/

Understanding how Carbon works https://github.com/Shopify/statsd-instrument StatsD.increment('revenue', 23.99) StatsD.measure('worker.time') do Worker.perform(value) end StatsD Instrument Gem Important: Carefully read StatsD documentation and ensure that your Carbon configuration explicitly matches StatsD naming conventions

Visualizing your metrics Using the Graphite dashboard Common functions: ● summarize ● timeShift ● keepLastValue ● integral ● drawAsInfinite ● sumSeries ● alias ● aliasByNode Full Documentation: http://graphite.readthedocs. org/en/1.0/functions.html

Visualizing your metrics Using the Graphite dashboard Time-shifting and aliasing: alias(timeShift(camaloon.sidekiq.workers.success.count,"24h"),"Yesterday") alias(camaloon.sidekiq.workers.success.count,"Today")

Visualizing your metrics Using the Graphite dashboard Summarizing data: summarize(camaloon.sidekiq.workers.success.count,"1h") 1h

Visualizing your metrics Using the Graphite dashboard Aliasing and combining series: aliasByNode(camaloon.sidekiq.workers.*.count, 3)

Visualizing your metrics Tasseo https://github.com/obfuscurity/tasseo Cubism.js https://github.com/square/cubism

Visualizing your metrics Graphene https://github.com/jondot/graphene

Visualizing your metrics Your own solution /render/?target=camaloon.users.count&format=json We internally use Highchart Highstock

Other resources Deployment https://github.com/manasg/chef-graphite https://github.com/gingerlime/graphite-fabric https://github.com/knoopx/puppet-graphite http://warwickp.com/2012/03/vagrant-box-with-graphite-statsd-gdash-nginx https://github.com/obfuscurity/backstop Learning http://obfuscurity.com/Tags/Graphite

Questions?