Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

Collecting metrics with Graphite and StatsD Víctor Martínez @knoopx http://github.com/knoopx http://knoopx.net

Slide 3

Slide 3 text

What is Graphite?

Slide 4

Slide 4 text

What is Graphite? An all-in-one solution for storing and visualizing realtime time-series data in an efficient manner. Key features: Efficient storage and ultra-fast retrieval. http://graphite.wikidot.com/

Slide 5

Slide 5 text

Graphite Architecture Graphite Web The frontend of Graphite. It provides a dashboard for retrieval and visualization of our metrics and a powerful plotting API. Carbon The core of Graphite. It consists of set of three processes that take care of receiving incoming data, aggregating it and persisting it to disk. Whisper The current (stable) data persistence storage; an efficient time-series based database. * There's a new persistence layer called Ceres in the works

Slide 6

Slide 6 text

Feeding in your data PORT=2003 SERVER=graphite.your.org echo "stats.app.logins.count 1 `date +%s`" | nc ${SERVER} ${PORT}; PORT=2003 SERVER=graphite.your.org echo "stats.app.logins.count 1 `date +%s`" | nc ${SERVER} ${PORT}; PORT=2003 SERVER=graphite.your.org echo "stats.app.logins.count 1 `date +%s`" | nc ${SERVER} ${PORT}; schema.application_name.namespace.metric_name.metric_type Value Name Timestamp

Slide 7

Slide 7 text

Understanding how Carbon works storage-schemas.conf Allows you to define how your data should be persisted (retention rules) and in which order (priority). storage-aggregation.conf Allows you to define how your metrics should be aggregated. Key configuration files

Slide 8

Slide 8 text

Understanding how Carbon works storage-schemas.conf What does retention mean? In a timespan of X time, in how much granularity I want to store my data

Slide 9

Slide 9 text

Understanding how Carbon works storage-schemas.conf ● For a timespan of 6 hours I want to store my data in intervals of 10 seconds. ● For a timespan of 7 days I want to store my data in intervals of 1 minute. ● For a timespan of 1 year I want to store my data in intervals of 1 hour. ● For a timespan of 5 year I want to store my data in intervals of 24 hours.

Slide 10

Slide 10 text

Understanding how Carbon works storage-schemas.conf

Slide 11

Slide 11 text

Understanding how Carbon works storage-aggregation.conf Some examples: stats.app.users.logins.count stats.app.emails.sent.count stats.app.orders.amount.max stats.app.orders.amount.min stats.app.response.time

Slide 12

Slide 12 text

Understanding how Carbon works storage-aggregation.conf

Slide 13

Slide 13 text

Understanding how Carbon works Carbon limitations What happens if I send two metrics at the same timespan (1h)? 1 new user – 6 minutes ago 2 new users – 5 minutes ago

Slide 14

Slide 14 text

Understanding how Carbon works Carbon limitations Carbon discards everything but the last one 2 users

Slide 15

Slide 15 text

Understanding how Carbon works Carbon limitations ● Increase schema retention: 1s:12h,[...] and never, ever, push metrics older than 12 hours/push more than once per second ● Pre-aggregate your data before pushing ● Use StatsD Possible Solutions:

Slide 16

Slide 16 text

Understanding how Carbon works Pre-aggregate your data https://gist.github.com/knoopx/4738209

Slide 17

Slide 17 text

Understanding how Carbon works Use StatsD What is StatdD? A network daemon that runs on the Node.js platform and listens for statistics, like counters and timers, sent over UDP and sends aggregates to one or more pluggable backend services (e.g., Graphite). Key features: Aggregates the data for us, fire and forget communication protocol (UDP) https://github.com/etsy/statsd/

Slide 18

Slide 18 text

Understanding how Carbon works https://github.com/Shopify/statsd-instrument StatsD.increment('revenue', 23.99) StatsD.measure('worker.time') do Worker.perform(value) end StatsD Instrument Gem Important: Carefully read StatsD documentation and ensure that your Carbon configuration explicitly matches StatsD naming conventions

Slide 19

Slide 19 text

Visualizing your metrics Using the Graphite dashboard Common functions: ● summarize ● timeShift ● keepLastValue ● integral ● drawAsInfinite ● sumSeries ● alias ● aliasByNode Full Documentation: http://graphite.readthedocs. org/en/1.0/functions.html

Slide 20

Slide 20 text

Visualizing your metrics Using the Graphite dashboard Time-shifting and aliasing: alias(timeShift(camaloon.sidekiq.workers.success.count,"24h"),"Yesterday") alias(camaloon.sidekiq.workers.success.count,"Today")

Slide 21

Slide 21 text

Visualizing your metrics Using the Graphite dashboard Summarizing data: summarize(camaloon.sidekiq.workers.success.count,"1h") 1h

Slide 22

Slide 22 text

Visualizing your metrics Using the Graphite dashboard Aliasing and combining series: aliasByNode(camaloon.sidekiq.workers.*.count, 3)

Slide 23

Slide 23 text

Visualizing your metrics Tasseo https://github.com/obfuscurity/tasseo Cubism.js https://github.com/square/cubism

Slide 24

Slide 24 text

Visualizing your metrics Graphene https://github.com/jondot/graphene

Slide 25

Slide 25 text

Visualizing your metrics Your own solution /render/?target=camaloon.users.count&format=json We internally use Highchart Highstock

Slide 26

Slide 26 text

Other resources Deployment https://github.com/manasg/chef-graphite https://github.com/gingerlime/graphite-fabric https://github.com/knoopx/puppet-graphite http://warwickp.com/2012/03/vagrant-box-with-graphite-statsd-gdash-nginx https://github.com/obfuscurity/backstop Learning http://obfuscurity.com/Tags/Graphite

Slide 27

Slide 27 text

Questions?