StatsD At Shopify

Having strong instrumentation of your services allows you to quickly understand what broken during an outage. This presentation gives an introduction to StatsD and Graphite and shows how to effectively instrument your services.

Transcript

1. STATSD
2. data & time
3. week day minute
4. week day minute email
5. week day minute email dashboard
6. week day minute email dashboard statsd
7. statsd aggregate
8. increment StatsD.increment('checkout_created')
9. measure StatsD.measure('checkout_create')  do    Checkout.create end
10. graphite graph
11. graphite.shopify.com
14. stats.timers.rc2.response_time. lower mean_90 count std upper_90
17. reports-dashboard-app
18. I, [2013-07-09T19:20:24.775600 #27430] INFO -- : Result of COPY into orders is: PGRES_COMMAND_OK I, [2013-07-09T19:22:40.996657 #27430] INFO -- : Import of: orders done in 156.915290356 seconds I, [2013-07-09T19:22:40.997121 #27430] INFO -- : Aggregating data from orders into orders_hourly I, [2013-07-09T19:22:41.000360 #27430] INFO -- : Deleting data in orders_hourly since 2013-07-08 07:00:00 UTC I, [2013-07-09T19:23:08.361929 #27430] INFO -- : Done aggregating data from orders into orders_hourly took: 27.364678771 secs. I, [2013-07-09T19:23:08.362411 #27430] INFO -- : Aggregating data from orders into orders_three_hourly I, [2013-07-09T19:23:08.364060 #27430] INFO -- : Deleting data in orders_three_hourly since 2013-07-08 06:00:00 UTC I, [2013-07-09T19:24:05.365206 #27430] INFO -- : Done aggregating data from orders into orders_three_hourly took: 57.002684493 secs. I, [2013-07-09T19:24:05.365590 #27430] INFO -- : Aggregating data from orders into orders_daily I, [2013-07-09T19:24:05.367022 #27430] INFO -- : Deleting data in orders_daily since 2013-07-08 00:00:00 UTC I, [2013-07-09T19:24:27.067976 #27430] INFO -- : Done aggregating data from orders into orders_daily took: 21.70228348 secs. I, [2013-07-09T19:24:27.068373 #27430] INFO -- : Aggregating data from orders into orders_daily_with_inactive I, [2013-07-09T19:24:27.069709 #27430] INFO -- : Deleting data in orders_daily_with_inactive since 2013-07-08 00:00:00 UTC I, [2013-07-09T19:24:48.167146 #27430] INFO -- : Done aggregating data from orders into orders_daily_with_inactive took: 21.098650945 secs. I, [2013-07-09T19:24:48.167536 #27430] INFO -- : Aggregating data from orders into orders_weekly I, [2013-07-09T19:24:48.169052 #27430] INFO -- : Deleting data in orders_weekly since 2013-07-06 00:00:00 UTC
20. 10 min 15 min 15 min aggregate response time
21. 10 min 15 min 15 min aggregate response time
23. reports-tracking
25. nginx response time
26. nginx tail response time log delay channel length
27. nginx tail response time log delay channel length redis redis length
28. nginx redis tail sub response time log delay sub delay channel length redis length
29. nginx redis tail sub bucket response time log delay sub delay bucket delay channel length channel length redis length
30. nginx redis upsert tail sub bucket response time log delay sub delay bucket delay channel length channel length redis length
31. nginx redis upsert tail sub bucket done response time log delay sub delay bucket delay upsert delay channel length channel length redis length upsert time
32. nginx redis upsert tail sub bucket done response time log delay sub delay bucket delay upsert delay channel length channel length redis length upsert time dashboard delay
33. vault.shopify.com/statsd