Collecting One Million Unique Metrics Per Day Using Sensu

A talk about how SendGrid evolved with Sensu to collect 1M unique metrics/day, and how it helped save our bacon.

Transcript

2. hi i’m sean (@log1kal) i have two titles I do ops things sometimes
3. we’re sendgrid we send all the mail we used to be small we’re not anymore
4. product/market fit rack up the tech debt initial chaotic growth ruthlessly prioritize things that allow growth sustained growth start reining in the debt mature focus on returns, optimize all the things
5. product/market fit rack up the tech debt initial chaotic growth ruthlessly prioritize things that allow growth sustained growth start reining in the debt mature focus on returns, optimize all the things
7. product/market fit rack up the tech debt initial chaotic growth ruthlessly prioritize things that allow growth sustained growth start reining in the debt mature focus on returns, optimize all the things
8. where are the metrics?
9. ok, how do we get metrics?
10. devs
11. [skilgore@filter1131p1mdw1 ~]$ sudo sendq.pl --stats $VAR1 = { 'send' => 18299656, 'bounced' => 68962, 'requeued' => 6673875, 'delivered' => 15964257, 'expired' => 103517, 'deferred' => 10103774, 'blocked' => 232772, 'scheduled_send' => 1449753 };
12. [skilgore@filter1131p1mdw1 ~]$ sudo sendq.pl --stats $VAR1 = { 'send' => 18299656, 'bounced' => 68962, 'requeued' => 6673875, 'delivered' => 15964257, 'expired' => 103517, 'deferred' => 10103774, 'blocked' => 232772, 'scheduled_send' => 1449753 };
13. custom metric plugins
14. yay!
15. checks write directly to graphite just ruby (et al.) processes on sensu clients
16. worked… until it didn’t
17. sensu-metrics-relay handler + extension + mutator processes on sensu servers
18. worked… until it didn’t
19. worked… until it didn’t
22. “An extension, on the other hand, could maintain persistent TCP connections to the firehose destination and simply write serialized event data over that connection. In high-volume metrics installations, 10-100k metrics per minute or more, this kind of optimization is a requirement.” - sensu-metrics-relay README
23. “An extension, on the other hand, could maintain persistent TCP connections to the firehose destination and simply write serialized event data over that connection. In high-volume metrics installations, 10-100k metrics per minute or more, this kind of optimization is a requirement.” - sensu-metrics-relay README
24. “no way we’re doing more than 100k”
25. “no way we’re doing more than 100k” - Sean Kilgore, March 2016
26. we were, by 5x
27. so we wrote our own relay in go it’s fast very fast
28. sensu_graphite_relay amqp + go magic processes on it’s own servers
30. ~2.1M total metrics per DC ~1.1M through Sensu runs on 5 cores
31. ~44 custom metrics/role one has >3000 custom metrics
32. y u no opensource?
33. because this is insane
35. Questions?