[2016.06 Meetup #3][TALK #2] Pedro Pessoa - Data That Matters

It is people that run ops. Lots of data out there, what does really matter so that we don't feel overwhelmed?

Nowadays you can collect uncountable metrics from devices. The challenge is to store and make good use of them. We'll look at practical use cases for alert monitoring.

Transcript

1. Data that matters Pedro Pessoa pessoa@serverdensity.com June 2016
2. http://slack.serverdensity.com
3. Agenda • Metrics by order of importance: ◦ Cats ◦ Meta monitoring ◦ Redis • HumanOps
4. Example #0 - Cats https://blog.serverdensity.com/how-to-monitor-cats/
5. Example #0 - Cats! • Cats are independent creatures.
6. Example #0 - Cats! • Cats are independent creatures. • They eat and drink as they please.
7. Example #0 - Cats! • Cats are independent creatures. • They eat and drink as they please. • But how often?
8. Example #0 - Cats! • Cats are independent creatures. • They eat and drink as they please. • But how often? • And while we’re at it, when do they use the litterbox? Wouldn’t it be cool to get a notification when it’s time to clean the litter?
9. Example #0 - Cats! SAFETY NOTICE: If you run this project (or any exercise involving pets) please do not use strings or laces to attach anything on your pet. To avoid potential injury, make sure you use safe, purpose-built cat collars.
10. Example #0 - Cats! • #0 - Charol
11. Example #0 - Cats! • #1 - Oli
12. Example #0 - Cats! • Bluetooth Low Energy with RSSI (Received Signal Strength Indication)
13. Example #0 - Cats! • Raspberry Pi. Wi-Fi and Bluetooth 4.0 dongles (you need Bluetooth version 4.0 for Low Energy support).
14. Example #0 - Cats! 0 - if we couldn’t find the device. The cat could be anywhere. 1- if the device is reachable but “not close”. The cat is probably snoozing somewhere. 2- if the device is “close”. The cat is probably drinking, eating, or using the litterbox. The reading doesn’t specify which facility the cat is using (if at all). A future iteration could employ weight or humidity sensors to do just that.
15. Example #0 - Cats!
16. Example #0 - Cats!
17. Example #0 - Cats! • Most activity appears to happen around meal times (which is to be expected).
18. Example #0 - Cats! • Most activity appears to happen around meal times (which is to be expected). • We noticed Charol made a few more visits to the food tower (no wonder he weighs 7kg).
19. Example #0 - Cats!
20. Example #0 - Cats! • The cats didn’t drink much water.
21. Example #0 - Cats! • The cats didn’t drink much water. • Charol visited the fountain a bit more than Oli. That’s probably because Charol eats more dry food than Oli.
22. Example #0 - Cats! • The cats didn’t drink much water. • Charol visited the fountain a bit more than Oli. That’s probably because Charol eats more dry food than Oli. • The water fountain is further out in a corner, that explains the occasional 0 readings, when the device was unreachable.
23. Example #0 - Cats!
24. Example #0 - Cats! • The cats don’t use the toilet that often.
25. Example #0 - Cats! • The cats don’t use the toilet that often. • Given that the litterbox is in the toilet upstairs, we have 0 readings as well.
26. Example #1 - Nagios https://blog.serverdensity.com/how-to-monitor-nagios-for-free/
27. Example #1 - Nagios • Meta monitoring! aka “Who will alert you when your monitoring service is down?”
28. Example #1 - Nagios • Meta monitoring! aka “Who will alert you when your monitoring service is down?” • Monitor the monitoring!
29. Example #1 - Nagios
30. Example #1 - Nagios
31. Example #1 - Nagios
32. Example #1 - Nagios • Non-critical alerting vs. Critical alerting ◦ what should you look into in the morning ◦ what should get you out of bed
33. Example #1 - Nagios
34. Example #3 - Redis https://blog.serverdensity.com/monitor-redis/
35. Example #3 - Redis • Redis is a key-value database, and one of the most popular NoSQL databases out there. Redis (REmote DIctionary Server) works in a similar fashion to memcached, albeit with a non-volatile dataset.
36. Example #3 - Redis • The dataset is stored entirely in memory (one of the reasons Redis is so fast) and it is periodically flushed to disk so it remains persistent.
37. Example #3 - Redis • Redis also provides native support for manipulating and querying data structures such as lists, sets and hashes.
38. Example #3 - Redis • Usually used to improve the response time of retrieving and/or matching data
39. Example #3 - Redis
40. Example #3 - Redis
41. Example #3 - Redis • Monitoring is not alerting: Our rule of thumb is: “collect all metrics that help with troubleshooting, alert only on those that require an action.”
42. Example #3 - Redis • Redis process running
43. Example #3 - Redis • Redis availability and queries
44. Example #3 - Redis
45. Example #3 - Redis
46. Example #3 - Redis • Redis performance
47. Example #3 - Redis
48. Example #3 - Redis • Redis errors
49. Example #3 - Redis
50. Example #3 - Redis • Monitoring is not alerting: Our rule of thumb is: “collect all metrics that help with troubleshooting, alert only on those that require an action.” • Use previous work (like this one!)
51. Example #3 - Redis
52. Example #3 - Redis
53. Trends
55. Consequences-external metrics
58. War story https://blog.serverdensity.com/cluster-optimisation/
59. www.humanops.com
60. www.CloudStatusApp.com
61. Takeaways • Learn from others that have walked the path • Issues pile up • Watch for trends • Avoid setting up your own monitoring system as much as you can - ‘DEVOPS’ coupon: https://www.serverdensity. com/conferences • We are only human