Monitoring and Health at Airbnb

Monitoring and Health at Airbnb by @joeyparsons at Monitorama 2016

Transcript

1. Monitoring & Health @ Airbnb JOEY PARSONS / WEDNESDAY, JUNE 29TH 2016 / MONITORAMA @joeyparsons
2. BEARSHARK!!!
3. Fun Fact!
9. 50-100 LISTINGS 101-300 LISTINGS 301-1000 LISTINGS 1001+ LISTINGS 2009
10. 50-100 LISTINGS 101-300 LISTINGS 301-1000 LISTINGS 1001+ LISTINGS 2010
11. 50-100 LISTINGS 101-300 LISTINGS 301-1000 LISTINGS 1001+ LISTINGS 2011
12. 50-100 LISTINGS 101-300 LISTINGS 301-1000 LISTINGS 1001+ LISTINGS 2012
13. 50-100 LISTINGS 101-300 LISTINGS 301-1000 LISTINGS 1001+ LISTINGS 2013
14. 50-100 LISTINGS 101-300 LISTINGS 301-1000 LISTINGS 1001+ LISTINGS 2014
15. 50-100 LISTINGS 101-300 LISTINGS 301-1000 LISTINGS 1001+ LISTINGS 2015
16. 2 Million TOTAL HOMES ON AIRBNB
17. 191 COUNTRIES 34K CITIES
18. Belong Anywhere
20. New Relic
21. Datadog
23. Interferon Store your alerts configuration in code. • host_sources • destinations • groups • alerts http://airbnb.io http://github.com/airbnb/interferon
24. Interferon Simply define the following: • name of the alert • the body of the alert (this is also your runbook!) • what does this alert apply to • who should it notify? • what are the conditions of the alert • some other useful metadata
25. Interferon Route to group sources: • yaml configuration of integrations • supports email, slack, pagerduty and whatever else you want to build or route alerts to http://airbnb.io http://github.com/airbnb/interferon
26. SYSOPS: VOLUNTEER ON CALL
27. How does one volunteer and become a Sysops!? Promoted to On-Call Shadow On-Call Sysops Training Go deeper into our monitoring systems, how to be effective on- call, and learn from historical incidents Daily rotation of new members to learn from the current primary and secondary on-call Added to the schedule once your daily rotation is complete and you feel comfortable. 2 shifts per week.
28. How does one volunteer and become a Sysops!? Promoted to On-Call Shadow On-Call Sysops Training Go deeper into our monitoring systems, how to be effective on- call, and learn from historical incidents Daily business hours rotation of new members to learn from the current primary and secondary on-call Added to the schedule once your daily rotation is complete and you feel comfortable. 2 shifts per week.
29. How does one volunteer and become a Sysops!? Promoted to On-Call Shadow On-Call Sysops Training Go deeper into our monitoring systems, how to be effective on- call, and learn from historical incidents Daily rotation of new members to learn from the current primary and secondary on-call Added to the schedule once your daily rotation is complete and you feel comfortable. 2 shifts per week.
30. Weekly Sysops Meeting
31. Yak Shaving Offsite! (fyi, no yaks were harmed)
32. Sweet New Hoodies!
33. Remediate Our stack was simpler then and compendium of knowledge was easy to learn and grasp Triage If you can remediate, go for it but triaging and finding the right person can be just as effective Health How much is this affecting the volunteer on-call (and all of our on-calls) Sysops: Past, Present and Future
34. Harmony
35. On-Call Health How do we answer the following? • Are our weekly alerts trending appropriately? • Do we understand the impact on our on-call engineers? • Are there false positive alerts that we need to tune? • Are there teams that need better on-call training? • Are rotations and notification policies appropriate?
40. Sleep Some weeks are tough — can someone on your team tap out if they’ve been sleep deprived or need a break? How can you build this in proactively? Life Can your team live an enriched life or do they feel bound to their laptops and are handling incidents even when they’re not on call? Recognition Are you appropriately recognizing both on-call work and work to eliminate tough on- calls? Be a Host: It’s all about people
41. BEARSHARK!!!
42. Thank You @joeyparsons