DevOpsDays Taipei 2022 - How we send alert to the right people

分享近期我們開始導入 SRE Practice 的經驗談，主要著重在 Alerting Management 的部分。會以實際案例說明，在我們的組織內發現了哪些問題，以及我們是如何改善它們。原來 Alert 並非只靠 Prometheus + Alert manager 就夠了？

Transcript

1. How We Send Alert To the right People DevOpsDays Taipei 2022
2. About Me Tsung-Ting, Wu ken.wu SRE Software engineer
3. Agenda Issues Issue with alert sending we encounter Alert Orchestration Concept and solution Conclusion Summary and takeway
4. Company A Organization Structure • Startup(3+ years) • Service Team: 4 • RD: 30(Ops: 4, SRE: 1) • You build it, You run it
5. Data Pipeline
6. Team in pipeline
7. Monitoring Prometheus + AlertManger
8. Service B worker Lost No consumer on Queue
9. Issue 1 Notify Team B their worker does not consume job • manually mention team b member in channel • No alert activity • who is the responder ? ops_channel pipeline_team_channel
10. Issue 2 No one knows who is already read/received the alert • manually mention team b member in channel • No alert activity • who is the responder ?
11. Issue 3 Alert Routing issue with other team • Different Team may have different way to collect metric and receive alert • No Standard way to deliver alert to others in the same company
12. Issue 4 Alert Routing with Multiple region/dc tw-1 eu-1
13. Issues Issue 1 Multiple Region/dc Management Issue 3 No standard way to deliver alert across teams Issue 2 lack of Alert Observability alert management should decouple with monitoring tool
14. Alert Orchestration Alert Centralization Alert Routing On-call management & Alert Escalation Incident Management
15. SRE Conference 2022 - How to Build a Healthy On-Call Culture
16. Pagerduty
17. Opsgenie ref: https://serhatcan.medium.com/avoiding-alerts-overload-from-microservices-with-opsgenie-4f671f38e8af
18. Opsgenie ref: https://serhatcan.medium.com/avoiding-alerts-overload-from-microservices-with-opsgenie-4f671f38e8af Detect Response
19. Grafana OnCall ref:https://grafana.com/docs/oncall/latest/getting-started/
21. 這不是業配
22. Alert Centralization Team based integration
23. Alert Centralization • All Alert in one place • Ack !!!! • Alert Activity
24. Alert Centralization • All Alert in one place • Ack !!!! • Alert Activity
25. Alert Centralization Alert Activity Log Alert Fired notify next user in Team A notify on-call user in Team A no ack
26. Alert Escalation Notify users in a desired order Alert Fired notify next user in Team A notify on-call user in Team A no ack notify Admin in Team A no ack
27. Alert Escalation P1 Alert
28. Alert Escalation • daily/Weekly rotation • Work-hour and off-hour rotation On-call management
29. Alert Escalation • usecase 1 ◦ P1: Phone call ◦ P2: App ◦ P3~P5: email • usecase 2 ◦ off-hour low priority alert :email Notification method
30. • Alert policy in opsgenie • Alert enrichment/silence • Global Policy and Team based Policy • processed by ◦ alert message title ◦ tag ◦ Key-value Alert Routing routing and enrichment
31. routing and enrichment • Enrich alert • Global Policy and Team based Policy • processed by ◦ alert message title ◦ tag ◦ Key-value ◦ etc Alert Routing
32. Opsgenie ref: https://serhatcan.medium.com/avoiding-alerts-overload-from-microservices-with-opsgenie-4f671f38e8af
33. Incident Management Alert associated with Incident Timeline
34. Opsgenie ref: https://serhatcan.medium.com/avoiding-alerts-overload-from-microservices-with-opsgenie-4f671f38e8af
35. Incident Management Post incident analysis
36. Recap Prometheus + AlertManger
37. Recap(cond.) Issue 1 Multiple Region/dc Management Issue 3 No standard way to deliver alert across teams Issue 2 lack of Alert Observability alert management should decouple with monitoring tool
38. Alert Lifecycle ref:https://grafana.com/docs/oncall/latest/getting-started/
39. Takeaway • Define alert label first(e.g reigon, service) • Actionable alert / Alert Severity/ Alert Priority • Use managed service ◦ Pagerduty, opsgenie, grafana OnCall(cloud) etc. • Opsgenie must use Standard plan to use advanced alert routing feature
40. CREDITS: This presentation template was created by Slidesgo, including icons by Flaticon, and infographics & images by Freepik. Thanks! CREDITS: this presentation template was created by slidesgo, Flaticon, Freepik tsungtwu @tsungtwu