Simple is beautiful: Building an SLO tracker at PagerDuty

Deck for the talk I gave at Code BEAM STO 2018 at Stockholm, Sweden.
Speaker Page: https://web.archive.org/web/20180603142224/https://codesync.global/speaker/aish-dahal/
Conference Page: https://web.archive.org/web/20180603142235/https://www.codesync.global/conferences/code-beam-sto-2018/

Transcript

1. Building an SLA tracking tool Aish Raj Dahal PagerDuty
2. Aish
4. Never build your own monitoring. Someone on Hacker News “
5. But……why ? A future comment on Hacker News “
6. …because sometimes there are a few “special” cases
7. Chapter I The problem
8. Enqueuer Pre-processor Processor Incoming Record Cassandra Poll REST API State of the world
 Circa 2014
9. How many of these entities were actually processed ?
10. How do we know if the processing speed is not fast enough ?
11. Some… None… All of them… ?
12. Chapter II The original solution
13. Enqueuer Pre-processor Processor Incoming Record Cassandra Poll REST API Pre metrics
 Circa 2014
14. Enqueuer Pre-processor Processor Incoming Record Cassandra Poll REST API Metrics Processor Cassandra Poll REST API Poll Post metrics
 Circa 2015
15. What was it ?
16. It was a handcrafted custom script
17. What did it do ?
18. Cassandra Metrics Processor Processor RPC REST API Poll The insides
 Or how we polled all the time
19. What did this mean?
20. We were polling our Cassandra Queue
21. We were polling our processor’s db.
22. Additional load made things worse
23. Our polling exacerbated the problem
24. Chapter III Enter Elixir
25. We replaced Cassandra based
 in-house queue with Kafka
26. Complexity is the root cause of the vast majority of problems with software today. Out of the Tar Pit (2006) “
27. Enqueuer Pre-processor Processor Incoming Record Kafka Poll REST API State of the world
 Circa 2017
28. This meant…
29. RPC based lookups were off the table
30. Polling the processing database was off the table
31. We needed a new way of measuring the processing status
32. Approach I Let’s store everything in ElasticSearch
33. Rather than querying state in two places we queried state in one place.
34. We built a pipeline to send the records and their state to a downstream store, ElasticSearch
35. Incoming Records Phoenix App Kafka Processed Records Elastic Search Approach I ElasticSearch Metrics Collector
36. It was okay until we had one small problem
37. The system crumbled when load increased.
38. We had just moved from polling one datastore to polling another
39. Approach II ETS all Streams
40. All our records were events
41. Processed event record was incoming event with some additional metadata
42. All events were streams
43. Solution: zipWith two Streams
44. Incoming Records ETS Kafka Processed Records Approach II
 ETS Metrics Collector
45. Why did we do it?
46. Memory access is thousands of times faster than random disk access
47. Modeling the data in its natural form as events was more intuitive
48. We only wanted the processing status for events within a window
49. We had fewer moving parts. Simple is beautiful indeed.
50. Epilogue Current State
51. Dealing with crashes
52. Recreate state from Kafka
53. Reconcile if required with a secondary datastore.
54. And if stuff still breaks…get paged
55. aishrajdahal PS: PagerDuty’s hiring