Of the Order of Billions: Building Observability at Twitter

Every minute Twitter’s Observability stack processes 1.5+ billion metrics in order to provide Visibility into Twitter’s distributed microservices architecture. In this talk will focus on some of the challenges associated with building and running this large scale distributed system. We will also focus on lessons learned and how to build services that scale that are applicable for services of any size.

Transcript

2. Caitie McCaffrey Tech Lead Observability | @caitie Of the Order of Billions: Building Observability at Twitter
3. Of the Order of Billions BUILDING OBSERVABILITY AT TWITTER
4. Observability PROVIDES VISIBILITY INTO TWITTER SERVICES
5. Dashboards & Charts
6. Alerts
7. Alerts KOALABIRD ALERTING
8. 2 Billion METRICS PER MINUTE 1 MINUTE 10 MILLION METRICS OCTOBER, 2015
9. 14 Days x 24 Hours 120 Billion Metrics x = 1 Hour 40 Trillion Metrics 2 Billion Metrics OCTOBER, 2015
10. ABSORBER SERVICE Serves requests, looks to Manhattan Life Of A Metric TWITTER SERVICE CUCKOO- WRITE CUCKOO- READ KOALABIRD VIZ Dashboards Manhattan Gathers Metrics Every Analyzes data & decides where it needs to be Alerts
11. Building Highly Available, Scalable & Fault Tolerant Systems
12. Idempotent Operations F(F(X) = F(X))
13. Idempotent Operations F(F(X) = F(X)) ABSORBER CUCKOO-WRITE
14. Idempotent Operations F(F(X) = F(X)) ABSORBER CUCKOO-WRITE
15. Idempotent Operations F(F(X) = F(X)) ABSORBER CUCKOO-WRITE
16. Avoid Coordination Minimizing coordination is key to maximizing scalability, availability, and high performance Value of n? n? N=2 N=2 N=?
17. Avoid Coordination Minimizing coordination is key to maximizing scalability, availability, and high performance Value of n? N=2 N=2 N=? Value of n?
18. Make Assumptions Explicit PRODUCER CONSUMER .
19. Make Assumptions Explicit PRODUCER . CONSUMER
20. Make Assumptions Explicit . CONSUMER PRODUCER
21. Make Assumptions Explicit . CONSUMER PRODUCER
22. Make Assumptions Explicit . CONSUMER PRODUCER
23. Make Assumptions Explicit . CONSUMER PRODUCER
24. Make Assumptions Explicit . CONSUMER PRODUCER
25. Make Assumptions Explicit • Queue Lengths • Input Data Length • Requests Per Second • In Memory Data Structure Sizes
26. SOME LESSONS LEARNED Idempotent Operations Avoid Coordination Make Assumptions Explicit
27. Thank You @caitie