Of the Order of Billions: Building Observability at Twitter

9128d500301ae51524e887bb680f471d?s=47 Caitie McCaffrey
October 21, 2015
500

Of the Order of Billions: Building Observability at Twitter

Every minute Twitter’s Observability stack processes 1.5+ billion metrics in order to provide Visibility into Twitter’s distributed microservices architecture. In this talk will focus on some of the challenges associated with building and running this large scale distributed system. We will also focus on lessons learned and how to build services that scale that are applicable for services of any size.

9128d500301ae51524e887bb680f471d?s=128

Caitie McCaffrey

October 21, 2015
Tweet

Transcript

  1. None
  2. Caitie McCaffrey Tech Lead Observability | @caitie Of the Order

    of Billions: Building Observability at Twitter
  3. Of the Order of Billions BUILDING OBSERVABILITY AT TWITTER

  4. Observability PROVIDES VISIBILITY INTO TWITTER SERVICES

  5. Dashboards & Charts

  6. Alerts

  7. Alerts KOALABIRD ALERTING

  8. 2 Billion METRICS PER MINUTE 1 MINUTE 10 MILLION METRICS

    OCTOBER, 2015
  9. 14 Days x 24 Hours 120 Billion Metrics x =

    1 Hour 40 Trillion Metrics 2 Billion Metrics OCTOBER, 2015
  10. ABSORBER SERVICE Serves requests, looks to Manhattan Life Of A

    Metric TWITTER SERVICE CUCKOO- WRITE CUCKOO- READ KOALABIRD VIZ Dashboards Manhattan Gathers Metrics Every Analyzes data & decides where it needs to be Alerts
  11. Building Highly Available, Scalable & Fault Tolerant Systems

  12. Idempotent Operations F(F(X) = F(X))

  13. Idempotent Operations F(F(X) = F(X)) ABSORBER CUCKOO-WRITE

  14. Idempotent Operations F(F(X) = F(X)) ABSORBER CUCKOO-WRITE

  15. Idempotent Operations F(F(X) = F(X)) ABSORBER CUCKOO-WRITE

  16. Avoid Coordination Minimizing coordination is key to maximizing scalability, availability,

    and high performance Value of n? n? N=2 N=2 N=?
  17. Avoid Coordination Minimizing coordination is key to maximizing scalability, availability,

    and high performance Value of n? N=2 N=2 N=? Value of n?
  18. Make Assumptions Explicit PRODUCER CONSUMER .

  19. Make Assumptions Explicit PRODUCER . CONSUMER

  20. Make Assumptions Explicit . CONSUMER PRODUCER

  21. Make Assumptions Explicit . CONSUMER PRODUCER

  22. Make Assumptions Explicit . CONSUMER PRODUCER

  23. Make Assumptions Explicit . CONSUMER PRODUCER

  24. Make Assumptions Explicit . CONSUMER PRODUCER

  25. Make Assumptions Explicit • Queue Lengths • Input Data Length

    • Requests Per Second • In Memory Data Structure Sizes
  26. SOME LESSONS LEARNED Idempotent Operations Avoid Coordination Make Assumptions Explicit

  27. Thank You @caitie