Of the Order of Billions: Building Observability at Twitter

9128d500301ae51524e887bb680f471d?s=47 Caitie McCaffrey
October 21, 2015
490

Of the Order of Billions: Building Observability at Twitter

Every minute Twitter’s Observability stack processes 1.5+ billion metrics in order to provide Visibility into Twitter’s distributed microservices architecture. In this talk will focus on some of the challenges associated with building and running this large scale distributed system. We will also focus on lessons learned and how to build services that scale that are applicable for services of any size.

9128d500301ae51524e887bb680f471d?s=128

Caitie McCaffrey

October 21, 2015
Tweet

Transcript

  1. 1.
  2. 2.

    Caitie McCaffrey Tech Lead Observability | @caitie Of the Order

    of Billions: Building Observability at Twitter
  3. 6.
  4. 9.

    14 Days x 24 Hours 120 Billion Metrics x =

    1 Hour 40 Trillion Metrics 2 Billion Metrics OCTOBER, 2015
  5. 10.

    ABSORBER SERVICE Serves requests, looks to Manhattan Life Of A

    Metric TWITTER SERVICE CUCKOO- WRITE CUCKOO- READ KOALABIRD VIZ Dashboards Manhattan Gathers Metrics Every Analyzes data & decides where it needs to be Alerts
  6. 17.
  7. 25.

    Make Assumptions Explicit • Queue Lengths • Input Data Length

    • Requests Per Second • In Memory Data Structure Sizes