Of the Order of Billions: Building Observability at Twitter

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

Caitie McCaﬀrey Tech Lead Observability | @caitie Of the Order of Billions: Building Observability at Twitter

Slide 3

Slide 3 text

Of the Order of Billions BUILDING OBSERVABILITY AT TWITTER

Slide 4

Slide 4 text

Observability PROVIDES VISIBILITY INTO TWITTER SERVICES

Slide 5

Slide 5 text

Dashboards & Charts

Slide 6

Slide 6 text

Alerts

Slide 7

Slide 7 text

Alerts KOALABIRD ALERTING

Slide 8

Slide 8 text

2 Billion METRICS PER MINUTE 1 MINUTE 10 MILLION METRICS OCTOBER, 2015

Slide 9

Slide 9 text

14 Days x 24 Hours 120 Billion Metrics x = 1 Hour 40 Trillion Metrics 2 Billion Metrics OCTOBER, 2015

Slide 10

Slide 10 text

ABSORBER SERVICE Serves requests, looks to Manhattan Life Of A Metric TWITTER SERVICE CUCKOO- WRITE CUCKOO- READ KOALABIRD VIZ Dashboards Manhattan Gathers Metrics Every Analyzes data & decides where it needs to be Alerts

Slide 11

Slide 11 text

Building Highly Available, Scalable & Fault Tolerant Systems

Slide 12

Slide 12 text

Idempotent Operations F(F(X) = F(X))

Slide 13

Slide 13 text

Idempotent Operations F(F(X) = F(X)) ABSORBER CUCKOO-WRITE

Slide 14

Slide 14 text

Idempotent Operations F(F(X) = F(X)) ABSORBER CUCKOO-WRITE

Slide 15

Slide 15 text

Idempotent Operations F(F(X) = F(X)) ABSORBER CUCKOO-WRITE

Slide 16

Slide 16 text

Avoid Coordination Minimizing coordination is key to maximizing scalability, availability, and high performance Value of n? n? N=2 N=2 N=?

Slide 17

Slide 17 text

Avoid Coordination Minimizing coordination is key to maximizing scalability, availability, and high performance Value of n? N=2 N=2 N=? Value of n?

Slide 18

Slide 18 text

Make Assumptions Explicit PRODUCER CONSUMER .

Slide 19