Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A Deep Dive into Monitoring with Skyline

A Deep Dive into Monitoring with Skyline

A talk I gave for the NYC Data Engineering Meetup at eBay. Video: http://g33ktalk.com/etsy-a-deep-dive-into-monitoring-with-skyline/

Abe Stanway

July 23, 2013
Tweet

More Decks by Abe Stanway

Other Decks in Programming

Transcript

  1. 41 shards 24 api servers 72 web servers 42 Gearman

    boxes 150 node Hadoop cluster 15 memcached boxes 60 search machines
  2. 41 shards 24 api servers 72 web servers 42 Gearman

    boxes 150 node Hadoop cluster 15 memcached boxes 60 search machines (plus a lot more for various services)
  3. de • ploy /diˈploi/ Verb To release your code for

    the world to see, hopefully without breaking the Internet
  4. [1358731200, 20] [1358731200, 20] [1358731200, 20] [1358731200, 20] [1358731200, 20]

    [1358731200, 20] [1358731200, 20] [1358731200, 20] [1358731200, 60] [1358731200, 20] [1358731200, 20]
  5. Graphite’s relay agent original graphite backup graphite [statsd.numStats, [1365603422, 82345]]

    pickles [statsd.numStats, [1365603432, 80611]] [statsd.numStats, [1365603412, 73421]]
  6. Graphite’s relay agent original graphite skyline [statsd.numStats, [1365603422, 82345]] pickles

    [statsd.numStats, [1365603432, 80611]] [statsd.numStats, [1365603412, 73421]]
  7. Basic algorithm: “A metric is anomalous if its latest datapoint

    is over three standard deviations above its moving average.”
  8. !=