Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introduction of Skyline

Introduction of Skyline

Takumi Sakamoto

July 12, 2013
Tweet

More Decks by Takumi Sakamoto

Other Decks in Programming

Transcript

  1. My LT Talk at Velocity • 3 Popular Ops Tools

    in Japan • serverspec, growthforecast, fluentd • Uploaded on youtube :-( • http://www.youtube.com/watch? v=bRYuBQyG5Sw
  2. The Most Interesting Talk • Avoiding Performance Regression at Twitter

    • fight against perf regression in an automated fashion • http://ameblo.jp/principia-ca/ entry-11561132297.html
  3. Background • Etsy deploys their app 30+ times per day

    • Optimize for quick recovery by anticipating problems, instead of fearing human error • Can’t fix what you don’t measure! • If it moves, graph it
  4. Too Many Graphs • 250,000+ dashboards • If a graph

    spikes and no one is watching, does it really spike? • There are things we do not know we don’t know.
  5. Kale • Skyline • Detect unknown anomalies • Oculus •

    Detect unknown correlations http://codeascraft.com/2013/06/11/introducing-kale/
  6. Architecture Horizon Horizon Horizon Redis Analyzer Manager Analyzer Analyzer Analyzer

    Assign Metrics Keys Get Metrics Keys Fetch Timeseries Metrics
  7. Horizon • Listeners • Receiving metrics & store them to

    queue • Workers • Inserting metrics to redis with Messagepack • Roombas • Purging metrics in Redis at a regular interval
  8. Analyzer • Assigning Redis keys to each process process •

    decode from Messagepack • run the detection algorithm
  9. How to Detect Anomalies? • Consensus model • If the

    majority of algorithms agree, the metric will be classified as anomalous • Use your own algorithm for each application • as long as you return a boolean, you can add any sort of algorithm you like
  10. Basic Algorithm • A metric is anomalous if its latest

    datapoint is over three standard deviations above its moving average
  11. Conclusion • Etsy monitors 250,000+ graphs • If a graph

    spikes and no one is watching, does it really spike? • Skyline • detects unknown anomalies • consensus with any algorithms you like