Introduction of Skyline

Transcript

1. Introduction of Skyline @takus monitoring casual #4 2013.07.12
2. Velocity 2013 • Web Performance & Operations Conference • Great talks • Great people • Great culture
3. My LT Talk at Velocity • 3 Popular Ops Tools in Japan • serverspec, growthforecast, fluentd • Uploaded on youtube :-( • http://www.youtube.com/watch? v=bRYuBQyG5Sw
4. Some Positive Feedbacks
5. The Most Interesting Talk • Avoiding Performance Regression at Twitter • fight against perf regression in an automated fashion • http://ameblo.jp/principia-ca/ entry-11561132297.html
6. Today’s Topic
7. Background • Etsy deploys their app 30+ times per day • Optimize for quick recovery by anticipating problems, instead of fearing human error • Can’t fix what you don’t measure! • If it moves, graph it
8. Too Many Graphs • 250,000+ dashboards • If a graph spikes and no one is watching, does it really spike? • There are things we do not know we don’t know.
9. Kale • Skyline • Detect unknown anomalies • Oculus • Detect unknown correlations http://codeascraft.com/2013/06/11/introducing-kale/
11. Architecture Horizon Horizon Horizon Redis Analyzer Manager Analyzer Analyzer Analyzer Assign Metrics Keys Get Metrics Keys Fetch Timeseries Metrics
12. Horizon • Listeners • Receiving metrics & store them to queue • Workers • Inserting metrics to redis with Messagepack • Roombas • Purging metrics in Redis at a regular interval
13. Analyzer • Assigning Redis keys to each process process • decode from Messagepack • run the detection algorithm
14. How to Detect Anomalies? • Consensus model • If the majority of algorithms agree, the metric will be classified as anomalous • Use your own algorithm for each application • as long as you return a boolean, you can add any sort of algorithm you like
15. Basic Algorithm • A metric is anomalous if its latest datapoint is over three standard deviations above its moving average
16. Anomaly?
17. Conclusion • Etsy monitors 250,000+ graphs • If a graph spikes and no one is watching, does it really spike? • Skyline • detects unknown anomalies • consensus with any algorithms you like