Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Introduction of Skyline
Search
Takumi Sakamoto
July 12, 2013
Programming
0
170
Introduction of Skyline
Takumi Sakamoto
July 12, 2013
Tweet
Share
More Decks by Takumi Sakamoto
See All by Takumi Sakamoto
Building Data Pipelines with Apache Airflow
takus
3
2.7k
The Missing Pieces of Amazon ECS (for me)
takus
2
890
マイクロサービスとSRECon / #microserv
takus
14
4.5k
Slack と Incident Command System / #slackbook
takus
2
670
Why and How SmartNews uses SaaS?
takus
2
2.9k
スマートニュースの世界進出を支えるログ解析基盤 #jawsdays #tech
takus
64
18k
SmartNews と Datadog
takus
6
3.9k
ぜんぶ AWS でやらないワケ
takus
55
110k
Velocity Conference 2013 Report
takus
2
1.7k
Other Decks in Programming
See All in Programming
ML.NETで始める機械学習
ymd65536
0
210
Rails アプリ地図考 Flush Cut
makicamel
1
120
GAEログのコスト削減
mot_techtalk
0
120
sappoRo.R #12 初心者セッション
kosugitti
0
260
クリーンアーキテクチャから見る依存の向きの大切さ
shimabox
4
880
PRレビューのお供にDanger
stoticdev
1
200
DROBEの生成AI活用事例 with AWS
ippey
0
140
Amazon ECS とマイクロサービスから考えるシステム構成
hiyanger
2
580
Pythonでもちょっとリッチな見た目のアプリを設計してみる
ueponx
1
600
pylint custom ruleで始めるレビュー自動化
shogoujiie
0
130
Software Architecture
hschwentner
6
2.1k
データの整合性を保つ非同期処理アーキテクチャパターン / Async Architecture Patterns
mokuo
53
18k
Featured
See All Featured
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
We Have a Design System, Now What?
morganepeng
51
7.4k
Code Reviewing Like a Champion
maltzj
521
39k
How GitHub (no longer) Works
holman
314
140k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
44
7k
VelocityConf: Rendering Performance Case Studies
addyosmani
328
24k
Site-Speed That Sticks
csswizardry
4
390
Producing Creativity
orderedlist
PRO
344
39k
Faster Mobile Websites
deanohume
306
31k
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
656
59k
Unsuck your backbone
ammeep
669
57k
Designing on Purpose - Digital PM Summit 2013
jponch
117
7.1k
Transcript
Introduction of Skyline @takus monitoring casual #4 2013.07.12
Velocity 2013 • Web Performance & Operations Conference • Great
talks • Great people • Great culture
My LT Talk at Velocity • 3 Popular Ops Tools
in Japan • serverspec, growthforecast, fluentd • Uploaded on youtube :-( • http://www.youtube.com/watch? v=bRYuBQyG5Sw
Some Positive Feedbacks
The Most Interesting Talk • Avoiding Performance Regression at Twitter
• fight against perf regression in an automated fashion • http://ameblo.jp/principia-ca/ entry-11561132297.html
Today’s Topic
Background • Etsy deploys their app 30+ times per day
• Optimize for quick recovery by anticipating problems, instead of fearing human error • Can’t fix what you don’t measure! • If it moves, graph it
Too Many Graphs • 250,000+ dashboards • If a graph
spikes and no one is watching, does it really spike? • There are things we do not know we don’t know.
Kale • Skyline • Detect unknown anomalies • Oculus •
Detect unknown correlations http://codeascraft.com/2013/06/11/introducing-kale/
None
Architecture Horizon Horizon Horizon Redis Analyzer Manager Analyzer Analyzer Analyzer
Assign Metrics Keys Get Metrics Keys Fetch Timeseries Metrics
Horizon • Listeners • Receiving metrics & store them to
queue • Workers • Inserting metrics to redis with Messagepack • Roombas • Purging metrics in Redis at a regular interval
Analyzer • Assigning Redis keys to each process process •
decode from Messagepack • run the detection algorithm
How to Detect Anomalies? • Consensus model • If the
majority of algorithms agree, the metric will be classified as anomalous • Use your own algorithm for each application • as long as you return a boolean, you can add any sort of algorithm you like
Basic Algorithm • A metric is anomalous if its latest
datapoint is over three standard deviations above its moving average
Anomaly?
Conclusion • Etsy monitors 250,000+ graphs • If a graph
spikes and no one is watching, does it really spike? • Skyline • detects unknown anomalies • consensus with any algorithms you like