Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Introduction of Skyline
Search
Takumi Sakamoto
July 12, 2013
Programming
0
160
Introduction of Skyline
Takumi Sakamoto
July 12, 2013
Tweet
Share
More Decks by Takumi Sakamoto
See All by Takumi Sakamoto
Building Data Pipelines with Apache Airflow
takus
3
2.5k
The Missing Pieces of Amazon ECS (for me)
takus
2
810
マイクロサービスとSRECon / #microserv
takus
14
4.4k
Slack と Incident Command System / #slackbook
takus
2
610
Why and How SmartNews uses SaaS?
takus
2
2.7k
スマートニュースの世界進出を支えるログ解析基盤 #jawsdays #tech
takus
63
18k
SmartNews と Datadog
takus
6
3.6k
ぜんぶ AWS でやらないワケ
takus
55
110k
Velocity Conference 2013 Report
takus
2
1.6k
Other Decks in Programming
See All in Programming
実践!RDRAを活用した既存システムの仕様変更 / Specification Changes in Existing Systems Utilizing RDRA
imamotohikaru
0
3k
ファイル先頭の use の意味、説明できますか? 〜PHP の namespace と autoloading の関係を正しく理解しよう〜 / namespace and autoloading in php
okashoi
2
470
ログラスを支える設計標準について / loglass-design-standards
urmot
2
200
Laravel OpenAPIによる"辛くない"スキーマ駆動開発
kentaroutakeda
2
2.1k
Dockerで始めるAWS Lambda開発
stutkhd0709
14
2.6k
Understanding Ast By Looking
inouehi
0
120
Why 1 + 1 = 2 in Swift?
1plus4
1
250
「コンパイル時のユニットテスト」導入するとユニットテストを 書かなくてよくなるのか?
tomohisa
9
2.2k
SwiftUI, Jetpack Composeの導入で変化した「家族アルバム みてね」のアプリ開発体験
hicka04
6
400
LPIXEL×CADDi_kaerururu
kaerururu
3
300
元気予報
suu_mire0726
0
230
クソコード動画『カプセル化 Mk-II』 で考える 上手くカプセル化できない理由 / encapsulation2
minodriven
11
8.2k
Featured
See All Featured
Making Projects Easy
brettharned
106
5.5k
The Art of Programming - Codeland 2020
erikaheidi
40
12k
The Illustrated Children's Guide to Kubernetes
chrisshort
28
46k
Web development in the modern age
philhawksworth
201
10k
Building Better People: How to give real-time feedback that sticks.
wjessup
350
18k
Designing the Hi-DPI Web
ddemaree
275
33k
Build your cross-platform service in a week with App Engine
jlugia
223
17k
Writing Fast Ruby
sferik
619
59k
Into the Great Unknown - MozCon
thekraken
10
830
Design by the Numbers
sachag
274
18k
VelocityConf: Rendering Performance Case Studies
addyosmani
319
23k
Designing with Data
zakiwarfel
94
4.8k
Transcript
Introduction of Skyline @takus monitoring casual #4 2013.07.12
Velocity 2013 • Web Performance & Operations Conference • Great
talks • Great people • Great culture
My LT Talk at Velocity • 3 Popular Ops Tools
in Japan • serverspec, growthforecast, fluentd • Uploaded on youtube :-( • http://www.youtube.com/watch? v=bRYuBQyG5Sw
Some Positive Feedbacks
The Most Interesting Talk • Avoiding Performance Regression at Twitter
• fight against perf regression in an automated fashion • http://ameblo.jp/principia-ca/ entry-11561132297.html
Today’s Topic
Background • Etsy deploys their app 30+ times per day
• Optimize for quick recovery by anticipating problems, instead of fearing human error • Can’t fix what you don’t measure! • If it moves, graph it
Too Many Graphs • 250,000+ dashboards • If a graph
spikes and no one is watching, does it really spike? • There are things we do not know we don’t know.
Kale • Skyline • Detect unknown anomalies • Oculus •
Detect unknown correlations http://codeascraft.com/2013/06/11/introducing-kale/
None
Architecture Horizon Horizon Horizon Redis Analyzer Manager Analyzer Analyzer Analyzer
Assign Metrics Keys Get Metrics Keys Fetch Timeseries Metrics
Horizon • Listeners • Receiving metrics & store them to
queue • Workers • Inserting metrics to redis with Messagepack • Roombas • Purging metrics in Redis at a regular interval
Analyzer • Assigning Redis keys to each process process •
decode from Messagepack • run the detection algorithm
How to Detect Anomalies? • Consensus model • If the
majority of algorithms agree, the metric will be classified as anomalous • Use your own algorithm for each application • as long as you return a boolean, you can add any sort of algorithm you like
Basic Algorithm • A metric is anomalous if its latest
datapoint is over three standard deviations above its moving average
Anomaly?
Conclusion • Etsy monitors 250,000+ graphs • If a graph
spikes and no one is watching, does it really spike? • Skyline • detects unknown anomalies • consensus with any algorithms you like