Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Introduction of Skyline
Search
Takumi Sakamoto
July 12, 2013
Programming
0
170
Introduction of Skyline
Takumi Sakamoto
July 12, 2013
Tweet
Share
More Decks by Takumi Sakamoto
See All by Takumi Sakamoto
Building Data Pipelines with Apache Airflow
takus
3
2.8k
The Missing Pieces of Amazon ECS (for me)
takus
2
910
マイクロサービスとSRECon / #microserv
takus
14
4.5k
Slack と Incident Command System / #slackbook
takus
2
700
Why and How SmartNews uses SaaS?
takus
2
3k
スマートニュースの世界進出を支えるログ解析基盤 #jawsdays #tech
takus
64
18k
SmartNews と Datadog
takus
6
3.9k
ぜんぶ AWS でやらないワケ
takus
55
110k
Velocity Conference 2013 Report
takus
2
1.7k
Other Decks in Programming
See All in Programming
UPDATEがシステムを複雑にする? イミュータブルデータモデルのすすめ
shimomura
1
530
Perplexity Slack Botを作ってAI活用を進めた話 / AI Engineering Summit プレイベント
n3xem
0
640
List Unfolding - 'unfold' as the Computational Dual of 'fold', and how 'unfold' relates to 'iterate'"
philipschwarz
PRO
0
190
Cline指示通りに動かない? AI小説エージェントで学ぶ指示書の書き方と自動アップデートの仕組み
kamomeashizawa
1
440
2度もゼロから書き直して、やっとブラウザでぬるぬる動くAIに辿り着いた話
tomoino
0
150
Passkeys for Java Developers
ynojima
2
840
レガシーシステムの機能調査・開発におけるAI利活用
takuya_ohtonari
0
590
Java on Azure で LangGraph!
kohei3110
0
110
eBPFを用いたAIネットワーク監視システム論文の実装 / eBPF Japan Meetup #4
yuukit
3
750
SODA - FACT BOOK
sodainc
1
840
Parallel::Pipesの紹介
skaji
2
910
既存デザインを変更せずにタップ領域を広げる方法
tahia910
1
140
Featured
See All Featured
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
45
7.3k
Visualization
eitanlees
146
16k
Speed Design
sergeychernyshev
30
990
KATA
mclloyd
29
14k
Why You Should Never Use an ORM
jnunemaker
PRO
56
9.4k
Writing Fast Ruby
sferik
628
61k
Facilitating Awesome Meetings
lara
54
6.4k
Rails Girls Zürich Keynote
gr2m
94
14k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
2.8k
How to Ace a Technical Interview
jacobian
276
23k
Statistics for Hackers
jakevdp
799
220k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
130
19k
Transcript
Introduction of Skyline @takus monitoring casual #4 2013.07.12
Velocity 2013 • Web Performance & Operations Conference • Great
talks • Great people • Great culture
My LT Talk at Velocity • 3 Popular Ops Tools
in Japan • serverspec, growthforecast, fluentd • Uploaded on youtube :-( • http://www.youtube.com/watch? v=bRYuBQyG5Sw
Some Positive Feedbacks
The Most Interesting Talk • Avoiding Performance Regression at Twitter
• fight against perf regression in an automated fashion • http://ameblo.jp/principia-ca/ entry-11561132297.html
Today’s Topic
Background • Etsy deploys their app 30+ times per day
• Optimize for quick recovery by anticipating problems, instead of fearing human error • Can’t fix what you don’t measure! • If it moves, graph it
Too Many Graphs • 250,000+ dashboards • If a graph
spikes and no one is watching, does it really spike? • There are things we do not know we don’t know.
Kale • Skyline • Detect unknown anomalies • Oculus •
Detect unknown correlations http://codeascraft.com/2013/06/11/introducing-kale/
None
Architecture Horizon Horizon Horizon Redis Analyzer Manager Analyzer Analyzer Analyzer
Assign Metrics Keys Get Metrics Keys Fetch Timeseries Metrics
Horizon • Listeners • Receiving metrics & store them to
queue • Workers • Inserting metrics to redis with Messagepack • Roombas • Purging metrics in Redis at a regular interval
Analyzer • Assigning Redis keys to each process process •
decode from Messagepack • run the detection algorithm
How to Detect Anomalies? • Consensus model • If the
majority of algorithms agree, the metric will be classified as anomalous • Use your own algorithm for each application • as long as you return a boolean, you can add any sort of algorithm you like
Basic Algorithm • A metric is anomalous if its latest
datapoint is over three standard deviations above its moving average
Anomaly?
Conclusion • Etsy monitors 250,000+ graphs • If a graph
spikes and no one is watching, does it really spike? • Skyline • detects unknown anomalies • consensus with any algorithms you like