Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Real-time application monitoring
Search
Igor Afonov
June 19, 2012
Programming
5
1k
Real-time application monitoring
Real-time application monitoring. Coffee and code Donetsk June 2012.
Igor Afonov
June 19, 2012
Tweet
Share
More Decks by Igor Afonov
See All by Igor Afonov
Application deployment with Chef
iafonov
6
890
Web servers - and how I created my own one
iafonov
8
1.1k
Chef - Infrastructure as code
iafonov
6
630
Other Decks in Programming
See All in Programming
AWS Lambdaから始まった Serverlessの「熱」とキャリアパス / It started with AWS Lambda Serverless “fever” and career path
seike460
PRO
1
260
弊社の「意識チョット低いアーキテクチャ」10選
texmeijin
5
24k
WebフロントエンドにおけるGraphQL(あるいはバックエンドのAPI)との向き合い方 / #241106_plk_frontend
izumin5210
4
1.4k
ふかぼれ!CSSセレクターモジュール / Fukabore! CSS Selectors Module
petamoriken
0
150
最新TCAキャッチアップ
0si43
0
140
ヤプリ新卒SREの オンボーディング
masaki12
0
130
CSC509 Lecture 12
javiergs
PRO
0
160
Less waste, more joy, and a lot more green: How Quarkus makes Java better
hollycummins
0
100
詳細解説! ArrayListの仕組みと実装
yujisoftware
0
580
レガシーシステムにどう立ち向かうか 複雑さと理想と現実/vs-legacy
suzukihoge
14
2.2k
2024/11/8 関西Kaggler会 2024 #3 / Kaggle Kernel で Gemma 2 × vLLM を動かす。
kohecchi
5
920
.NET のための通信フレームワーク MagicOnion 入門 / Introduction to MagicOnion
mayuki
1
1.5k
Featured
See All Featured
Optimizing for Happiness
mojombo
376
70k
10 Git Anti Patterns You Should be Aware of
lemiorhan
654
59k
Statistics for Hackers
jakevdp
796
220k
Designing on Purpose - Digital PM Summit 2013
jponch
115
7k
jQuery: Nuts, Bolts and Bling
dougneiner
61
7.5k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
42
9.2k
Speed Design
sergeychernyshev
24
610
How to Ace a Technical Interview
jacobian
276
23k
Scaling GitHub
holman
458
140k
Fontdeck: Realign not Redesign
paulrobertlloyd
82
5.2k
Building Applications with DynamoDB
mza
90
6.1k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
0
89
Transcript
Real-time application monitoring Igor Afonov @iafonov
Background • SaaS application • 5 production servers (2 auxiliary
servers) • Everything managed by chef • Apache/Passenger/Rails/MySQL/Postfix
Metrics that matter • Server state • Application state
How metrics data is stored • Round-robin database • RRDTool
(C), Whisper (Python) (Minor offtopic)
Server state • Munin - storage and graphs • Monit
- alerts, basic rescue actions Shard munin-server munin-node Shard munin-node
Munin • Server pulls data from nodes • Gathers basic
server health data • A lot of custom plugins
Munin + Chef = — # Client config template munin_servers
= search(:node, "role:monitoring") <% munin_servers.sort.each do |server| -%> allow ^<%= server[:ipaddress].to_s.gsub(/\./, '\.') %>$ <% end %> # Server config template munin_nodes = search(:node, "munin:[* TO *]") <% munin_nodes.each do |system| -%> [<%= system[:hostname] %>] address <%= system[:ipaddress] %> use_node_name yes <% end %>
Application state • Subscriptions • Logins • Orders • Business
metrics • ...
Our setup Shard Monitoring Server StatsD Graphite Whisper Carbon WebApp
Shard Shard
StatsD • Lightweight proxy to graphite • 300 lines of
Javascript • Uses UDP (small size, non-blocking) • 10+ implementations • Simple protocol
StatsD • Count: counter:1|c • Measure: metric:200|ms • Gauge: value:9000|g
• Supports sampling • A lot of client-side libraries
StatsD StatsD.server = '178.22.33.88:8125' # increment counter StatsD.increment("users.new") # measure
task StatsD.measure("cron.#{task}") do task.run end # meta-programming - track subscriptions Subscription.extend StatsD::Instrument Subscription.statsd_count :subscribe, 'subscriptions'
Graphite • Python everywhere • Whisper - RRD, stores data
• Carbon - backend • Graphite - draws graphs, works with data
Graphite [stats] priority = 100 pattern = ^stats\..* retentions =
10:2160,60:10080,600:262974 (Storage schemas)
Problems • Basic UI • Complex installation and setup
telemetry.io • Fun project • SaaS application • Custom front-end
for StatsD + Graphite • Optimized for big screens (TVs) • Free • Maybe open-source
telemetry.io Client telemetry.io StatsD Graphite Whisper Carbon WebApp Custom Front-end
Client Client Client
Workflow • Get access token • Use one of available
libs or create own • Prepend token to metric name • Integrate into application StatsD.increment("#{token}.subscribers")
None
None
None
None
Implementation • Ruby on Rails • CoffeeScript • Chef (full
node bootstrap in 10 minutes)
Links • http://telemetry.io • http://code.flickr.com/blog/2008/10/27/ counting-timing/ • http://codeascraft.etsy.com/2011/02/15/ measure-anything-measure-everything/
http://iafonov.github.com/ @iafonov