Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Real-time application monitoring
Search
Igor Afonov
June 19, 2012
Programming
5
1k
Real-time application monitoring
Real-time application monitoring. Coffee and code Donetsk June 2012.
Igor Afonov
June 19, 2012
Tweet
Share
More Decks by Igor Afonov
See All by Igor Afonov
Application deployment with Chef
iafonov
6
980
Web servers - and how I created my own one
iafonov
8
1.2k
Chef - Infrastructure as code
iafonov
6
670
Other Decks in Programming
See All in Programming
テストカバレッジ100%を10年続けて得られた学びと品質
mottyzzz
2
610
はじめてのMaterial3 Expressive
ym223
2
910
複雑なフォームに立ち向かう Next.js の技術選定
macchiitaka
2
250
AIを活用し、今後に備えるための技術知識 / Basic Knowledge to Utilize AI
kishida
22
5.9k
もうちょっといいRubyプロファイラを作りたい (2025)
osyoyu
1
460
print("Hello, World")
eddie
2
530
デザイナーが Androidエンジニアに 挑戦してみた
874wokiite
0
570
Flutter with Dart MCP: All You Need - 박제창 2025 I/O Extended Busan
itsmedreamwalker
0
150
🔨 小さなビルドシステムを作る
momeemt
4
690
Updates on MLS on Ruby (and maybe more)
sylph01
1
180
アルテニア コンサル/ITエンジニア向け 採用ピッチ資料
altenir
0
110
Ruby×iOSアプリ開発 ~共に歩んだエコシステムの物語~
temoki
0
360
Featured
See All Featured
For a Future-Friendly Web
brad_frost
180
9.9k
A designer walks into a library…
pauljervisheath
207
24k
Rails Girls Zürich Keynote
gr2m
95
14k
Imperfection Machines: The Place of Print at Facebook
scottboms
268
13k
Visualization
eitanlees
148
16k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
35
3.1k
Automating Front-end Workflow
addyosmani
1370
200k
4 Signs Your Business is Dying
shpigford
184
22k
Agile that works and the tools we love
rasmusluckow
330
21k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
32
1.6k
The Power of CSS Pseudo Elements
geoffreycrofte
77
6k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
667
120k
Transcript
Real-time application monitoring Igor Afonov @iafonov
Background • SaaS application • 5 production servers (2 auxiliary
servers) • Everything managed by chef • Apache/Passenger/Rails/MySQL/Postfix
Metrics that matter • Server state • Application state
How metrics data is stored • Round-robin database • RRDTool
(C), Whisper (Python) (Minor offtopic)
Server state • Munin - storage and graphs • Monit
- alerts, basic rescue actions Shard munin-server munin-node Shard munin-node
Munin • Server pulls data from nodes • Gathers basic
server health data • A lot of custom plugins
Munin + Chef = — # Client config template munin_servers
= search(:node, "role:monitoring") <% munin_servers.sort.each do |server| -%> allow ^<%= server[:ipaddress].to_s.gsub(/\./, '\.') %>$ <% end %> # Server config template munin_nodes = search(:node, "munin:[* TO *]") <% munin_nodes.each do |system| -%> [<%= system[:hostname] %>] address <%= system[:ipaddress] %> use_node_name yes <% end %>
Application state • Subscriptions • Logins • Orders • Business
metrics • ...
Our setup Shard Monitoring Server StatsD Graphite Whisper Carbon WebApp
Shard Shard
StatsD • Lightweight proxy to graphite • 300 lines of
Javascript • Uses UDP (small size, non-blocking) • 10+ implementations • Simple protocol
StatsD • Count: counter:1|c • Measure: metric:200|ms • Gauge: value:9000|g
• Supports sampling • A lot of client-side libraries
StatsD StatsD.server = '178.22.33.88:8125' # increment counter StatsD.increment("users.new") # measure
task StatsD.measure("cron.#{task}") do task.run end # meta-programming - track subscriptions Subscription.extend StatsD::Instrument Subscription.statsd_count :subscribe, 'subscriptions'
Graphite • Python everywhere • Whisper - RRD, stores data
• Carbon - backend • Graphite - draws graphs, works with data
Graphite [stats] priority = 100 pattern = ^stats\..* retentions =
10:2160,60:10080,600:262974 (Storage schemas)
Problems • Basic UI • Complex installation and setup
telemetry.io • Fun project • SaaS application • Custom front-end
for StatsD + Graphite • Optimized for big screens (TVs) • Free • Maybe open-source
telemetry.io Client telemetry.io StatsD Graphite Whisper Carbon WebApp Custom Front-end
Client Client Client
Workflow • Get access token • Use one of available
libs or create own • Prepend token to metric name • Integrate into application StatsD.increment("#{token}.subscribers")
None
None
None
None
Implementation • Ruby on Rails • CoffeeScript • Chef (full
node bootstrap in 10 minutes)
Links • http://telemetry.io • http://code.flickr.com/blog/2008/10/27/ counting-timing/ • http://codeascraft.etsy.com/2011/02/15/ measure-anything-measure-everything/
http://iafonov.github.com/ @iafonov