Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Real-time application monitoring
Search
Igor Afonov
June 19, 2012
Programming
5
1k
Real-time application monitoring
Real-time application monitoring. Coffee and code Donetsk June 2012.
Igor Afonov
June 19, 2012
Tweet
Share
More Decks by Igor Afonov
See All by Igor Afonov
Application deployment with Chef
iafonov
6
970
Web servers - and how I created my own one
iafonov
8
1.2k
Chef - Infrastructure as code
iafonov
6
670
Other Decks in Programming
See All in Programming
FindyにおけるTakumi活用と脆弱性管理のこれから
rvirus0817
0
370
AIエージェント開発、DevOps and LLMOps
ymd65536
1
370
Jakarta EE Core Profile and Helidon - Speed, Simplicity, and AI Integration
ivargrimstad
0
310
AWS発のAIエディタKiroを使ってみた
iriikeita
1
140
奥深くて厄介な「改行」と仲良くなる20分
oguemon
1
280
testingを眺める
matumoto
1
130
Laravel Boost 超入門
fire_arlo
2
180
Ruby Parser progress report 2025
yui_knk
1
260
AIでLINEスタンプを作ってみた
eycjur
1
220
Langfuseと歩む生成AI活用推進
licux
3
320
The Past, Present, and Future of Enterprise Java
ivargrimstad
0
150
Namespace and Its Future
tagomoris
6
690
Featured
See All Featured
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
358
30k
How to Think Like a Performance Engineer
csswizardry
26
1.9k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
333
22k
Balancing Empowerment & Direction
lara
3
610
VelocityConf: Rendering Performance Case Studies
addyosmani
332
24k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
18
1.1k
Building a Modern Day E-commerce SEO Strategy
aleyda
43
7.5k
Rebuilding a faster, lazier Slack
samanthasiow
83
9.1k
A designer walks into a library…
pauljervisheath
207
24k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
We Have a Design System, Now What?
morganepeng
53
7.8k
Being A Developer After 40
akosma
90
590k
Transcript
Real-time application monitoring Igor Afonov @iafonov
Background • SaaS application • 5 production servers (2 auxiliary
servers) • Everything managed by chef • Apache/Passenger/Rails/MySQL/Postfix
Metrics that matter • Server state • Application state
How metrics data is stored • Round-robin database • RRDTool
(C), Whisper (Python) (Minor offtopic)
Server state • Munin - storage and graphs • Monit
- alerts, basic rescue actions Shard munin-server munin-node Shard munin-node
Munin • Server pulls data from nodes • Gathers basic
server health data • A lot of custom plugins
Munin + Chef = — # Client config template munin_servers
= search(:node, "role:monitoring") <% munin_servers.sort.each do |server| -%> allow ^<%= server[:ipaddress].to_s.gsub(/\./, '\.') %>$ <% end %> # Server config template munin_nodes = search(:node, "munin:[* TO *]") <% munin_nodes.each do |system| -%> [<%= system[:hostname] %>] address <%= system[:ipaddress] %> use_node_name yes <% end %>
Application state • Subscriptions • Logins • Orders • Business
metrics • ...
Our setup Shard Monitoring Server StatsD Graphite Whisper Carbon WebApp
Shard Shard
StatsD • Lightweight proxy to graphite • 300 lines of
Javascript • Uses UDP (small size, non-blocking) • 10+ implementations • Simple protocol
StatsD • Count: counter:1|c • Measure: metric:200|ms • Gauge: value:9000|g
• Supports sampling • A lot of client-side libraries
StatsD StatsD.server = '178.22.33.88:8125' # increment counter StatsD.increment("users.new") # measure
task StatsD.measure("cron.#{task}") do task.run end # meta-programming - track subscriptions Subscription.extend StatsD::Instrument Subscription.statsd_count :subscribe, 'subscriptions'
Graphite • Python everywhere • Whisper - RRD, stores data
• Carbon - backend • Graphite - draws graphs, works with data
Graphite [stats] priority = 100 pattern = ^stats\..* retentions =
10:2160,60:10080,600:262974 (Storage schemas)
Problems • Basic UI • Complex installation and setup
telemetry.io • Fun project • SaaS application • Custom front-end
for StatsD + Graphite • Optimized for big screens (TVs) • Free • Maybe open-source
telemetry.io Client telemetry.io StatsD Graphite Whisper Carbon WebApp Custom Front-end
Client Client Client
Workflow • Get access token • Use one of available
libs or create own • Prepend token to metric name • Integrate into application StatsD.increment("#{token}.subscribers")
None
None
None
None
Implementation • Ruby on Rails • CoffeeScript • Chef (full
node bootstrap in 10 minutes)
Links • http://telemetry.io • http://code.flickr.com/blog/2008/10/27/ counting-timing/ • http://codeascraft.etsy.com/2011/02/15/ measure-anything-measure-everything/
http://iafonov.github.com/ @iafonov