Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Real-time application monitoring
Search
Igor Afonov
June 19, 2012
Programming
5
970
Real-time application monitoring
Real-time application monitoring. Coffee and code Donetsk June 2012.
Igor Afonov
June 19, 2012
Tweet
Share
More Decks by Igor Afonov
See All by Igor Afonov
Application deployment with Chef
iafonov
6
790
Web servers - and how I created my own one
iafonov
8
1.1k
Chef - Infrastructure as code
iafonov
6
590
Other Decks in Programming
See All in Programming
StoreKit2によるiOSのアプリ内課金のリニューアル
kangnux
0
110
『Railsオワコン』と言われる時代に、なぜブルーモ証券はRailsを選ぶのか
free_world21
0
190
AWS CDKコントリビュートTIPS / aws-cdk-contribution-tips
gotok365
2
180
educure_カリキュラム生操作マニュアル.pdf
linew_official
0
790
Apache Hive 4 on Treasure Data
ryukobayashi
0
290
大規模Reactアプリのリアーキテクチャ~8万行のTanStack Query移行の軌跡~
kj455
4
960
雑に思考を整理する技術と効能
konifar
59
29k
使ってみよう Azure AI Document Intelligence
kosmosebi
2
310
0→1と1→10の狭間で Javaという技術選定を振り返る/Reflecting on the Decision to Choose Java Between Scaling from 0 to 1 and 1 to 10
jaguar_imo
2
380
Rubyでたのしむクリエイティブコーディング/Enjoy Creative coding with Ruby
chobishiba
1
180
Zero Waste, Radical Magic, and Italian Graft – Quarkus Efficiency Secrets
hollycummins
0
230
Komplexe Oberflächen mit SVG und der Web Animation API
joergneumann
0
670
Featured
See All Featured
Fantastic passwords and where to find them - at NoRuKo
philnash
37
2.5k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
9
8.3k
Intergalactic Javascript Robots from Outer Space
tanoku
266
26k
The Cult of Friendly URLs
andyhume
74
5.7k
How GitHub (no longer) Works
holman
304
140k
Making the Leap to Tech Lead
cromwellryan
124
8.5k
Clear Off the Table
cherdarchuk
84
310k
Git: the NoSQL Database
bkeepers
PRO
422
63k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
116
18k
Large-scale JavaScript Application Architecture
addyosmani
504
110k
Six Lessons from altMBA
skipperchong
21
3k
How to train your dragon (web standard)
notwaldorf
73
5.2k
Transcript
Real-time application monitoring Igor Afonov @iafonov
Background • SaaS application • 5 production servers (2 auxiliary
servers) • Everything managed by chef • Apache/Passenger/Rails/MySQL/Postfix
Metrics that matter • Server state • Application state
How metrics data is stored • Round-robin database • RRDTool
(C), Whisper (Python) (Minor offtopic)
Server state • Munin - storage and graphs • Monit
- alerts, basic rescue actions Shard munin-server munin-node Shard munin-node
Munin • Server pulls data from nodes • Gathers basic
server health data • A lot of custom plugins
Munin + Chef = — # Client config template munin_servers
= search(:node, "role:monitoring") <% munin_servers.sort.each do |server| -%> allow ^<%= server[:ipaddress].to_s.gsub(/\./, '\.') %>$ <% end %> # Server config template munin_nodes = search(:node, "munin:[* TO *]") <% munin_nodes.each do |system| -%> [<%= system[:hostname] %>] address <%= system[:ipaddress] %> use_node_name yes <% end %>
Application state • Subscriptions • Logins • Orders • Business
metrics • ...
Our setup Shard Monitoring Server StatsD Graphite Whisper Carbon WebApp
Shard Shard
StatsD • Lightweight proxy to graphite • 300 lines of
Javascript • Uses UDP (small size, non-blocking) • 10+ implementations • Simple protocol
StatsD • Count: counter:1|c • Measure: metric:200|ms • Gauge: value:9000|g
• Supports sampling • A lot of client-side libraries
StatsD StatsD.server = '178.22.33.88:8125' # increment counter StatsD.increment("users.new") # measure
task StatsD.measure("cron.#{task}") do task.run end # meta-programming - track subscriptions Subscription.extend StatsD::Instrument Subscription.statsd_count :subscribe, 'subscriptions'
Graphite • Python everywhere • Whisper - RRD, stores data
• Carbon - backend • Graphite - draws graphs, works with data
Graphite [stats] priority = 100 pattern = ^stats\..* retentions =
10:2160,60:10080,600:262974 (Storage schemas)
Problems • Basic UI • Complex installation and setup
telemetry.io • Fun project • SaaS application • Custom front-end
for StatsD + Graphite • Optimized for big screens (TVs) • Free • Maybe open-source
telemetry.io Client telemetry.io StatsD Graphite Whisper Carbon WebApp Custom Front-end
Client Client Client
Workflow • Get access token • Use one of available
libs or create own • Prepend token to metric name • Integrate into application StatsD.increment("#{token}.subscribers")
None
None
None
None
Implementation • Ruby on Rails • CoffeeScript • Chef (full
node bootstrap in 10 minutes)
Links • http://telemetry.io • http://code.flickr.com/blog/2008/10/27/ counting-timing/ • http://codeascraft.etsy.com/2011/02/15/ measure-anything-measure-everything/
http://iafonov.github.com/ @iafonov