Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Real-time application monitoring
Search
Igor Afonov
June 19, 2012
Programming
5
960
Real-time application monitoring
Real-time application monitoring. Coffee and code Donetsk June 2012.
Igor Afonov
June 19, 2012
Tweet
Share
More Decks by Igor Afonov
See All by Igor Afonov
Application deployment with Chef
iafonov
6
770
Web servers - and how I created my own one
iafonov
8
1.1k
Chef - Infrastructure as code
iafonov
6
590
Other Decks in Programming
See All in Programming
incrementalモデルの理解を深める
ikkimiyazaki
2
640
【KMC春合宿2024】実装視点で見るNeural Radiance Fields
runningoutrate
0
150
TCAの Shared Stateって どういう仕組みになってんの?
yimajo
0
330
Dockerで始めるAWS Lambda開発
stutkhd0709
14
2.5k
htmx is fun!
codehex
2
190
PHPerKaigi 2024〜10年以上動いているレガシーなバッチシステムを Kubernetes(Amazon EKS) に移行する取り組み〜
tshinowpub
1
220
OpenAPI を守るのは難しい
ohmori_yusuke
1
140
自動テスト実行結果の目的を整理する / Organizing objectives of automated test results
twada
PRO
10
2.1k
品質とスピードを両立: TypeScriptの柔軟な型システムをバックエンドで活用する
kosui
5
1.1k
コミュニティに参加したことで起きた変化
ohmori_yusuke
3
130
[スクリプト] Swiftの型推論を学ぼう
omochi
0
120
Deep Dive 大規模システムアーキテクチャ/開発組織エンジニアリング / Deep Dive Large-Scale System Architecture, Development Organization Engineering
nrslib
15
2.9k
Featured
See All Featured
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
19
1.9k
Happy Clients
brianwarren
91
6.3k
The Cult of Friendly URLs
andyhume
73
5.6k
What the flash - Photography Introduction
edds
64
11k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
225
51k
Build your cross-platform service in a week with App Engine
jlugia
223
17k
KATA
mclloyd
14
11k
Navigating Team Friction
lara
177
13k
How GitHub (no longer) Works
holman
301
140k
Web development in the modern age
philhawksworth
201
10k
A Tale of Four Properties
chriscoyier
150
22k
The Illustrated Children's Guide to Kubernetes
chrisshort
28
46k
Transcript
Real-time application monitoring Igor Afonov @iafonov
Background • SaaS application • 5 production servers (2 auxiliary
servers) • Everything managed by chef • Apache/Passenger/Rails/MySQL/Postfix
Metrics that matter • Server state • Application state
How metrics data is stored • Round-robin database • RRDTool
(C), Whisper (Python) (Minor offtopic)
Server state • Munin - storage and graphs • Monit
- alerts, basic rescue actions Shard munin-server munin-node Shard munin-node
Munin • Server pulls data from nodes • Gathers basic
server health data • A lot of custom plugins
Munin + Chef = — # Client config template munin_servers
= search(:node, "role:monitoring") <% munin_servers.sort.each do |server| -%> allow ^<%= server[:ipaddress].to_s.gsub(/\./, '\.') %>$ <% end %> # Server config template munin_nodes = search(:node, "munin:[* TO *]") <% munin_nodes.each do |system| -%> [<%= system[:hostname] %>] address <%= system[:ipaddress] %> use_node_name yes <% end %>
Application state • Subscriptions • Logins • Orders • Business
metrics • ...
Our setup Shard Monitoring Server StatsD Graphite Whisper Carbon WebApp
Shard Shard
StatsD • Lightweight proxy to graphite • 300 lines of
Javascript • Uses UDP (small size, non-blocking) • 10+ implementations • Simple protocol
StatsD • Count: counter:1|c • Measure: metric:200|ms • Gauge: value:9000|g
• Supports sampling • A lot of client-side libraries
StatsD StatsD.server = '178.22.33.88:8125' # increment counter StatsD.increment("users.new") # measure
task StatsD.measure("cron.#{task}") do task.run end # meta-programming - track subscriptions Subscription.extend StatsD::Instrument Subscription.statsd_count :subscribe, 'subscriptions'
Graphite • Python everywhere • Whisper - RRD, stores data
• Carbon - backend • Graphite - draws graphs, works with data
Graphite [stats] priority = 100 pattern = ^stats\..* retentions =
10:2160,60:10080,600:262974 (Storage schemas)
Problems • Basic UI • Complex installation and setup
telemetry.io • Fun project • SaaS application • Custom front-end
for StatsD + Graphite • Optimized for big screens (TVs) • Free • Maybe open-source
telemetry.io Client telemetry.io StatsD Graphite Whisper Carbon WebApp Custom Front-end
Client Client Client
Workflow • Get access token • Use one of available
libs or create own • Prepend token to metric name • Integrate into application StatsD.increment("#{token}.subscribers")
None
None
None
None
Implementation • Ruby on Rails • CoffeeScript • Chef (full
node bootstrap in 10 minutes)
Links • http://telemetry.io • http://code.flickr.com/blog/2008/10/27/ counting-timing/ • http://codeascraft.etsy.com/2011/02/15/ measure-anything-measure-everything/
http://iafonov.github.com/ @iafonov