Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Real-time application monitoring
Search
Igor Afonov
June 19, 2012
Programming
5
970
Real-time application monitoring
Real-time application monitoring. Coffee and code Donetsk June 2012.
Igor Afonov
June 19, 2012
Tweet
Share
More Decks by Igor Afonov
See All by Igor Afonov
Application deployment with Chef
iafonov
6
790
Web servers - and how I created my own one
iafonov
8
1.1k
Chef - Infrastructure as code
iafonov
6
590
Other Decks in Programming
See All in Programming
SIMD Parallel Programming with the Vector API
josepaumard
0
220
デフォルトにして至高、RubyMineの大好きな所
ruzia
0
640
DMMプラットフォームがTiDB Cloudを採用した背景
pospome
9
4.2k
AWS CDKコントリビュートTIPS / aws-cdk-contribution-tips
gotok365
3
310
StoreKit2によるiOSのアプリ内課金のリニューアル
kangnux
0
120
効率化に挑戦してみたらモバイル開発が少し快適になった話
ryunakayama
0
130
禅の心を手に入れよ
eltociear
1
270
はてなにおける CSS Modules、及び CSS Modules に足りないもの / CSS Modules in Hatena, and CSS Modules missing parts
mizdra
7
960
GitHub Copilotのススメ
marcy731
1
220
スキーマ駆動開発による品質とスピードの両立 - 私達は何故、スキーマを書くのか
kentaroutakeda
0
170
From Spring Boot 2 to Spring Boot 3 with Java 21 and Jakarta EE
ivargrimstad
0
390
OpenAPIを中心に考えるAPI開発入門 / Introduction to API Development with a Focus on OpenAPI
seike460
PRO
2
170
Featured
See All Featured
Mobile First: as difficult as doing things right
swwweet
217
8.6k
Agile that works and the tools we love
rasmusluckow
325
20k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
358
22k
Done Done
chrislema
178
15k
In The Pink: A Labor of Love
frogandcode
138
21k
The Straight Up "How To Draw Better" Workshop
denniskardys
228
130k
How GitHub Uses GitHub to Build GitHub
holman
468
290k
The Art of Programming - Codeland 2020
erikaheidi
43
12k
Optimising Largest Contentful Paint
csswizardry
12
2.4k
Adopting Sorbet at Scale
ufuk
69
8.6k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
155
14k
Product Roadmaps are Hard
iamctodd
45
9.7k
Transcript
Real-time application monitoring Igor Afonov @iafonov
Background • SaaS application • 5 production servers (2 auxiliary
servers) • Everything managed by chef • Apache/Passenger/Rails/MySQL/Postfix
Metrics that matter • Server state • Application state
How metrics data is stored • Round-robin database • RRDTool
(C), Whisper (Python) (Minor offtopic)
Server state • Munin - storage and graphs • Monit
- alerts, basic rescue actions Shard munin-server munin-node Shard munin-node
Munin • Server pulls data from nodes • Gathers basic
server health data • A lot of custom plugins
Munin + Chef = — # Client config template munin_servers
= search(:node, "role:monitoring") <% munin_servers.sort.each do |server| -%> allow ^<%= server[:ipaddress].to_s.gsub(/\./, '\.') %>$ <% end %> # Server config template munin_nodes = search(:node, "munin:[* TO *]") <% munin_nodes.each do |system| -%> [<%= system[:hostname] %>] address <%= system[:ipaddress] %> use_node_name yes <% end %>
Application state • Subscriptions • Logins • Orders • Business
metrics • ...
Our setup Shard Monitoring Server StatsD Graphite Whisper Carbon WebApp
Shard Shard
StatsD • Lightweight proxy to graphite • 300 lines of
Javascript • Uses UDP (small size, non-blocking) • 10+ implementations • Simple protocol
StatsD • Count: counter:1|c • Measure: metric:200|ms • Gauge: value:9000|g
• Supports sampling • A lot of client-side libraries
StatsD StatsD.server = '178.22.33.88:8125' # increment counter StatsD.increment("users.new") # measure
task StatsD.measure("cron.#{task}") do task.run end # meta-programming - track subscriptions Subscription.extend StatsD::Instrument Subscription.statsd_count :subscribe, 'subscriptions'
Graphite • Python everywhere • Whisper - RRD, stores data
• Carbon - backend • Graphite - draws graphs, works with data
Graphite [stats] priority = 100 pattern = ^stats\..* retentions =
10:2160,60:10080,600:262974 (Storage schemas)
Problems • Basic UI • Complex installation and setup
telemetry.io • Fun project • SaaS application • Custom front-end
for StatsD + Graphite • Optimized for big screens (TVs) • Free • Maybe open-source
telemetry.io Client telemetry.io StatsD Graphite Whisper Carbon WebApp Custom Front-end
Client Client Client
Workflow • Get access token • Use one of available
libs or create own • Prepend token to metric name • Integrate into application StatsD.increment("#{token}.subscribers")
None
None
None
None
Implementation • Ruby on Rails • CoffeeScript • Chef (full
node bootstrap in 10 minutes)
Links • http://telemetry.io • http://code.flickr.com/blog/2008/10/27/ counting-timing/ • http://codeascraft.etsy.com/2011/02/15/ measure-anything-measure-everything/
http://iafonov.github.com/ @iafonov