Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Real-time application monitoring
Search
Igor Afonov
June 19, 2012
Programming
5
1k
Real-time application monitoring
Real-time application monitoring. Coffee and code Donetsk June 2012.
Igor Afonov
June 19, 2012
Tweet
Share
More Decks by Igor Afonov
See All by Igor Afonov
Application deployment with Chef
iafonov
6
880
Web servers - and how I created my own one
iafonov
8
1.1k
Chef - Infrastructure as code
iafonov
6
630
Other Decks in Programming
See All in Programming
LLM生成文章の精度評価自動化とプロンプトチューニングの効率化について
layerx
PRO
2
140
Streams APIとTCPフロー制御 / Web Streams API and TCP flow control
tasshi
2
320
Amazon Neptuneで始めてみるグラフDB-OpenSearchによるグラフの全文検索-
satoshi256kbyte
4
340
讓數據說話:用 Python、Prometheus 和 Grafana 講故事
eddie
0
360
cXML という電子商取引の トランザクションを支える プロトコルと向きあっている話
phigasui
3
2.3k
AWS IaCの注目アップデート 2024年10月版
konokenj
3
3.2k
CSC509 Lecture 08
javiergs
PRO
0
110
Kaigi on Rails 2024 - Rails APIモードのためのシンプルで効果的なCSRF対策 / kaigionrails-2024-csrf
corocn
5
3.5k
ピラミッド、アイスクリームコーン、SMURF: 自動テストの最適バランスを求めて / Pyramid Ice-Cream-Cone and SMURF
twada
PRO
9
1k
Webの技術スタックで マルチプラットフォームアプリ開発を可能にするElixirDesktopの紹介
thehaigo
2
990
開発効率向上のためのリファクタリングの一歩目の選択肢 ~コード分割~ / JJUG CCC 2024 Fall
ryounasso
0
370
NSOutlineView何もわからん:( 前編 / I Don't Understand About NSOutlineView :( Pt. 1
usagimaru
0
210
Featured
See All Featured
Happy Clients
brianwarren
97
6.7k
Build your cross-platform service in a week with App Engine
jlugia
229
18k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
47
5k
Facilitating Awesome Meetings
lara
49
6.1k
GraphQLとの向き合い方2022年版
quramy
43
13k
Typedesign – Prime Four
hannesfritz
39
2.4k
Learning to Love Humans: Emotional Interface Design
aarron
272
40k
Art, The Web, and Tiny UX
lynnandtonic
297
20k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
4
380
Documentation Writing (for coders)
carmenintech
65
4.4k
Optimising Largest Contentful Paint
csswizardry
33
2.9k
Teambox: Starting and Learning
jrom
132
8.7k
Transcript
Real-time application monitoring Igor Afonov @iafonov
Background • SaaS application • 5 production servers (2 auxiliary
servers) • Everything managed by chef • Apache/Passenger/Rails/MySQL/Postfix
Metrics that matter • Server state • Application state
How metrics data is stored • Round-robin database • RRDTool
(C), Whisper (Python) (Minor offtopic)
Server state • Munin - storage and graphs • Monit
- alerts, basic rescue actions Shard munin-server munin-node Shard munin-node
Munin • Server pulls data from nodes • Gathers basic
server health data • A lot of custom plugins
Munin + Chef = — # Client config template munin_servers
= search(:node, "role:monitoring") <% munin_servers.sort.each do |server| -%> allow ^<%= server[:ipaddress].to_s.gsub(/\./, '\.') %>$ <% end %> # Server config template munin_nodes = search(:node, "munin:[* TO *]") <% munin_nodes.each do |system| -%> [<%= system[:hostname] %>] address <%= system[:ipaddress] %> use_node_name yes <% end %>
Application state • Subscriptions • Logins • Orders • Business
metrics • ...
Our setup Shard Monitoring Server StatsD Graphite Whisper Carbon WebApp
Shard Shard
StatsD • Lightweight proxy to graphite • 300 lines of
Javascript • Uses UDP (small size, non-blocking) • 10+ implementations • Simple protocol
StatsD • Count: counter:1|c • Measure: metric:200|ms • Gauge: value:9000|g
• Supports sampling • A lot of client-side libraries
StatsD StatsD.server = '178.22.33.88:8125' # increment counter StatsD.increment("users.new") # measure
task StatsD.measure("cron.#{task}") do task.run end # meta-programming - track subscriptions Subscription.extend StatsD::Instrument Subscription.statsd_count :subscribe, 'subscriptions'
Graphite • Python everywhere • Whisper - RRD, stores data
• Carbon - backend • Graphite - draws graphs, works with data
Graphite [stats] priority = 100 pattern = ^stats\..* retentions =
10:2160,60:10080,600:262974 (Storage schemas)
Problems • Basic UI • Complex installation and setup
telemetry.io • Fun project • SaaS application • Custom front-end
for StatsD + Graphite • Optimized for big screens (TVs) • Free • Maybe open-source
telemetry.io Client telemetry.io StatsD Graphite Whisper Carbon WebApp Custom Front-end
Client Client Client
Workflow • Get access token • Use one of available
libs or create own • Prepend token to metric name • Integrate into application StatsD.increment("#{token}.subscribers")
None
None
None
None
Implementation • Ruby on Rails • CoffeeScript • Chef (full
node bootstrap in 10 minutes)
Links • http://telemetry.io • http://code.flickr.com/blog/2008/10/27/ counting-timing/ • http://codeascraft.etsy.com/2011/02/15/ measure-anything-measure-everything/
http://iafonov.github.com/ @iafonov