Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Real-time application monitoring

Real-time application monitoring

Real-time application monitoring. Coffee and code Donetsk June 2012.

Igor Afonov

June 19, 2012
Tweet

More Decks by Igor Afonov

Other Decks in Programming

Transcript

  1. Background • SaaS application • 5 production servers (2 auxiliary

    servers) • Everything managed by chef • Apache/Passenger/Rails/MySQL/Postfix
  2. Server state • Munin - storage and graphs • Monit

    - alerts, basic rescue actions Shard munin-server munin-node Shard munin-node
  3. Munin • Server pulls data from nodes • Gathers basic

    server health data • A lot of custom plugins
  4. Munin + Chef = — # Client config template munin_servers

    = search(:node, "role:monitoring") <% munin_servers.sort.each do |server| -%> allow ^<%= server[:ipaddress].to_s.gsub(/\./, '\.') %>$ <% end %> # Server config template munin_nodes = search(:node, "munin:[* TO *]") <% munin_nodes.each do |system| -%> [<%= system[:hostname] %>] address <%= system[:ipaddress] %> use_node_name yes <% end %>
  5. StatsD • Lightweight proxy to graphite • 300 lines of

    Javascript • Uses UDP (small size, non-blocking) • 10+ implementations • Simple protocol
  6. StatsD • Count: counter:1|c • Measure: metric:200|ms • Gauge: value:9000|g

    • Supports sampling • A lot of client-side libraries
  7. StatsD StatsD.server = '178.22.33.88:8125' # increment counter StatsD.increment("users.new") # measure

    task StatsD.measure("cron.#{task}") do task.run end # meta-programming - track subscriptions Subscription.extend StatsD::Instrument Subscription.statsd_count :subscribe, 'subscriptions'
  8. Graphite • Python everywhere • Whisper - RRD, stores data

    • Carbon - backend • Graphite - draws graphs, works with data
  9. Graphite [stats] priority = 100 pattern = ^stats\..* retentions =

    10:2160,60:10080,600:262974 (Storage schemas)
  10. telemetry.io • Fun project • SaaS application • Custom front-end

    for StatsD + Graphite • Optimized for big screens (TVs) • Free • Maybe open-source
  11. Workflow • Get access token • Use one of available

    libs or create own • Prepend token to metric name • Integrate into application StatsD.increment("#{token}.subscribers")