Pro Yearly is on sale from $80 to $50! »

Berlin 2013 - collectd Workshop - Florian Forster

0580d500edfdb2e5e80e4732ac8df1ea?s=47 Monitorama
September 20, 2013
470

Berlin 2013 - collectd Workshop - Florian Forster

0580d500edfdb2e5e80e4732ac8df1ea?s=128

Monitorama

September 20, 2013
Tweet

Transcript

  1. collectd Tracking custom metrics Florian “octo” Forster at monitorama.eu 2013

  2. Agenda • About myself and collectd • collectd data model

    • The exec plugin • The StatsD plugin • Other possibilities
  3. • Florian “octo” Forster ◦ Twitter: @flocto ◦ Github: octo

    • Started the collectd project in 2005 (while at university) • Still maintaining collectd as a spare time project About myself
  4. About collectd [1/3] • Free / open-source software ◦ Daemon:

    GPLv2 ◦ Plugins: Various free licenses (GPL, BSD, MIT) • Continuously running daemon • Specific logic in plugins ◦ Over 100 plugins, from AMQP to ZFS.
  5. About collectd [2/3] • “Output” options include: ◦ Graphite ◦

    MongoDB ◦ Riemann ◦ RRDtool / RRDCacheD • Various networking options
  6. About collectd [3/3] • Website: http://collectd.org/ • Twitter: @collectd •

    Github: collectd • Hackathon tomorrow at Michelberger Hotel.
  7. Agenda • About myself and collectd • collectd data model

    • The exec plugin • The StatsD plugin • Other possibilities
  8. Data model [1/5] • Four “data source types” • Most

    important: ◦ GAUGE Absolute value, e.g. temperature ◦ DERIVE Counting values, e.g. number of requests served • Special case: ◦ COUNTER
  9. Data model [2/5] Strict naming schema: • Hostname • Plugin

    • Type Plugin and type can have an “instance” each. Examples: • example.com • df – root • df_complex – free
  10. Data model [3/5] • The Type is defined in the

    types.db file (types.db(5) manual page). • Specifies the “data source type”, i.e. GAUGE or DERIVE. • Specifies minimum and maximum values.
  11. Data model [4/5] • Plugin instance is used to differentiate

    between “devices” the plugin can query. ◦ For example interfaces (“eth0”), CPU number (“0”), mount point (“var-spool”). • Type instance is used to to differentiate between multiple values of the same type. ◦ For example, number of bytes used, number of bytes reserved, number of bytes free on a file system.
  12. Data model [5/5] • Notation: • Examples: ◦ example.com/df-root/df_complex-free ◦

    example.com/cpu-0/cpu-wait ◦ example.com/memory/memory-used host/plugin-instance/type-instance host/plugin/type
  13. Agenda • About myself and collectd • collectd data model

    • The exec plugin • The StatsD plugin • Other possibilities
  14. The exec plugin • Executes arbitrary scripts / binaries •

    Scripts print metrics to STDOUT • Scripts print error messages to STDERR
  15. The exec plugin collectd exec plugin Custom script STDOUT /

    STDERR
  16. The exec plugin Configuration LoadPlugin "exec" <Plugin "exec"> Exec "gandalf:wizard"

    "/usr/lib/collectd/exec/magic.sh" # Exec "user:group" "/path/to/my/script" </Plugin>
  17. The exec plugin First version #!/bin/sh level=`magic-ctl --level` echo "PUTVAL

    \"example.com/exec/gauge-magic\" N:${level}"
  18. The exec plugin • Performance? • Problem: Forking is expensive.

    • Solution: Let scripts loop and keep them running.
  19. The exec plugin Second version #!/bin/sh while sleep 10; do

    level=`magic-ctl --level` echo "PUTVAL \"example.com/exec/gauge-magic\"" \ "interval=10 N:${level}" done
  20. The exec plugin • Automation? • Problem: Hard-coded hostname and

    interval must match collectd configuration. • Solution: Use environment variables set by collectd: COLLECTD_HOSTNAME and COLLECTD_INTERVAL.
  21. The exec plugin Final version #!/bin/sh interval=${COLLECTD_INTERVAL:-10} hostname=${COLLECTD_HOSTNAME:-localhost} while sleep

    ${interval}; do level=`magic-ctl --level` echo "PUTVAL \"${hostname}/exec/gauge-magic\" " \ "interval=${interval} N:${level}" done
  22. The exec plugin Caveat: Buffering • collectd uses pipes. •

    Many environments switch to (block) buffered I/O in this case. → Switch to line buffering or disable buffering.
  23. Agenda • About myself and collectd • collectd data model

    • The exec plugin • The StatsD plugin • Other possibilities
  24. The StatsD plugin • Opens a UDP socket and waits

    for asynchronous “events”. • Implements the “StatsD” network protocol. ◦ Wire compatible to Etsy's reference implementation. ◦ Lots of tooling available. • Very efficient.
  25. The StatsD plugin Configuration LoadPlugin "statsd" <Plugin "statsd"> DeleteCounters true

    DeleteSets false TimerPercentile 95 </Plugin>
  26. The StatsD plugin The StatsD protocol: • Four metric types:

    Counter, Timer, Gauge and Set. • Mapped to collectd “types” derive, latency, gauge and objects.
  27. The StatsD plugin The StatsD protocol: Counter • Increment “foo”

    by 3: ◦ foo:3|c • Mapped to: ◦ <host>/statsd/derive-foo
  28. The StatsD plugin The StatsD protocol: Timer • Operation “bar”

    took 3117 ms: ◦ bar:3117|ms • Mapped to: ◦ <host>/statsd/latency-bar-average ◦ <host>/statsd/latency-bar-percentile-95 ◦ …
  29. The StatsD plugin The StatsD protocol: Gauge • Set “qux”

    to 42: ◦ qux:42|g • Increase “qux” by 23: ◦ qux:+23|g • Mapped to: ◦ <host>/statsd/gauge-qux
  30. The StatsD plugin The StatsD protocol: Set • Add “bird”

    to set “animals”: ◦ animals:bird|s • Mapped to: ◦ <host>/statsd/objects-animals
  31. Agenda • About myself and collectd • collectd data model

    • The exec plugin • The StatsD plugin • Other possibilities
  32. Other possibilities Network protocol: • Send metrics in the “binary

    network protocol”. • Various client libraries available. • Virtually no coupling.
  33. Other possibilities UnixSock plugin • Listens on a UNIX domain

    socket and waits for commands, including PUTVAL. • Client libraries available. • Loose coupling.
  34. Other possibilities Java, Perl and Python plugins • Embed the

    VM / interpreters into the daemon (like mod_perl and friends). • “Best of two worlds” (YMMV) • Tight coupling
  35. Other possibilities C based plugins: • Play with the cool

    kids, drink the Kool-Aid and feel the power!
  36. Thank you! Please ask some questions now! ;-) Hackathon, tomorrow

    (September 21) at Michelberger Hotel, starting at 9:00.