Berlin 2013 - collectd Workshop - Florian Forster

Slide 1

Slide 1 text

collectd Tracking custom metrics Florian “octo” Forster at monitorama.eu 2013

Slide 2

Slide 2 text

Agenda ● About myself and collectd ● collectd data model ● The exec plugin ● The StatsD plugin ● Other possibilities

Slide 3

Slide 3 text

● Florian “octo” Forster ○ Twitter: @flocto ○ Github: octo ● Started the collectd project in 2005 (while at university) ● Still maintaining collectd as a spare time project About myself

Slide 4

Slide 4 text

About collectd [1/3] ● Free / open-source software ○ Daemon: GPLv2 ○ Plugins: Various free licenses (GPL, BSD, MIT) ● Continuously running daemon ● Specific logic in plugins ○ Over 100 plugins, from AMQP to ZFS.

Slide 5

Slide 5 text

About collectd [2/3] ● “Output” options include: ○ Graphite ○ MongoDB ○ Riemann ○ RRDtool / RRDCacheD ● Various networking options

Slide 6

Slide 6 text

About collectd [3/3] ● Website: http://collectd.org/ ● Twitter: @collectd ● Github: collectd ● Hackathon tomorrow at Michelberger Hotel.

Slide 7

Slide 7 text

Agenda ● About myself and collectd ● collectd data model ● The exec plugin ● The StatsD plugin ● Other possibilities

Slide 8

Slide 8 text

Data model [1/5] ● Four “data source types” ● Most important: ○ GAUGE Absolute value, e.g. temperature ○ DERIVE Counting values, e.g. number of requests served ● Special case: ○ COUNTER

Slide 9

Slide 9 text

Data model [2/5] Strict naming schema: ● Hostname ● Plugin ● Type Plugin and type can have an “instance” each. Examples: ● example.com ● df – root ● df_complex – free

Slide 10

Slide 10 text

Data model [3/5] ● The Type is defined in the types.db file (types.db(5) manual page). ● Specifies the “data source type”, i.e. GAUGE or DERIVE. ● Specifies minimum and maximum values.

Slide 11

Slide 11 text

Data model [4/5] ● Plugin instance is used to differentiate between “devices” the plugin can query. ○ For example interfaces (“eth0”), CPU number (“0”), mount point (“var-spool”). ● Type instance is used to to differentiate between multiple values of the same type. ○ For example, number of bytes used, number of bytes reserved, number of bytes free on a file system.

Slide 12

Slide 12 text

Data model [5/5] ● Notation: ● Examples: ○ example.com/df-root/df_complex-free ○ example.com/cpu-0/cpu-wait ○ example.com/memory/memory-used host/plugin-instance/type-instance host/plugin/type

Slide 13

Slide 13 text

Agenda ● About myself and collectd ● collectd data model ● The exec plugin ● The StatsD plugin ● Other possibilities

Slide 14

Slide 14 text

The exec plugin ● Executes arbitrary scripts / binaries ● Scripts print metrics to STDOUT ● Scripts print error messages to STDERR

Slide 15

Slide 15 text

The exec plugin collectd exec plugin Custom script STDOUT / STDERR

Slide 16

Slide 16 text

The exec plugin Configuration LoadPlugin "exec" Exec "gandalf:wizard" "/usr/lib/collectd/exec/magic.sh" # Exec "user:group" "/path/to/my/script"

Slide 17

Slide 17 text

The exec plugin First version #!/bin/sh level=`magic-ctl --level` echo "PUTVAL \"example.com/exec/gauge-magic\" N:${level}"

Slide 18

Slide 18 text

The exec plugin ● Performance? ● Problem: Forking is expensive. ● Solution: Let scripts loop and keep them running.

Slide 19

Slide 19 text

The exec plugin Second version #!/bin/sh while sleep 10; do level=`magic-ctl --level` echo "PUTVAL \"example.com/exec/gauge-magic\"" \ "interval=10 N:${level}" done

Slide 20

Slide 20 text

The exec plugin ● Automation? ● Problem: Hard-coded hostname and interval must match collectd configuration. ● Solution: Use environment variables set by collectd: COLLECTD_HOSTNAME and COLLECTD_INTERVAL.

Slide 21

Slide 21 text

The exec plugin Final version #!/bin/sh interval=${COLLECTD_INTERVAL:-10} hostname=${COLLECTD_HOSTNAME:-localhost} while sleep ${interval}; do level=`magic-ctl --level` echo "PUTVAL \"${hostname}/exec/gauge-magic\" " \ "interval=${interval} N:${level}" done

Slide 22

Slide 22 text

The exec plugin Caveat: Buffering ● collectd uses pipes. ● Many environments switch to (block) buffered I/O in this case. → Switch to line buffering or disable buffering.

Slide 23

Slide 23 text

Agenda ● About myself and collectd ● collectd data model ● The exec plugin ● The StatsD plugin ● Other possibilities

Slide 24

Slide 24 text

The StatsD plugin ● Opens a UDP socket and waits for asynchronous “events”. ● Implements the “StatsD” network protocol. ○ Wire compatible to Etsy's reference implementation. ○ Lots of tooling available. ● Very efficient.

Slide 25

Slide 25 text

The StatsD plugin Configuration LoadPlugin "statsd" DeleteCounters true DeleteSets false TimerPercentile 95

Slide 26

Slide 26 text

The StatsD plugin The StatsD protocol: ● Four metric types: Counter, Timer, Gauge and Set. ● Mapped to collectd “types” derive, latency, gauge and objects.

Slide 27

Slide 27 text

The StatsD plugin The StatsD protocol: Counter ● Increment “foo” by 3: ○ foo:3|c ● Mapped to: ○ /statsd/derive-foo

Slide 28

Slide 28 text

The StatsD plugin The StatsD protocol: Timer ● Operation “bar” took 3117 ms: ○ bar:3117|ms ● Mapped to: ○ /statsd/latency-bar-average ○ /statsd/latency-bar-percentile-95 ○ …

Slide 29

Slide 29 text

The StatsD plugin The StatsD protocol: Gauge ● Set “qux” to 42: ○ qux:42|g ● Increase “qux” by 23: ○ qux:+23|g ● Mapped to: ○ /statsd/gauge-qux

Slide 30

Slide 30 text

The StatsD plugin The StatsD protocol: Set ● Add “bird” to set “animals”: ○ animals:bird|s ● Mapped to: ○ /statsd/objects-animals

Slide 31

Slide 31 text

Agenda ● About myself and collectd ● collectd data model ● The exec plugin ● The StatsD plugin ● Other possibilities

Slide 32

Slide 32 text

Other possibilities Network protocol: ● Send metrics in the “binary network protocol”. ● Various client libraries available. ● Virtually no coupling.

Slide 33

Slide 33 text

Other possibilities UnixSock plugin ● Listens on a UNIX domain socket and waits for commands, including PUTVAL. ● Client libraries available. ● Loose coupling.

Slide 34

Slide 34 text

Other possibilities Java, Perl and Python plugins ● Embed the VM / interpreters into the daemon (like mod_perl and friends). ● “Best of two worlds” (YMMV) ● Tight coupling

Slide 35

Slide 35 text

Other possibilities C based plugins: ● Play with the cool kids, drink the Kool-Aid and feel the power!

Slide 36

Slide 36 text

Thank you! Please ask some questions now! ;-) Hackathon, tomorrow (September 21) at Michelberger Hotel, starting at 9:00.