$30 off During Our Annual Pro Sale. View Details »

Berlin 2013 - collectd Workshop - Florian Forster

Monitorama
September 20, 2013
560

Berlin 2013 - collectd Workshop - Florian Forster

Monitorama

September 20, 2013
Tweet

Transcript

  1. collectd
    Tracking custom metrics
    Florian “octo” Forster at monitorama.eu 2013

    View Slide

  2. Agenda
    ● About myself and collectd
    ● collectd data model
    ● The exec plugin
    ● The StatsD plugin
    ● Other possibilities

    View Slide

  3. ● Florian “octo” Forster
    ○ Twitter: @flocto
    ○ Github: octo
    ● Started the collectd project in 2005
    (while at university)
    ● Still maintaining collectd as a spare time
    project
    About myself

    View Slide

  4. About collectd [1/3]
    ● Free / open-source software
    ○ Daemon: GPLv2
    ○ Plugins: Various free licenses (GPL, BSD, MIT)
    ● Continuously running daemon
    ● Specific logic in plugins
    ○ Over 100 plugins, from AMQP to ZFS.

    View Slide

  5. About collectd [2/3]
    ● “Output” options include:
    ○ Graphite
    ○ MongoDB
    ○ Riemann
    ○ RRDtool / RRDCacheD
    ● Various networking options

    View Slide

  6. About collectd [3/3]
    ● Website: http://collectd.org/
    ● Twitter: @collectd
    ● Github: collectd
    ● Hackathon tomorrow at Michelberger Hotel.

    View Slide

  7. Agenda
    ● About myself and collectd
    ● collectd data model
    ● The exec plugin
    ● The StatsD plugin
    ● Other possibilities

    View Slide

  8. Data model [1/5]
    ● Four “data source types”
    ● Most important:
    ○ GAUGE
    Absolute value, e.g. temperature
    ○ DERIVE
    Counting values, e.g. number of requests served
    ● Special case:
    ○ COUNTER

    View Slide

  9. Data model [2/5]
    Strict naming schema:
    ● Hostname
    ● Plugin
    ● Type
    Plugin and type can
    have an “instance”
    each.
    Examples:
    ● example.com
    ● df – root
    ● df_complex – free

    View Slide

  10. Data model [3/5]
    ● The Type is defined in the types.db file
    (types.db(5) manual page).
    ● Specifies the “data source type”, i.e. GAUGE
    or DERIVE.
    ● Specifies minimum and maximum values.

    View Slide

  11. Data model [4/5]
    ● Plugin instance is used to differentiate
    between “devices” the plugin can query.
    ○ For example interfaces (“eth0”), CPU number (“0”),
    mount point (“var-spool”).
    ● Type instance is used to to differentiate
    between multiple values of the same type.
    ○ For example, number of bytes used, number of
    bytes reserved, number of bytes free on a file
    system.

    View Slide

  12. Data model [5/5]
    ● Notation:
    ● Examples:
    ○ example.com/df-root/df_complex-free
    ○ example.com/cpu-0/cpu-wait
    ○ example.com/memory/memory-used
    host/plugin-instance/type-instance
    host/plugin/type

    View Slide

  13. Agenda
    ● About myself and collectd
    ● collectd data model
    ● The exec plugin
    ● The StatsD plugin
    ● Other possibilities

    View Slide

  14. The exec plugin
    ● Executes arbitrary scripts / binaries
    ● Scripts print metrics to STDOUT
    ● Scripts print error messages to STDERR

    View Slide

  15. The exec plugin
    collectd
    exec plugin
    Custom script
    STDOUT / STDERR

    View Slide

  16. The exec plugin
    Configuration
    LoadPlugin "exec"

    Exec "gandalf:wizard" "/usr/lib/collectd/exec/magic.sh"
    # Exec "user:group" "/path/to/my/script"

    View Slide

  17. The exec plugin
    First version
    #!/bin/sh
    level=`magic-ctl --level`
    echo "PUTVAL \"example.com/exec/gauge-magic\" N:${level}"

    View Slide

  18. The exec plugin
    ● Performance?
    ● Problem: Forking is expensive.
    ● Solution: Let scripts loop and keep them
    running.

    View Slide

  19. The exec plugin
    Second version
    #!/bin/sh
    while sleep 10; do
    level=`magic-ctl --level`
    echo "PUTVAL \"example.com/exec/gauge-magic\"" \
    "interval=10 N:${level}"
    done

    View Slide

  20. The exec plugin
    ● Automation?
    ● Problem: Hard-coded hostname and
    interval must match collectd configuration.
    ● Solution: Use environment variables set by
    collectd: COLLECTD_HOSTNAME and
    COLLECTD_INTERVAL.

    View Slide

  21. The exec plugin
    Final version
    #!/bin/sh
    interval=${COLLECTD_INTERVAL:-10}
    hostname=${COLLECTD_HOSTNAME:-localhost}
    while sleep ${interval}; do
    level=`magic-ctl --level`
    echo "PUTVAL \"${hostname}/exec/gauge-magic\" " \
    "interval=${interval} N:${level}"
    done

    View Slide

  22. The exec plugin
    Caveat: Buffering
    ● collectd uses pipes.
    ● Many environments switch to (block)
    buffered I/O in this case.
    → Switch to line buffering or disable buffering.

    View Slide

  23. Agenda
    ● About myself and collectd
    ● collectd data model
    ● The exec plugin
    ● The StatsD plugin
    ● Other possibilities

    View Slide

  24. The StatsD plugin
    ● Opens a UDP socket and waits for
    asynchronous “events”.
    ● Implements the “StatsD” network protocol.
    ○ Wire compatible to Etsy's reference implementation.
    ○ Lots of tooling available.
    ● Very efficient.

    View Slide

  25. The StatsD plugin
    Configuration
    LoadPlugin "statsd"

    DeleteCounters true
    DeleteSets false
    TimerPercentile 95

    View Slide

  26. The StatsD plugin
    The StatsD protocol:
    ● Four metric types: Counter, Timer, Gauge
    and Set.
    ● Mapped to collectd “types” derive,
    latency, gauge and objects.

    View Slide

  27. The StatsD plugin
    The StatsD protocol: Counter
    ● Increment “foo” by 3:
    ○ foo:3|c
    ● Mapped to:
    ○ /statsd/derive-foo

    View Slide

  28. The StatsD plugin
    The StatsD protocol: Timer
    ● Operation “bar” took 3117 ms:
    ○ bar:3117|ms
    ● Mapped to:
    ○ /statsd/latency-bar-average
    ○ /statsd/latency-bar-percentile-95
    ○ …

    View Slide

  29. The StatsD plugin
    The StatsD protocol: Gauge
    ● Set “qux” to 42:
    ○ qux:42|g
    ● Increase “qux” by 23:
    ○ qux:+23|g
    ● Mapped to:
    ○ /statsd/gauge-qux

    View Slide

  30. The StatsD plugin
    The StatsD protocol: Set
    ● Add “bird” to set “animals”:
    ○ animals:bird|s
    ● Mapped to:
    ○ /statsd/objects-animals

    View Slide

  31. Agenda
    ● About myself and collectd
    ● collectd data model
    ● The exec plugin
    ● The StatsD plugin
    ● Other possibilities

    View Slide

  32. Other possibilities
    Network protocol:
    ● Send metrics in the “binary network
    protocol”.
    ● Various client libraries available.
    ● Virtually no coupling.

    View Slide

  33. Other possibilities
    UnixSock plugin
    ● Listens on a UNIX domain socket and waits
    for commands, including PUTVAL.
    ● Client libraries available.
    ● Loose coupling.

    View Slide

  34. Other possibilities
    Java, Perl and Python plugins
    ● Embed the VM / interpreters into the
    daemon (like mod_perl and friends).
    ● “Best of two worlds” (YMMV)
    ● Tight coupling

    View Slide

  35. Other possibilities
    C based plugins:
    ● Play with the cool kids, drink the Kool-Aid
    and feel the power!

    View Slide

  36. Thank you!
    Please ask some questions now! ;-)
    Hackathon, tomorrow (September 21) at
    Michelberger Hotel, starting at 9:00.

    View Slide