Berlin 2013 - collectd Workshop - Florian Forster

Transcript

1. collectd
   Tracking custom metrics
   Florian “octo” Forster at monitorama.eu 2013
   

   View Slide

2. Agenda
   ● About myself and collectd
   ● collectd data model
   ● The exec plugin
   ● The StatsD plugin
   ● Other possibilities
   

   View Slide

3. ● Florian “octo” Forster
   ○ Twitter: @flocto
   ○ Github: octo
   ● Started the collectd project in 2005
   (while at university)
   ● Still maintaining collectd as a spare time
   project
   About myself
   

   View Slide

4. About collectd [1/3]
   ● Free / open-source software
   ○ Daemon: GPLv2
   ○ Plugins: Various free licenses (GPL, BSD, MIT)
   ● Continuously running daemon
   ● Specific logic in plugins
   ○ Over 100 plugins, from AMQP to ZFS.
   

   View Slide

5. About collectd [2/3]
   ● “Output” options include:
   ○ Graphite
   ○ MongoDB
   ○ Riemann
   ○ RRDtool / RRDCacheD
   ● Various networking options
   

   View Slide

6. About collectd [3/3]
   ● Website: http://collectd.org/
   ● Twitter: @collectd
   ● Github: collectd
   ● Hackathon tomorrow at Michelberger Hotel.
   

   View Slide

7. Agenda
   ● About myself and collectd
   ● collectd data model
   ● The exec plugin
   ● The StatsD plugin
   ● Other possibilities
   

   View Slide

8. Data model [1/5]
   ● Four “data source types”
   ● Most important:
   ○ GAUGE
   Absolute value, e.g. temperature
   ○ DERIVE
   Counting values, e.g. number of requests served
   ● Special case:
   ○ COUNTER
   

   View Slide

9. Data model [2/5]
   Strict naming schema:
   ● Hostname
   ● Plugin
   ● Type
   Plugin and type can
   have an “instance”
   each.
   Examples:
   ● example.com
   ● df – root
   ● df_complex – free
   

   View Slide

10. Data model [3/5]
    ● The Type is defined in the types.db file
    (types.db(5) manual page).
    ● Specifies the “data source type”, i.e. GAUGE
    or DERIVE.
    ● Specifies minimum and maximum values.
    

    View Slide

11. Data model [4/5]
    ● Plugin instance is used to differentiate
    between “devices” the plugin can query.
    ○ For example interfaces (“eth0”), CPU number (“0”),
    mount point (“var-spool”).
    ● Type instance is used to to differentiate
    between multiple values of the same type.
    ○ For example, number of bytes used, number of
    bytes reserved, number of bytes free on a file
    system.
    

    View Slide

12. Data model [5/5]
    ● Notation:
    ● Examples:
    ○ example.com/df-root/df_complex-free
    ○ example.com/cpu-0/cpu-wait
    ○ example.com/memory/memory-used
    host/plugin-instance/type-instance
    host/plugin/type
    

    View Slide

13. Agenda
    ● About myself and collectd
    ● collectd data model
    ● The exec plugin
    ● The StatsD plugin
    ● Other possibilities
    

    View Slide

14. The exec plugin
    ● Executes arbitrary scripts / binaries
    ● Scripts print metrics to STDOUT
    ● Scripts print error messages to STDERR
    

    View Slide

15. The exec plugin
    collectd
    exec plugin
    Custom script
    STDOUT / STDERR
    

    View Slide

16. The exec plugin
    Configuration
    LoadPlugin "exec"
    
    Exec "gandalf:wizard" "/usr/lib/collectd/exec/magic.sh"
    # Exec "user:group" "/path/to/my/script"
    
    

    View Slide

17. The exec plugin
    First version
    #!/bin/sh
    level=`magic-ctl --level`
    echo "PUTVAL \"example.com/exec/gauge-magic\" N:${level}"
    

    View Slide

18. The exec plugin
    ● Performance?
    ● Problem: Forking is expensive.
    ● Solution: Let scripts loop and keep them
    running.
    

    View Slide

19. The exec plugin
    Second version
    #!/bin/sh
    while sleep 10; do
    level=`magic-ctl --level`
    echo "PUTVAL \"example.com/exec/gauge-magic\"" \
    "interval=10 N:${level}"
    done
    

    View Slide

20. The exec plugin
    ● Automation?
    ● Problem: Hard-coded hostname and
    interval must match collectd configuration.
    ● Solution: Use environment variables set by
    collectd: COLLECTD_HOSTNAME and
    COLLECTD_INTERVAL.
    

    View Slide

21. The exec plugin
    Final version
    #!/bin/sh
    interval=${COLLECTD_INTERVAL:-10}
    hostname=${COLLECTD_HOSTNAME:-localhost}
    while sleep ${interval}; do
    level=`magic-ctl --level`
    echo "PUTVAL \"${hostname}/exec/gauge-magic\" " \
    "interval=${interval} N:${level}"
    done
    

    View Slide

22. The exec plugin
    Caveat: Buffering
    ● collectd uses pipes.
    ● Many environments switch to (block)
    buffered I/O in this case.
    → Switch to line buffering or disable buffering.
    

    View Slide

23. Agenda
    ● About myself and collectd
    ● collectd data model
    ● The exec plugin
    ● The StatsD plugin
    ● Other possibilities
    

    View Slide

24. The StatsD plugin
    ● Opens a UDP socket and waits for
    asynchronous “events”.
    ● Implements the “StatsD” network protocol.
    ○ Wire compatible to Etsy's reference implementation.
    ○ Lots of tooling available.
    ● Very efficient.
    

    View Slide

25. The StatsD plugin
    Configuration
    LoadPlugin "statsd"
    
    DeleteCounters true
    DeleteSets false
    TimerPercentile 95
    
    

    View Slide

26. The StatsD plugin
    The StatsD protocol:
    ● Four metric types: Counter, Timer, Gauge
    and Set.
    ● Mapped to collectd “types” derive,
    latency, gauge and objects.
    

    View Slide

27. The StatsD plugin
    The StatsD protocol: Counter
    ● Increment “foo” by 3:
    ○ foo:3|c
    ● Mapped to:
    ○ /statsd/derive-foo
    

    View Slide

28. The StatsD plugin
    The StatsD protocol: Timer
    ● Operation “bar” took 3117 ms:
    ○ bar:3117|ms
    ● Mapped to:
    ○ /statsd/latency-bar-average
    ○ /statsd/latency-bar-percentile-95
    ○ …
    

    View Slide

29. The StatsD plugin
    The StatsD protocol: Gauge
    ● Set “qux” to 42:
    ○ qux:42|g
    ● Increase “qux” by 23:
    ○ qux:+23|g
    ● Mapped to:
    ○ /statsd/gauge-qux
    

    View Slide

30. The StatsD plugin
    The StatsD protocol: Set
    ● Add “bird” to set “animals”:
    ○ animals:bird|s
    ● Mapped to:
    ○ /statsd/objects-animals
    

    View Slide

31. Agenda
    ● About myself and collectd
    ● collectd data model
    ● The exec plugin
    ● The StatsD plugin
    ● Other possibilities
    

    View Slide

32. Other possibilities
    Network protocol:
    ● Send metrics in the “binary network
    protocol”.
    ● Various client libraries available.
    ● Virtually no coupling.
    

    View Slide

33. Other possibilities
    UnixSock plugin
    ● Listens on a UNIX domain socket and waits
    for commands, including PUTVAL.
    ● Client libraries available.
    ● Loose coupling.
    

    View Slide

34. Other possibilities
    Java, Perl and Python plugins
    ● Embed the VM / interpreters into the
    daemon (like mod_perl and friends).
    ● “Best of two worlds” (YMMV)
    ● Tight coupling
    

    View Slide

35. Other possibilities
    C based plugins:
    ● Play with the cool kids, drink the Kool-Aid
    and feel the power!
    

    View Slide

36. Thank you!
    Please ask some questions now! ;-)
    Hackathon, tomorrow (September 21) at
    Michelberger Hotel, starting at 9:00.
    

    View Slide