Performance Monitoring with the ELK Stack: collectd 

{ } CC-BY-ND 4.0 Why would I want to do performance monitoring with ELK? 2 

{ } CC-BY-ND 4.0 Performance Metrics • CPU • Disk • Memory • Network • More! 3 

{ } CC-BY-ND 4.0 Introducing: collectd https://collectd.org 4 

{ } CC-BY-ND 4.0 For Windows users... • SSC Serv – Commercial product – Uses collectd protocol – Disk, df, CPU, Interface, Terminal Services – Can monitor any performance counter available via the Performance Data Handles interface. – http://ssc-serv.com 5 

{ } CC-BY-ND 4.0 Log data... 6 Logs 

{ } CC-BY-ND 4.0 Performance metrics... 7 Metrics 

{ } CC-BY-ND 4.0 Correlation! 8 Logs Metrics 

{ } CC-BY-ND 4.0 Get the whole picture! 9 

{ } CC-BY-ND 4.0 Configuring Logstash... input { udp { host => "x.x.x.x" port => 25826 buffer_size => 1452 type => "collectd" codec => collectd { } } } 10 

{ } CC-BY-ND 4.0 Configuring Logstash... • Authentication & Security • NaN handling • Interval pruning • typesdb 11 

{ } CC-BY-ND 4.0 Configuring collectd... Hostname "host.example.com" LoadPlugin interface LoadPlugin load LoadPlugin memory LoadPlugin network Interface "eth0" IgnoreSelected false 12 

{ } CC-BY-ND 4.0 Configuring collectd... • Intervals are configurable – Global – Per Plugin 13 

{ } CC-BY-ND 4.0 Plugins • df, disk – Disk usage statistics. • load – The 1m, 5m, and 15m load averages • memory – free, buffered, cached, used, etc. • interface – Per-interface network usage/traffic statistics. 14 

{ } CC-BY-ND 4.0 Plugins • ConnTrack – Tracks the number of entries in Linux's connection tracking table. • ContextSwitch – Collects the number of context switches done by the operating system. 15 

{ } CC-BY-ND 4.0 Plugins • DBI/PostgreSQL/Oracle – Returns values from queries. • Entropy – Collects the available entropy on a system 16 

{ } CC-BY-ND 4.0 Plugins • memcached – Collects the number of connections and requests handled by the daemon, the CPU resources consumed, number of items cached, number of threads, and bytes sent and received. • MySQL – Connects to a MySQL db, issues a SHOW STATUS command, and returns many of the variables. 17 

{ } CC-BY-ND 4.0 Plugins • Swap – Collects the amount of memory currently written onto hard disk (or whatever the system calls “swap”) • TCPConns – Counts the number of TCP connections to or from a specified port. Results include each state: LISTEN, ESTABLISHED, CLOSE_WAIT, etc. 18 

{ } CC-BY-ND 4.0 BIND (9.5.0+) Global statistics ▪ OpCodes ▪ Query types (A, MX, AAAA, …) ▪ Overall server statistics (#Queries, #Responses, …) ▪ Zone maintenance statistics (#Notifications, #Updates, …) ▪ Resolver statistics (usually empty) ▪ Memory statistics Per-view statistics ▪ Query types ▪ Resolver statistics (#Queries, #Responses, #NXDOMAIN, …) ▪ RR-set cache statistics (#entries by type) Per-zone statistics ▪ Overall statistics (Success, #NXRRSET, …) 19 

{ } CC-BY-ND 4.0 IP Tables • Per-rule byte and packet counters, selected by: – Position (e.g. “the fourth rule in the ‘INPUT’ queue in the ‘filter’ table”) – Comment (using the “COMMENT” match). • Low overhead – Uses libiptc. Communicates with the kernel directly. 20 

{ } CC-BY-ND 4.0 SNMP • Uses Net-SNMP • Use collectd to collect stats from: – Switches – Routers – UPS – Rack monitoring systems, – and more! 21 

{ } CC-BY-ND 4.0 Custom Plugins & Extensions • C • Perl • Python • Exec • Unix-sockets • Java • Java MBean support, via jcollectd 22 

{ } CC-BY-ND 4.0 Logstash output { "host":"host.example.com", "@timestamp":"2015-03-06T12:26:43.790-07:00", "@version":"1", "type":"collectd", "plugin":"memory", "collectd_type":"memory", "type_instance":"used", "value":8517087232, } 23 

{ } CC-BY-ND 4.0 Logstash output { "host":"host.example.com", "@timestamp":"2015-03-06T12:26:43.790-07:00", "@version":"1", "type":"collectd", "plugin":"memory", "collectd_type":"memory", "type_instance":"used", "value":8517087232, } 24 

{ } CC-BY-ND 4.0 Logstash output { "host":"host.example.com", "@timestamp":"2015-03-06T12:38:45.789-07:00", "@version":"1", "type":"collectd", "plugin":"interface", "plugin_instance":"eth0", "collectd_type":"if_packets", "rx":0, "tx":0 } 25 

{ } CC-BY-ND 4.0 Logstash output { "host":"host.example.com", "@timestamp":"2015-03-06T12:38:45.789-07:00", "@version":"1", "type":"collectd", "plugin":"interface", "plugin_instance":"eth0", "collectd_type":"if_packets", "rx":0, "tx":0 } 26 

{ } CC-BY-ND 4.0 27 What now?