Performance Monitoring with ELK & collectd

Use collectd to capture performance metrics and send them to the ELK stack: Elasticsearch, Logstash & Kibana.

Transcript

1. Performance Monitoring with the ELK Stack: collectd
2. { } CC-BY-ND 4.0 Why would I want to do performance monitoring with ELK? 2
3. { } CC-BY-ND 4.0 Performance Metrics • CPU • Disk • Memory • Network • More! 3
4. { } CC-BY-ND 4.0 Introducing: collectd https://collectd.org 4
5. { } CC-BY-ND 4.0 For Windows users... • SSC Serv – Commercial product – Uses collectd protocol – Disk, df, CPU, Interface, Terminal Services – Can monitor any performance counter available via the Performance Data Handles interface. – http://ssc-serv.com 5
6. { } CC-BY-ND 4.0 Log data... 6 Logs
7. { } CC-BY-ND 4.0 Performance metrics... 7 Metrics
8. { } CC-BY-ND 4.0 Correlation! 8 Logs Metrics
9. { } CC-BY-ND 4.0 Get the whole picture! 9
10. { } CC-BY-ND 4.0 Configuring Logstash... input { udp { host => "x.x.x.x" port => 25826 buffer_size => 1452 type => "collectd" codec => collectd { } } } 10
11. { } CC-BY-ND 4.0 Configuring Logstash... • Authentication & Security • NaN handling • Interval pruning • typesdb 11
12. { } CC-BY-ND 4.0 Configuring collectd... Hostname "host.example.com" LoadPlugin interface LoadPlugin load LoadPlugin memory LoadPlugin network <Plugin interface> Interface "eth0" IgnoreSelected false </Plugin> <Plugin network> <Server "10.0.0.1" "25826"> </Server> </Plugin> 12
13. { } CC-BY-ND 4.0 Configuring collectd... • Intervals are configurable – Global – Per Plugin 13
14. { } CC-BY-ND 4.0 Plugins • df, disk – Disk usage statistics. • load – The 1m, 5m, and 15m load averages • memory – free, buffered, cached, used, etc. • interface – Per-interface network usage/traffic statistics. 14
15. { } CC-BY-ND 4.0 Plugins • ConnTrack – Tracks the number of entries in Linux's connection tracking table. • ContextSwitch – Collects the number of context switches done by the operating system. 15
16. { } CC-BY-ND 4.0 Plugins • DBI/PostgreSQL/Oracle – Returns values from queries. • Entropy – Collects the available entropy on a system 16
17. { } CC-BY-ND 4.0 Plugins • memcached – Collects the number of connections and requests handled by the daemon, the CPU resources consumed, number of items cached, number of threads, and bytes sent and received. • MySQL – Connects to a MySQL db, issues a SHOW STATUS command, and returns many of the variables. 17
18. { } CC-BY-ND 4.0 Plugins • Swap – Collects the amount of memory currently written onto hard disk (or whatever the system calls “swap”) • TCPConns – Counts the number of TCP connections to or from a specified port. Results include each state: LISTEN, ESTABLISHED, CLOSE_WAIT, etc. 18
19. { } CC-BY-ND 4.0 BIND (9.5.0+) Global statistics ▪ OpCodes ▪ Query types (A, MX, AAAA, …) ▪ Overall server statistics (#Queries, #Responses, …) ▪ Zone maintenance statistics (#Notifications, #Updates, …) ▪ Resolver statistics (usually empty) ▪ Memory statistics Per-view statistics ▪ Query types ▪ Resolver statistics (#Queries, #Responses, #NXDOMAIN, …) ▪ RR-set cache statistics (#entries by type) Per-zone statistics ▪ Overall statistics (Success, #NXRRSET, …) 19
20. { } CC-BY-ND 4.0 IP Tables • Per-rule byte and packet counters, selected by: – Position (e.g. “the fourth rule in the ‘INPUT’ queue in the ‘filter’ table”) – Comment (using the “COMMENT” match). • Low overhead – Uses libiptc. Communicates with the kernel directly. 20
21. { } CC-BY-ND 4.0 SNMP • Uses Net-SNMP • Use collectd to collect stats from: – Switches – Routers – UPS – Rack monitoring systems, – and more! 21
22. { } CC-BY-ND 4.0 Custom Plugins & Extensions • C • Perl • Python • Exec • Unix-sockets • Java • Java MBean support, via jcollectd 22
23. { } CC-BY-ND 4.0 Logstash output { "host":"host.example.com", "@timestamp":"2015-03-06T12:26:43.790-07:00", "@version":"1", "type":"collectd", "plugin":"memory", "collectd_type":"memory", "type_instance":"used", "value":8517087232, } 23
24. { } CC-BY-ND 4.0 Logstash output { "host":"host.example.com", "@timestamp":"2015-03-06T12:26:43.790-07:00", "@version":"1", "type":"collectd", "plugin":"memory", "collectd_type":"memory", "type_instance":"used", "value":8517087232, } 24
25. { } CC-BY-ND 4.0 Logstash output { "host":"host.example.com", "@timestamp":"2015-03-06T12:38:45.789-07:00", "@version":"1", "type":"collectd", "plugin":"interface", "plugin_instance":"eth0", "collectd_type":"if_packets", "rx":0, "tx":0 } 25
26. { } CC-BY-ND 4.0 Logstash output { "host":"host.example.com", "@timestamp":"2015-03-06T12:38:45.789-07:00", "@version":"1", "type":"collectd", "plugin":"interface", "plugin_instance":"eth0", "collectd_type":"if_packets", "rx":0, "tx":0 } 26
27. { } CC-BY-ND 4.0 27 What now?