And the beats go on - Riviera Dev 2016

Talk given at Riviera Dev 2016: http://rivieradev.fr/session/58

Transcript

1. ‹#› And the beats go on! David Pilato Developer | Evangelist @dadoonet
3. Elastic Subscriptions: Product, Experience, & Support 3 Open Source Elasticsearch Kibana Logstash Beats Elastic Stack Expertise and Support Elasticsearch as a Service (Found) Development Production Plugins Security (Shield) Alerting (Watcher) Monitoring (Marvel) Technical Guidance • Architecture (hardware/software) • Cluster management (tuning) • Index / shard design • Query optimization • Integration with other products • Backup and HA strategy • Dev to production migration / upgrades • Best practices Troubleshooting & Support • Dedicated, hands-on SLA-based support • Analysis of internal logs • Proactively monitoring of clusters • Escalation to engineering team
5. Beats are lightweight shippers that collect and ship all kinds of operational data to Elasticsearch
6. Beats are lightweight shippers that collect and ship all kinds of operational data to Elasticsearch
7. Beats are lightweight shippers that collect and ship all kinds of operational data to Elasticsearch
8. Examples of operational data 8 wire data system stats logs Packetbeat Metricbeat Filebeat Winlogbeat
9. Captures insights from network packets 9 Packetbeat
10. Sniffing the network traffic 10 Client Server sniff sniff • Copy traffic at OS or hardware level • Is completely passive • ZERO latency overhead • Not in the request/response path, cannot break your application
11. Packetbeat: Available decoders 11 HTTP MySQL PostgreSQL MongoDB Memcache ICMP + Add your own Thrift-RPC DNS Redis AMQP
12. Like the Unix top command but sends the output periodically to Elasticsearch. Also works on Windows. 12 Metricsbeat
13. Topbeat: Exported data 13 • system load • total CPU usage • CPU usage per core • Swap, memory usage System wide • state • name • command line • pid • CPU usage • memory usage Per process • available disks • used, free space • mounted points Disk usage
14. Forwards log lines to Elasticsearch 14 Filebeat
15. Filebeat: Never lose a log line 15 line line line line line read pointer Filebeat Back-pressure sensitive protocol Yo Filebeat, slow it down a bit, pls K buddy line The original log lines act like a queue
16. Filebeat: Parse logs with Logstash Parse logs with Logstash 16 • Filebeat sends out unparsed log lines • Use filters from Logstash to parse the log lines • Flexible, with conditionals & custom filters • Forward data to other systems using the Logstash output plugins Filebeat Other systems
17. Filebeat: Parse logs with Ingest Node Parse logs with Ingest node in Elasticsearch 17 • Filebeat sends out unparsed log lines directly to Elasticsearch • Use Ingest Node processors (grok, geoip…) to parse the log lines • Easier to setup Filebeat 5.0
18. Forwards Windows Event logs to Elasticsearch 18 Winlogbeat
19. Beats Platform 19 Explore & Visualize Search & Analyze Enrich & Transport Optional libbeat {Community} Beats Elastic Beats Collect, Parse & Ship
20. Architecture Overview - libbeat 20 {Community}Beat libbeat Outputs * Logstash Elasticsearch Config Management Debugging Logging * Syslog File Cmd Line Handling Filtering Testing Testing Environment System Test Framework
21. Produces RPMs, DEBs, … 21 Beats Packer https://github.com/elastic/beats-packer
24. ‹#› https://github.com/dadoonet/soundbeat metricbeat, packetbeat and soundbeat
25. ‹#› https://github.com/dadoonet/soundbeat thanks!