And The Beats Go On! JUG Summer Camp 2016

And the beat goes on - The Whispers

Venez découvrir les nouveaux Data Shippers pour Elasticsearch:

Packetbeat : décortiquez les protocoles réseau
Topbeat: récoltez vos métriques
Filebeat: analysez vos logs en temps réel ou transmettez les vers logstash pour enrichissement
Et comment contribuer au mix et ajouter vos propres beats avec Libbeat.

Talk given at JUG Summer Camp in La Rochelle.

Transcript

1. ‹#› And the beats go on! David Pilato Developer | Evangelist @dadoonet
4. 4 The only Elasticsearch as a Service offering powered by the creators of the Elastic Stack • Always runs on the latest software • One-click to scale/upgrade with no downtime • Free Kibana and backups every 30 minutes • Dedicated, SLA-based support • Easily add X-Pack features: security (Shield), alerting (Watcher), and monitoring (Marvel) • Pricing starts at $45 a month
5. Elastic Subscription Packages 5 Elastic Stack BASIC GOLD PLATINUM Elasticsearch,Kibana, Logstash, Beats ✓ ✓ ✓ X-Pack Marvel (Monitoring & Management) ✓ ✓ Multicluster Support >7 days data retention ✓ Multicluster Support >7 days data retention Shield (Security) ✓ ✓ Field/document security Custom realms Watcher (Alerting) ✓ ✓ Reporting ✓ ✓ Graph Analytics & Visualization ✓ Support Support coverage and response times provided by Elastic Support Engineers Business Hours 4 hour L1 response times 24/7/365 1 hour L1 response times Emergency patches
6. Elastic’s Unique Subscription Value 6 Extending beyond traditional support, Elastic subscriptions include: • Support for the Elastic Stack: Elasticsearch, Kibana, Logstash, Beats and ES-Hadoop • Use and support of X-Pack features like security (Shield), alerting (Watcher), monitoring (Marvel), Reporting, and Graph • Expertise and best practices advice by Elastic’s support engineering team • Flexible subscription packages for users of both on-premise and the Elastic Cloud Support Expertise Includes: Architecture / Index / Shard Design Cluster Management (Tuning) Query Performance Optimization Dev to Production Migration & Upgrades Best Practices (Elastic Stack, X-Pack)
8. Beats are lightweight shippers that collect and ship all kinds of operational data to Elasticsearch
9. Beats are lightweight shippers that collect and ship all kinds of operational data to Elasticsearch
10. Beats are lightweight shippers that collect and ship all kinds of operational data to Elasticsearch
11. Examples of operational data 11 wire data system stats logs Packetbeat Metricbeat Filebeat Winlogbeat
12. Captures insights from network packets 12 Packetbeat
13. Sniffing the network traffic 13 Client Server sniff sniff • Copy traffic at OS or hardware level • Is completely passive • ZERO latency overhead • Not in the request/response path, cannot break your application
14. Sniffing use cases • Security • Intrusion Detection Systems • Troubleshooting network issues • Troubleshooting applications • Performance analysis 14
15. Packetbeat: Real-time application monitoring 15 1 2 3 4 capture network traffic decodes network traffic correlates request with response in transactions extract measurements like response time, status 5 group meta info in json objects to send to Elasticsearch It does all of these in real-time directly on the target servers.
16. Packetbeat: Available decoders 16 HTTP MySQL PostgreSQL MongoDB Memcache ICMP + Add your own Thrift-RPC DNS Redis AMQP
17. Like the Unix top command but sends the output periodically to Elasticsearch. Also works on Windows. 17 Metricsbeat
18. Topbeat: Exported data 18 • system load • total CPU usage • CPU usage per core • Swap, memory usage System wide • state • name • command line • pid • CPU usage • memory usage Per process • available disks • used, free space • mounted points Disk usage
19. Forwards log lines to Elasticsearch 19 Filebeat
20. Filebeat: Never lose a log line 20 line line line line line read pointer Filebeat Back-pressure sensitive protocol Yo Filebeat, slow it down a bit, pls K buddy line The original log lines act like a queue
21. Filebeat: Parse logs with Logstash Parse logs with Logstash 21 • Filebeat sends out unparsed log lines • Use filters from Logstash to parse the log lines • Flexible, with conditionals & custom filters • Forward data to other systems using the Logstash output plugins Filebeat Other systems
22. Filebeat: Parse logs with Ingest Node Parse logs with Ingest node in Elasticsearch 22 • Filebeat sends out unparsed log lines directly to Elasticsearch • Use Ingest Node processors (grok, geoip…) to parse the log lines • Easier to setup Filebeat 5.0
23. 23 multiline: # Sticks together all lines # that don’t start with a [ pattern: ^\[ negate: true match: after Filebeat extra power
24. Forwards Windows Event logs to Elasticsearch 24 Winlogbeat
25. Beats Platform 25 Explore & Visualize Search & Analyze Enrich & Transport Optional libbeat {Community} Beats Elastic Beats Collect, Parse & Ship
26. Architecture Overview - libbeat 26 {Community}Beat libbeat Outputs * Logstash Elasticsearch Config Management Debugging Logging * Syslog File Cmd Line Handling Filtering Testing Testing Environment System Test Framework
27. Produces RPMs, DEBs, … 27 Beats Packer https://github.com/elastic/beats-packer
30. ‹#› https://github.com/dadoonet/soundbeat metricbeat, packetbeat and soundbeat
31. ‹#› https://github.com/dadoonet/soundbeat thanks!