What's Brewing in Beats?

Beats are a family of lightweight shippers that send data from edge machines to Elasticsearch. They started with a single Beat – Packetbeat – for network data, and have since expanded to four additional Elastic Beats for log files, metrics, windows event logs, and availability monitoring, and more than 30 community-created Beats for all kinds of operational data.

In this session, Monica and Tudor will walk you through the latest Beats and their features. You will learn about the new Filebeat modules, which simplify the collection and parsing of common log files down to a single command. You will also see Heartbeat in action, our newest Beat responsible for uptime monitoring.

Tudor Golubenco l Beats Creator & Tech Lead l Elastic
Monica Sarbu l Beats Creator & Team Lead l Elastic

Transcript

1. What’s brewin’ in Beats Monica Sarbu @monicasarbu Tudor Golubenco @tudor_g
2. 2 The road so far
3. Recap of our road so far 3 Packetbeat Network data
4. Recap of our road so far 4 Packetbeat Network data libbeat Beats library
5. Recap of our road so far 5 Packetbeat Network data libbeat Beats library Topbeat System statistics
6. Recap of our road so far 6 Packetbeat Network data libbeat Beats library Filebeat Log files Topbeat System statistics
7. Recap of our road so far 7 Packetbeat Network data libbeat Beats library Filebeat Log files Winlogbeat Windows Event Logs Topbeat System statistics
8. Recap of our road so far 8 Packetbeat Network data libbeat Beats library Filebeat Log files Winlogbeat Windows Event Logs Topbeat System statistics +40 community Beats
9. Recap of our road so far 9 Packetbeat Network data libbeat Beats library Filebeat Log files Winlogbeat Windows Event Logs +40 community Beats Metricbeat Metrics
10. Recap of our road so far 10 Packetbeat Network data libbeat Beats library Filebeat Log files Winlogbeat Windows Event Logs Heartbeat Uptime monitoring +40 community Beats Metricbeat Metrics
11. What have we learned
12. Out-of-the-box experience For metrics and network traffic 12
13. What about logs? 13 Configure Filebeat paths & multiline Tune the Elasticsearch template Write Logstash Grok patterns Create Kibana dashboards
14. What about logs? 14 Configure Filebeat paths & multiline Tune the Elasticsearch template Write Logstash Grok patterns Create Kibana dashboards very powerful, but complex ☹
15. Grok patterns in Ingest Node 15 Create Kibana dashboards Write Ingest Grok patterns and Elasticsearch template I N G E S T Configure Filebeat paths & multiline
16. Grok patterns in Ingest Node 16 Create Kibana dashboards Write Ingest Grok patterns and Elasticsearch template I N G E S T Configure Filebeat paths & multiline duplicated effort ☹
17. Filebeat modules 17 Filebeat configuration Ingest pipelines Elasticsearch template Kibana dashboards
18. Visualize your Apache logs in minutes 18 $ filebeat -setup -modules=apache2
19. … and more to come 19 Filebeat configuration Ingest pipelines Elasticsearch template Kibana dashboards ML jobs Watcher alerts
20. Modules … modules everywhere 20 MySQL Packetbeat MySQL Filebeat MySQL Metricbeat queries slow queries stats
21. Demo
22. Recently in Metricbeat
23. Metricbeat modules 23 MySQL Memcache PHP-FPM CEPH Zoo keeper Golang Docker Apache Kafka HAProxy System Redis Couchbase NGINX Postgres Prometheus Jolokia
24. What about monitoring your apps?
25. host Instrument using Prometheus 25 Your app push metricbeat pull
26. host JMX monitoring with Jolokia 26 metricbeat pull push Your app JVM
27. host Profiling Go programs 27 Your app push metricbeat pull
28. What if your application is running in Docker?
29. • cgroup metrics • Docker API Docker metrics 29
30. Run Metricbeat in a container 30 App1 App2 App3 Host
31. Configuration reloading 31 /etc/metricbeat/conf.d/ metricbeat App1 App2 App3
32. Monitor network connections You can now see who is talking with whom and how much 32
33. What if your application is running in the cloud?
34. Add cloud metadata add_cloud_metadata Beat processor 34
35. I just called … to say … HTTP GET Heartbeat
36. Heartbeat - Ping all the things 36 host Your app OS TCP/TLS connect ICMP ping HTTP/S request
37. • Round Trip Times: • resolve • icmp • tcp_connect • socks5_connect • tls_handshake • http Heartbeat metrics 37
38. Many to many 38
39. Heartbeat - Ping all IPs behind a DNS name 39 h ELB
40. Coming up next
41. Because the Beats also need a bit of monitoring love Central monitoring for Beats 41 production monitoring forwards monitoring data Specialized monitoring UI
42. Make upgrades easier 42 filebeat-2017.01.14 Filebeat 5.x: filebeat-6.0.0-2017.01.14 Filebeat 6.0:
43. Improve Metricbeat dashboards
44. Future themes
45. 45 Enable more security use cases Photo credit: https://www.flickr.com/photos/charbelakhras/2535524197
46. 46 MOAR MODULES
47. 47 Add a touch of magic
48. 48 De-duplicate
49. Central configuration management 49
50. More Beats topics on Wednesday and Thursday At Elastic{ON}
51. 51 Andrew Monitoring Docker with Metricbeat Wednesday, 4:40 pm Spotlight Theater Be there to learn how to monitor your containers with Beats
52. 52 Nicolas & Steffen Ship Your Own Data: Tailoring Beats to Your Use Case Thursday, 11am Stage B Be there to learn how to extend Beats
53. 53 Chris BoF: Kibana Visualizations Wednesday, 1:15 pm Spotlight Theater Demo Time series visualization builder in the Birds of a Feather session
54. 54 Thanks! Meet us at the AMA booth Sept 2016