What's Brewing in Beats

1. ‹#› Monica Sarbu & Tudor Golubenco What’s Brewing in Beats
2. Beats are lightweight shippers that collect and ship all kinds of operational data to Elasticsearch 2
3. Beats are lightweight shippers that collect and ship all kinds of operational data to Elasticsearch 3
4. 4 Lightweight shipper • Small application • Install as agent on your servers • Written in Golang • No runtime dependencies • Single purpose https://www.flickr.com/photos/8barbikes/17256970434/
5. Beats are lightweight shippers that collect and ship all kinds of operational data to Elasticsearch 5
6. Examples of operational data 6 wire data system stats logs Packetbeat Topbeat Filebeat Winlogbeat
7. Captures insights from network packets 7 Packetbeat
8. Sniffing the network traffic 8 • Copy traffic at OS or hardware level • Is completely passive • ZERO latency overhead • Not in the request/response path, cannot break your application Client Server sniff sniff
9. 9 Sniffing use cases • Security • Intrusion Detection Systems • Troubleshooting network issues • Troubleshooting applications • Performance analysis
10. Packetbeat: Real-time application monitoring 10 1 2 3 4 capture network traffic decodes network traffic correlates request with response in transactions extract measurements like response time, status 5 group meta info in json objects to send to Elasticsearch It does all of these in real-time directly on the target servers.
11. Packetbeat: Available decoders 11 HTTP MySQL PostgreSQL MongoDB Memcache ICMP + Add your own Thrift-RPC DNS Redis AMQP
12. Check out the demo on Elastic{ON} traffic! 12 At demo booth no. 2, in the lunch table area
13. Like the Unix top command but sends the output periodically to Elasticsearch. Also works on Windows. 13 Topbeat
14. Topbeat: Exported data 14 • system load • total CPU usage • CPU usage per core • Swap, memory usage System wide • state • name • command line • pid • CPU usage • memory usage Per process • available disks • used, free space • mounted points Disk usage
15. Forwards log lines to Elasticsearch 15 Filebeat
16. 16 Filebeat: Never lose a log line line line line line line read pointer Filebeat Logstash Back-pressure sensitive protocol Yo Filebeat, slow it down a bit, pls K buddy line The original log lines act like a queue
17. Filebeat: Parse logs with Logstash 17 • Filebeat sends out unparsed log lines • Use filters from Logstash to parse the log lines • Flexible, with conditionals & custom filters • Forward data to other systems using the Logstash output plugins Filebeat Elasticsearch Logstash Other systems
18. Filebeat: Parse logs with Ingest Node 18 • Upcoming in 5.0 • Filebeat sends out unparsed log lines directly to Elasticsearch • Use Ingest Node processors to parse the log lines • Easier to setup Filebeat Elasticsearch Don’t miss the Ingest Node presentation tomorrow at 2:15 p.m.
19. ‹#› Multiline 19 multiline: # Sticks together all lines # that don’t start with a [ pattern: ^\[ negate: true match: after Filebeat extra power • Sticks together related log lines in a single event • For all those long exceptions • Can also be done by Logstash, but it’s sometimes easier to configure the patterns closer to the source
20. Forwards Windows Event logs to Elasticsearch 20 Winlogbeat
21. 21 Winlogbeat overview • Supports Windows versions starting with XP • It remembers how far it read, so it never loses log events • Winlogbeat sends out unparsed Windows event logs • Use Ingest Node or Logstash to parse the Windows event logs
22. ‹#› DEMO!
23. ‹#› Community Beats
24. 24 1 Apachebeat 2 Dockerbeat 3 Elasticbeat 4 Execbeat 5 Factbeat 6 Hsbeat 14 COMMUNITY BEATS Sending all sorts of data to Logstash and Elasticsearch 7 Httpbeat 8 Nagioscheckbeat 9 Nginxbeat 10 Phpfpmbeat 11 Pingbeat 13 Unifiedbeat 12 Redisbeat 14 Uwsgibeat
25. Community Beats: libbeat 25 libbeat Community Beats Elastic Beats • Golang library • Outputs for Elasticsearch and Logstash • At least once guarantees • Encryption & authentication • Common code for configuration files, logging, daemonizing, CLI flags, etc. Elasticsearch Logstash
26. ‹#› 26 input: # Loop every 5 seconds period: 5 # Use raw sockets for ping # Requires root! privileged: true # Whether to perform IPv4/v6 pings useipv4: true useipv6: false # List targets under the tag # you want assigned to targets: # tag: google google: - google.com.au - google.com You know, for pings • Sends ICMP (v4 or v6) pings periodically to a list of hosts • Can send also UDP pings (no root required) • Resolves DNS • Records RTT Pingbeat
27. ‹#› 27 Unifiedbeat Parse IDS logs • Reads Unified2 binary files • Generated by Snort/Suricata IDS • Alternative for Barnyard2 + MySQL • Indexes also the rule/signature that have triggered the alert • Adds GeoIP information unifiedbeat: # Configure Snort Rules/Signatures # support: rules: paths: - “sample_data/rules/*.rules” # List of sensors sensor: - paths: - “sample_data/snort.log” fields: sensor_hostname: nucy sensor_interface: em1 sensor_type: snort
28. Kibana dashboard for Unifiedbeat 28
29. ‹#› 29 Docker Monitoring • Uses the Docker API • Per container stats about: • CPU • Memory • Disk • Network Dockerbeat input: # In seconds, defines how often to # read server statistics period: 5 # Define the docker socket path # By default, this will get the # unix:///var/run/docker.sock socket: 29
30. ‹#› 30 Nagioscheckbeat Run Nagios checks • Can execute any Nagios plugin • Execution period configurable per check • Sends alerts (Warning/Critical) to Elasticsearch • Sends performance data to Elasticsearch input: checks: - name: "disks" cmd: "plugins/check_disk" args: "-w 80 -c 90 -x /dev" period: "1h" - name: "load" cmd: "plugins/check_load" args: "-w 5 -c 10" period: "1m"
31. ‹#› How can we make it even easier to create a new Beat? 31
32. Beat generator Quickly get started with the development of a new Beat 32 $ pip install cookiecutter $ cookiecutter https://github.com/elastic/beat-generator.git project_name [Examplebeat]: Mybeat github_name [your-github-name]: tsg beat [examplebeat]: mybeat beat_path [github.com/your-github-name]: github.com/tsg full_name [Firstname Lastname]: Tudor Golubenco
33. 33 Beats Packer • Cross-compiles to all our supported platforms • Produces RPMs, DEBs, • Same tools that we use to build the official Elastic Beats • Can be executed from Travis CI
34. ‹#› Upcoming in 5.0
35. 35 1 Apachebeat 2 Dockerbeat 3 Elasticbeat 4 Execbeat 5 Factbeat 6 Hsbeat 7 Httpbeat 8 Nagioscheckbeat 9 Nginxbeat 10 Phpfpmbeat 11 Pingbeat 12 Redisbeat 13 Unifiedbeat 14 Uwsgibeat
36. 36 1 Apachebeat 2 Dockerbeat 3 Elasticbeat 4 Execbeat 5 Factbeat 6 Hsbeat 7 Httpbeat 8 Nagioscheckbeat 9 Nginxbeat 10 Phpfpmbeat 11 Pingbeat 14 Uwsgibeat 12 Redisbeat 13 Unifiedbeat
37. ‹#› Can they share the same infrastructure? 37
38. Provides a common infrastructure for all “metrics” related Beats. Upcoming in 5.0.0-alpha1. 38 Metricbeat
39. Metricbeat: Collecting metrics from other systems 39 1 2 3 Periodically polls monitoring APIs of various services Groups performance data into documents Ships them to Logstash / Elasticsearch
40. Metricbeat A module for each monitored system 40 libbeat Metricbeat Apache MySQL Redis Golang +
41. 41 Metricbeat: It is also a library! • Use the Metricbeat infrastructure, to create a standalone Beat • One can create a Beat with a single Metricbeat module libbeat Metricbeat df module github.com/ruflin/df2beat
42. Metricbeat module vs standalone Beat 42 • Contributed via PR to the elastic/beats Github repository • Officially supported • Supports common systems • Docker based integration tests Metricbeat module • In a separate Github repository • Supported by the community • Supports specialized systems • Optional Docker based integration tests Standalone Beat
43. ‹#› How can we make it easier to deploy the Beats? 43
44. Intended deployment models 44 Topbeat Elasticsearch Topbeat & Packetbeat: Send directly to Elasticsearch Filebeat Logstash Elasticsearch Filebeat & Winlogbeat: Use Logstash for parsing
45. Adding a queuing system 45 Filebeat Logstash Logstash Elasticsearch Redis / Kafka • You need two sets of Logstash instances: • One Logstash to send the data to Redis/Kafka • One Logstash to enrich and transport data securely to Elasticsearch
46. 46 Kafka and Redis output • Have Redis output support in libbeat • Add Kafka output in Beats 5.0.0-alpha1 Filebeat Logstash Elasticsearch Redis / Kafka
47. ‹#› Generic filtering 47 filter: # In Packetbeat, don’t store the # 200 OK transactions - drop_event: equals: “http.response.code”: 200 # In Topbeat, drop selected # fields - drop_fields: fields: - cpu.user - proc.cpu.total • Flexibly reduce the amount of data sent of the wire and stored • Filter fields or events with simple conditions • Part of libbeat
48. ‹#› 48 Packetbeat flows { "@timestamp": "2016-02-11T11:12:09.718Z", "count": 1, "final": true, "icmp_id": 5, "ip4_dest": "10.0.0.2", "ip4_source": "10.0.0.1", "last_time": "2016-02-11T11:12:09.416Z", "mac_dest": "00:00:00:00:00:02", "mac_source": "00:00:00:00:00:01", "start_time": "2016-02-11T11:12:09.416Z", "stats_dest": { "net_bytes_total": 50, "net_packets_total": 1 }, "stats_source": { "net_bytes_total": 50, "net_packets_total": 1 }, "type": "flow", "vlan": 10 } • Look into data for which we don’t understand the application layer protocol • TLS • Protocols we don’t yet support • Get data about IP / TCP / UDP layers • number of packets • retransmissions • inter-arrival time
49. ‹#› Q&A Find us on: • github.com/elastic/beats • discuss.elastic.co • @elastic #elasticbeats • #beats on freenode Or Here. In Real Life!
50. ‹#› Please attribute Elastic with a link to elastic.co Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 50