Upgrade to Pro — share decks privately, control downloads, hide ads and more …

What's Brewing in Beats

Elastic Co
February 17, 2016

What's Brewing in Beats

Elastic Co

February 17, 2016
Tweet

More Decks by Elastic Co

Other Decks in Technology

Transcript

  1. 4 Lightweight shipper • Small application • Install as agent

    on your servers • Written in Golang • No runtime dependencies • Single purpose https://www.flickr.com/photos/8barbikes/17256970434/
  2. Examples of operational data 6 wire data system stats logs

    Packetbeat Topbeat Filebeat Winlogbeat
  3. Sniffing the network traffic 8 • Copy traffic at OS

    or hardware level • Is completely passive • ZERO latency overhead • Not in the request/response path, cannot break your application Client Server sniff sniff
  4. 9 Sniffing use cases • Security • Intrusion Detection Systems

    • Troubleshooting network issues • Troubleshooting applications • Performance analysis
  5. Packetbeat: Real-time application monitoring 10 1 2 3 4 capture

    network traffic decodes network traffic correlates request with response in transactions extract measurements like response time, status 5 group meta info in json objects to send to Elasticsearch It does all of these in real-time directly on the target servers.
  6. Check out the demo on Elastic{ON} traffic! 12 At demo

    booth no. 2, in the lunch table area
  7. Like the Unix top command but sends the output periodically

    to Elasticsearch. Also works on Windows. 13 Topbeat
  8. Topbeat: Exported data 14 • system load • total CPU

    usage • CPU usage per core • Swap, memory usage System wide • state • name • command line • pid • CPU usage • memory usage Per process • available disks • used, free space • mounted points Disk usage
  9. 16 Filebeat: Never lose a log line line line line

    line line read pointer Filebeat Logstash Back-pressure sensitive protocol Yo Filebeat, slow it down a bit, pls K buddy line The original log lines act like a queue
  10. Filebeat: Parse logs with Logstash 17 • Filebeat sends out

    unparsed log lines • Use filters from Logstash to parse the log lines • Flexible, with conditionals & custom filters • Forward data to other systems using the Logstash output plugins Filebeat Elasticsearch Logstash Other systems
  11. Filebeat: Parse logs with Ingest Node 18 • Upcoming in

    5.0 • Filebeat sends out unparsed log lines directly to Elasticsearch • Use Ingest Node processors to parse the log lines • Easier to setup Filebeat Elasticsearch Don’t miss the Ingest Node presentation tomorrow at 2:15 p.m.
  12. ‹#› Multiline 19 multiline: # Sticks together all lines #

    that don’t start with a [ pattern: ^\[ negate: true match: after Filebeat extra power • Sticks together related log lines in a single event • For all those long exceptions • Can also be done by Logstash, but it’s sometimes easier to configure the patterns closer to the source
  13. 21 Winlogbeat overview • Supports Windows versions starting with XP

    • It remembers how far it read, so it never loses log events • Winlogbeat sends out unparsed Windows event logs • Use Ingest Node or Logstash to parse the Windows event logs
  14. 24 1 Apachebeat 2 Dockerbeat 3 Elasticbeat 4 Execbeat 5

    Factbeat 6 Hsbeat 14 COMMUNITY BEATS Sending all sorts of data to Logstash and Elasticsearch 7 Httpbeat 8 Nagioscheckbeat 9 Nginxbeat 10 Phpfpmbeat 11 Pingbeat 13 Unifiedbeat 12 Redisbeat 14 Uwsgibeat
  15. Community Beats: libbeat 25 libbeat Community Beats Elastic Beats •

    Golang library • Outputs for Elasticsearch and Logstash • At least once guarantees • Encryption & authentication • Common code for configuration files, logging, daemonizing, CLI flags, etc. Elasticsearch Logstash
  16. ‹#› 26 input: # Loop every 5 seconds period: 5

    # Use raw sockets for ping # Requires root! privileged: true # Whether to perform IPv4/v6 pings useipv4: true useipv6: false # List targets under the tag # you want assigned to targets: # tag: google google: - google.com.au - google.com You know, for pings • Sends ICMP (v4 or v6) pings periodically to a list of hosts • Can send also UDP pings (no root required) • Resolves DNS • Records RTT Pingbeat
  17. ‹#› 27 Unifiedbeat Parse IDS logs • Reads Unified2 binary

    files • Generated by Snort/Suricata IDS • Alternative for Barnyard2 + MySQL • Indexes also the rule/signature that have triggered the alert • Adds GeoIP information unifiedbeat: # Configure Snort Rules/Signatures # support: rules: paths: - “sample_data/rules/*.rules” # List of sensors sensor: - paths: - “sample_data/snort.log” fields: sensor_hostname: nucy sensor_interface: em1 sensor_type: snort
  18. ‹#› 29 Docker Monitoring • Uses the Docker API •

    Per container stats about: • CPU • Memory • Disk • Network Dockerbeat input: # In seconds, defines how often to # read server statistics period: 5 # Define the docker socket path # By default, this will get the # unix:///var/run/docker.sock socket: 29
  19. ‹#› 30 Nagioscheckbeat Run Nagios checks • Can execute any

    Nagios plugin • Execution period configurable per check • Sends alerts (Warning/Critical) to Elasticsearch • Sends performance data to Elasticsearch input: checks: - name: "disks" cmd: "plugins/check_disk" args: "-w 80 -c 90 -x /dev" period: "1h" - name: "load" cmd: "plugins/check_load" args: "-w 5 -c 10" period: "1m"
  20. Beat generator Quickly get started with the development of a

    new Beat 32 $ pip install cookiecutter $ cookiecutter https://github.com/elastic/beat-generator.git project_name [Examplebeat]: Mybeat github_name [your-github-name]: tsg beat [examplebeat]: mybeat beat_path [github.com/your-github-name]: github.com/tsg full_name [Firstname Lastname]: Tudor Golubenco
  21. 33 Beats Packer • Cross-compiles to all our supported platforms

    • Produces RPMs, DEBs, • Same tools that we use to build the official Elastic Beats • Can be executed from Travis CI
  22. 35 1 Apachebeat 2 Dockerbeat 3 Elasticbeat 4 Execbeat 5

    Factbeat 6 Hsbeat 7 Httpbeat 8 Nagioscheckbeat 9 Nginxbeat 10 Phpfpmbeat 11 Pingbeat 12 Redisbeat 13 Unifiedbeat 14 Uwsgibeat
  23. 36 1 Apachebeat 2 Dockerbeat 3 Elasticbeat 4 Execbeat 5

    Factbeat 6 Hsbeat 7 Httpbeat 8 Nagioscheckbeat 9 Nginxbeat 10 Phpfpmbeat 11 Pingbeat 14 Uwsgibeat 12 Redisbeat 13 Unifiedbeat
  24. Metricbeat: Collecting metrics from other systems 39 1 2 3

    Periodically polls monitoring APIs of various services Groups performance data into documents Ships them to Logstash / Elasticsearch
  25. 41 Metricbeat: It is also a library! • Use the

    Metricbeat infrastructure, to create a standalone Beat • One can create a Beat with a single Metricbeat module libbeat Metricbeat df module github.com/ruflin/df2beat
  26. Metricbeat module vs standalone Beat 42 • Contributed via PR

    to the elastic/beats Github repository • Officially supported • Supports common systems • Docker based integration tests Metricbeat module • In a separate Github repository • Supported by the community • Supports specialized systems • Optional Docker based integration tests Standalone Beat
  27. Intended deployment models 44 Topbeat Elasticsearch Topbeat & Packetbeat: Send

    directly to Elasticsearch Filebeat Logstash Elasticsearch Filebeat & Winlogbeat: Use Logstash for parsing
  28. Adding a queuing system 45 Filebeat Logstash Logstash Elasticsearch Redis

    / Kafka • You need two sets of Logstash instances: • One Logstash to send the data to Redis/Kafka • One Logstash to enrich and transport data securely to Elasticsearch
  29. 46 Kafka and Redis output • Have Redis output support

    in libbeat • Add Kafka output in Beats 5.0.0-alpha1 Filebeat Logstash Elasticsearch Redis / Kafka
  30. ‹#› Generic filtering 47 filter: # In Packetbeat, don’t store

    the # 200 OK transactions - drop_event: equals: “http.response.code”: 200 # In Topbeat, drop selected # fields - drop_fields: fields: - cpu.user - proc.cpu.total • Flexibly reduce the amount of data sent of the wire and stored • Filter fields or events with simple conditions • Part of libbeat
  31. ‹#› 48 Packetbeat flows { "@timestamp": "2016-02-11T11:12:09.718Z", "count": 1, "final":

    true, "icmp_id": 5, "ip4_dest": "10.0.0.2", "ip4_source": "10.0.0.1", "last_time": "2016-02-11T11:12:09.416Z", "mac_dest": "00:00:00:00:00:02", "mac_source": "00:00:00:00:00:01", "start_time": "2016-02-11T11:12:09.416Z", "stats_dest": { "net_bytes_total": 50, "net_packets_total": 1 }, "stats_source": { "net_bytes_total": 50, "net_packets_total": 1 }, "type": "flow", "vlan": 10 } • Look into data for which we don’t understand the application layer protocol • TLS • Protocols we don’t yet support • Get data about IP / TCP / UDP layers • number of packets • retransmissions • inter-arrival time
  32. ‹#› Q&A Find us on: • github.com/elastic/beats • discuss.elastic.co •

    @elastic #elasticbeats • #beats on freenode Or Here. In Real Life!
  33. ‹#› Please attribute Elastic with a link to elastic.co Except

    where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 50