Unifying logs and metrics with Elastic Beats

Slide 1

Slide 1 text

‹#› Unifying logs and metrics data with Elastic Beats Monica Sarbu Team lead, Elastic Beats

Slide 2

Slide 2 text

Who am I 2 https://www.flickr.com/photos/ofernandezberrios/7176474422 • Team lead at Elastic Beats • Software engineer • Joined Elastic 1 year ago @monicasarbu

Slide 3

Slide 3 text

Beats are lightweight shippers that collect and ship all kinds of operational data to Elasticsearch 3

Slide 4

Slide 4 text

Beats are lightweight shippers that collect and ship all kinds of operational data to Elasticsearch 4

Slide 5

Slide 5 text

5 Lightweight shippers • Lightweight application • Written in Golang • Install as agent on your servers • No runtime dependencies • Single purpose

Slide 6

Slide 6 text

Beats are lightweight shippers that collect and ship all kinds of operational data to Elasticsearch 6

Slide 7

Slide 7 text

7 All kinds of operational data • Filebeat • collects logs • Winlogbeat • collects Windows event logs • Packetbeat • collects insides from the network packets not released • Topbeat • collects system statistics like CPU usage, disk usage, memory usage per process, etc • Metricbeat • collects metrics by interrogating periodically external services

Slide 8

Slide 8 text

Beats are lightweight shippers that collect and ship all kinds of operational data to Elasticsearch 8

Slide 9

Slide 9 text

‹#› In Elasticsearch .. you are storing the raw value … You have the ability to ask and answer questions that you didn’t think about when the data was stored! Felix Barnsteiner

Slide 10

Slide 10 text

The Elastic Stack 10

Slide 11

Slide 11 text

‹#›

Slide 12

Slide 12 text

Captures insights from network packets 12 Packetbeat

Slide 13

Slide 13 text

Sniffing the network traffic 13 • Copy traffic at OS or hardware level • ZERO latency overhead • Not in the request/response path, cannot break your application Client Server sniff sniff

Slide 14

Slide 14 text

14 Sniffing use cases • Security • Intrusion Detection Systems • Troubleshooting network issues • Troubleshooting applications • Performance analysis

Slide 15

Slide 15 text

Monitor the network traffic with OSS tools 15 1 2 3 4 ssh to each of your server start a trace using tcpdump on each of your server download trace from each server to a common location merge all traces 5 analyze it with Wireshark

Slide 16

Slide 16 text

The Problem 16 1 2 3 you have lots of servers challenging to see the traffic exchanged between your servers Packetbeat makes it easy

Slide 17

Slide 17 text

Packetbeat overview It does all of this in real time directly on the target servers 17 1 2 3 4 capture network traffic decodes network traffic correlates request & response into transactions extract measurements 5 send data to Elasticsearch

Slide 18

Slide 18 text

Packetbeat: Available decoders 18 HTTP MySQL PostgreSQL MongoDB (community) Memcache ICMP (community) + Add your own Thrift-RPC DNS (community) Redis AMQP (community) NFS (community)

Slide 19

Slide 19 text

Packetbeat: Configuration 19 # Network interfaces where to sniff the data interfaces: device: any # Specify the type of your network data protocols: dns: ports: [53] http: ports: [80, 8080, 8081, 5000, 8002] mysql: ports: [3306] …

Slide 20

Slide 20 text

Slide 21

Slide 21 text

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Slide 24

Slide 24 text

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Slide 31

Slide 31 text

‹#› 31 Packetbeat flows • Look into data for which we don’t understand the application layer protocol • TLS • Protocols we don’t yet support • Get data about IP / TCP / UDP layers • number of packets • retransmissions • inter-arrival time flows: # network flow timeout timeout: 30s # reporting period period: 10s

Slide 32

Slide 32 text

Slide 33

Slide 33 text

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Collects log lines 35 Filebeat

Slide 36

Slide 36 text

36 Filebeat overview • Simple log forwarder that sends the log lines to Elasticsearch • Successor of Logstash Forwarder • It remembers how far it read, so it never loses log line • Reads the log files line by line • It doesn’t parse the log lines!

Slide 37

Slide 37 text

Filebeat: Parse logs with Logstash 37 • Filebeat sends out unparsed log lines • Use filters like Grok, mutate, geoip to parse the log lines • Combine the filters with conditionals or create custom filters in ruby • Forward data to other systems using the Logstash output plugins Filebeat Elasticsearch Logstash Other systems

Slide 38

Slide 38 text

Filebeat: Parse logs with Ingest Node 38 • Ingest node plugin is available starting with Elasticsearch 5.0.0- alpha1 • Filebeat sends out unparsed log lines directly to Elasticsearch • Use Ingest Node processors to parse the log lines • Easier to setup Filebeat Elasticsearch

Slide 39

Slide 39 text

Filebeat: Configuration Configure prospectors to forward the log lines 39 filebeat: # List of prospectors to fetch data. prospectors: # Type of files: log or stdin - input_type: log # Files that should be crawled and fetched. paths: - “/var/log/apache2/*” # File encoding: plain, utf-8, big5, gb18030, … encoding: plain

Slide 40

Slide 40 text

Slide 41

Slide 41 text

Slide 42

Slide 42 text

‹#› Multiline 42 multiline: # Sticks together all lines # that don’t start with a [ pattern: ^\[ negate: true match: after Filebeat extra power • Sticks together related log lines in a single event • For all those long exceptions • Can also be done by Logstash, but it’s sometimes easier to configure the patterns closer to the source

Slide 43

Slide 43 text

Slide 44

Slide 44 text

‹#›

Slide 45

Slide 45 text

‹#› 45 json: keys_under_root: false message_key: “message” overwrite_keys: false add_error_key: false Filebeat extra power JSON logs • application logs in JSON format • you don’t have to choose what data to include in the log line • don’t need to use grok filters from Logstash to parse the application logs

Slide 46

Slide 46 text

Slide 47

Slide 47 text

Slide 48

Slide 48 text

‹#› Basic filtering 48 # Only send lines starting with # ERR or WARN include_lines: [“^ERR”, “^WARN”] # Exclude lines containing # a keyword exclude_lines: [“Request received”] # Exclude files all together exclude_files: [“.gz$”] Filebeat extra power • Because removing stuff at the source is more efficient • Flexible Whitelist + Blacklist regexp log line filtering • Efficient log files filtering (excluded files are never opened) • Works on multiline too

Slide 49

Slide 49 text

Collects Windows Event logs 49 Winlogbeat

Slide 50

Slide 50 text

50 Winlogbeat overview • Sends out unparsed Windows event logs • Remembers how far it read, so it never loses any Windows event logs • Use Ingest Node or Logstash to parse the Windows event logs

Slide 51

Slide 51 text

Winlogbeat: Configuration Specify the event logs that you want to monitor 51 winlogbeat: #list of event logs to monitor event_logs: - name: Application - name: Security - name: System

Slide 52

Slide 52 text

Slide 53

Slide 53 text

Collects system statistics 53 Topbeat

Slide 54

Slide 54 text

54 Topbeat overview • Like the Unix top command but instead of printing the system statistics on the screen it sends them periodically to Elasticsearch • Works also on Windows

Slide 55

Slide 55 text

Topbeat: Exported data 55 • system load • total CPU usage • CPU usage per core • Swap, memory usage System wide • state • name • command line • pid • CPU usage • memory usage Per process • available disks • used, free space • mounted points Disk usage

Slide 56

Slide 56 text

Topbeat configuration Specify the system statistics that you want to monitor 56 topbeat: # how often to send system statistics period: 10 # specify the processes to monitor procs: [".*"] # Statistics to collect (all enabled by default) stats: system: true process: true filesystem: true

Slide 57

Slide 57 text

Slide 58

Slide 58 text

‹#›

Slide 59

Slide 59 text

‹#›

Slide 60

Slide 60 text

‹#›

Slide 61

Slide 61 text

‹#›

Slide 62

Slide 62 text

‹#›

Slide 63

Slide 63 text

‹#›

Slide 64

Slide 64 text

‹#›

Slide 65

Slide 65 text

‹#›

Slide 66

Slide 66 text

Collects periodically metrics from external systems. 66 Metricbeat in progress

Slide 67

Slide 67 text

Metricbeat: how it works 67 1 2 3 Periodically polls monitoring APIs of various services Groups performance data into documents Ships them to Elasticsearch

Slide 68

Slide 68 text

Metricbeat: A module for each metric type 68 Metricbeat apache module mysql module redis module system module +

Slide 69

Slide 69 text

69 Metricbeat: It is also a library! • Use the Metricbeat infrastructure, to create a standalone Beat • You can create a Beat with a single module that exports your custom data • Can use the built in Metricbeat modules Metricbeat df module github.com/ruflin/df2beat

Slide 70

Slide 70 text

Metricbeat module vs standalone Beat 70 • Contributed via PR to the elastic/beats Github repository • Officially supported • Supports common systems • Docker based integration tests Metricbeat module • In a separate Github repository • Supported by the community • Supports specialized systems • Optional Docker based integration tests Standalone Beat

Slide 71

Slide 71 text

Provide a platform to make it easier to build custom Beats on top of it 71

Slide 72

Slide 72 text

Beats platform 72 Beat 1 libbeat Beat 2 Beat 3 +

Slide 73

Slide 73 text

libbeat 73 • Written in Go • Provide common functionality for reading configuration files, for handling CLI arguments, for logging • Makes sure reliably send the data out • Provide things like encryption, authentication with certificates • Has support for different outputs: Elasticsearch, Logstash, Redis, Kafka libbeat Outputs

Slide 74

Slide 74 text

‹#› Community Beats

Slide 75

Slide 75 text

Community Beats 75 libbeat Community Beats Elastic Beats Collect, Parse & Ship • Standalone projects • Written in Go • Use libbeat • Concentrate only on collecting the data • Solve a specific use case

Slide 76

Slide 76 text

Official vs Community Beats 76 • In the elastic/beats Github repository • Officially supported • Synced releases with the whole stack Official Beats • In another Github repository • Supported by the community • Releases at any time Community Beats

Slide 77

Slide 77 text

77 1 Apachebeat 2 Dockerbeat 3 Elasticbeat 4 Execbeat 5 Factbeat 6 Hsbeat 20 COMMUNITY BEATS Sending all sorts of data to Elasticsearch 7 Httpbeat 8 Nagioscheckbeat 9 Nginxbeat 10 Phpfpmbeat 11 Pingbeat 13 Unifiedbeat 12 Redisbeat 14 Uwsgibeat 15 Flowbeat 16 Lmsensorsbeat 17 Twitterbeat 18 Upbeat 19 Wmibeat 20 Packagebeat

Slide 78

Slide 78 text

‹#› 78 input: # Loop every 5 seconds period: 5 # Use raw sockets for ping # Requires root! privileged: true # Whether to perform IPv4/v6 pings useipv4: true useipv6: false # List targets under the tag # you want assigned to targets: # tag: google google: - google.com.au - google.com You know, for pings • Sends ICMP (v4 or v6) pings periodically to a list of hosts • Can send also UDP pings (no root required) • Resolves DNS • Records RTT Pingbeat

Slide 79

Slide 79 text

Pingbeat output 79 { "@timestamp": "2016-02-08T11:02:22.675Z", "beat": { "hostname": "Tudors-MBP", "name": "Tudors-MBP" }, "count": 1, "rtt": 25.336089, "tag": "google", "target_addr": "216.58.213.227", "target_name": "google.com.au", "type": "pingbeat" }

Slide 80

Slide 80 text

‹#› 80 Execbeat execbeat: execs: # Each - Commands to execute. - # Cron expression # Default is every 1 minute. cron: "@every 10s" # The command to execute command: echo args: "Hello World" document_type: jolokia fields: host: test2 • Accepts cron expressions • Sends stdout and stderr to Elastic search • Use Logstash and Grok to further parse the output Run any command

Slide 81

Slide 81 text

Execbeat output 81 { "@timestamp": "2016-02-08T11:59:36.007Z", "beat": { "hostname": "Tudors-MBP", "name": "Tudors-MBP" }, "exec": { "command": "echo", "stdout": "Hello World\n" }, "fields": { "host": "test2" }, "type": "jolokia" }

Slide 82

Slide 82 text

‹#› 82 Dockerbeat Docker Monitoring • Uses the Docker API • Exports per container stats about: • CPU • Memory • Disk • Network • IO access • Log input: # In seconds, defines how often to # read server statistics period: 5 # Define the docker socket path # By default, this will get the # unix:///var/run/docker.sock socket:

Slide 83

Slide 83 text

Dockerbeat output 83 { "@timestamp": "2016-02-08T12:44:56.136Z", "containerID": "17021c571d69fe4e93ee395b129c0f073d8aed6d618c9d0d805f68e0b66b2c3f", "containerName": "kibana", "memory": { "failcnt": 0, "limit": 1044586496, "maxUsage": 68485120, "usage": 9732096, "usage_p": 0.009316697121077851 }, "type": "memory" }

Slide 84

Slide 84 text

‹#› 84 Nagioscheckbeat Run Nagios checks • Can execute any Nagios plugin • Execution period configurable per check • Sends alerts (Warning/Critical) to Elasticsearch • Sends performance data to Elasticsearch input: checks: - name: "disks" cmd: "plugins/check_disk" args: "-w 80 -c 90 -x /dev" period: "1h" - name: "load" cmd: "plugins/check_load" args: "-w 5 -c 10" period: "1m"

Slide 85

Slide 85 text

Nagioscheckbeat output 85 { "@timestamp": "2015-12-30T18:56:33.933Z", "args": "-w 5 -c 10", "cmd": "/usr/lib64/nagios/plugins/check_load", "count": 1, "message": "OK - load average: 0.16, 0.05, 0.06", "status": "OK", "took_ms": 14, "type": "nagioscheck" }

Slide 86

Slide 86 text

Provide a platform to make it easier to build custom Beats on top of it 86

Slide 87

Slide 87 text

Beat generator Generate the boilerplate code for you 87 $ pip install cookiecutter $ cookiecutter https://github.com/elastic/beat-generator.git project_name [Examplebeat]: Mybeat github_name [your-github-name]: monicasarbu beat [examplebeat]: mybeat beat_path [github.com/your-github-name]: github.com/ monicasarbu full_name [Firstname Lastname]: Monica Sarbu

Slide 88

Slide 88 text

88 Beats Packer • Cross-compiles to all our supported platforms • Produces RPMs, DEBs, • Same tools that we use to build the official Elastic Beats • Can be executed from Travis CI

Slide 89

Slide 89 text

Multiple data types, one view in Kibana 89 • metrics • flows • logs • system stats • transactions • transactions • metrics • metrics • logs • logs • system stats • flows • flows • metrics • logs

Slide 90

Slide 90 text

Monitor MySQL with Elastic Stack 90 Metricbeat mysql … Filebeat log … Packetbeat mysql … Elasticsearch Kibana stats queries slow queries

Slide 91

Slide 91 text

Monitor web server with Elastic Stack 91 Metricbeat mysql apache Filebeat log … Packetbeat mysql http Elasticsearch Kibana mysql & apache stats queries & HTTP transactions slow queries apache logs

Slide 92

Slide 92 text

‹#› Thank you

Slide 93

Slide 93 text

‹#› 93 Want to hear more about Logstash? Don’t miss Ingest Logs with Style by Pere Urbon-Bayes Thursday 12:00pm - 1:00pm in MOA 05

Slide 94

Slide 94 text

‹#› Q&A Find us on: • github.com/elastic/beats • discuss.elastic.co • @elastic #elasticbeats • #beats on freenode Or Here. In Real Life!

Slide 95

Slide 95 text

‹#› Please attribute Elastic with a link to elastic.co Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 95