FileBeat (Won't save you from the JVM)

A quick presentation about using Elastic's FileBeat for log aggregation.

Transcript

1. FileBeat (Won’t save you from the JVM)
2. Beats Data Shippers for Elasticsearch (written in Golang)
3. PacketBeat TopBeat
4. FileBeat
5. Versus:
6. !
10. Goal: "
11. # Download Filebeat Package (Debian/Ubuntu) curl -L -O https://download.elastic.co/beats/filebeat/ filebeat_1.0.1_amd64.deb # Install from .deb file, # without worrying about dependencies, # because Golang ! sudo dpkg -i filebeat_1.0.1_amd64.deb #!/usr/bin/env bash
12. filebeat: prospectors: - paths: - /var/log/nginx/*.log input_type: log - paths: - /var/log/php7.0-fpm.log input_type: log output: elasticsearch: hosts: [“https://search-sadevops.us-east-1.es.aws.com:443"] shipper: tags: ["web-service", "or-like-whatever"] /etc/filebeat/filebeat.yml
13. useless (un-parsed message)
14. Plaintext Log Message useless
15. You Can’t Escape the JVM
17. Plaintext Log Message Parsed Log (JSON)
18. # Install Java sudo apt-get install -y openjdk-7-jdk # Add ES Key wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add - # Add Repo echo "deb http://packages.elastic.co/logstash/2.1/debian stable main" | sudo tee /etc/apt/sources.list.d/logstash.list # Update and install package sudo apt-get update sudo apt-get install -y logstash # Install Filebeat Plugin sudo /opt/logstash/bin/plugin install logstash-input-beats On a new server…
19. input { beats { type => beats port => 5044 } } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } remove_tag => ["_grokparsefailure"] add_tag => ["nginx_access"] } } output { elasticsearch { hosts => ["search-sadevops.us-east-1.es.aws.com:80"] } } /etc/logstash/conf.d/filebeat.conf
20. filebeat: prospectors: - paths: - /var/log/nginx/*.log input_type: log - paths: - /var/log/php7.0-fpm.log input_type: log output: logstash: hosts: ["172.31.28.187:5044"] shipper: tags: ["web-service", "or-like-whatever"] /etc/filebeat/filebeat.yml
23. $ ab -n 50000 -c 2 localhost/ mehhhhhh
24. 40% 250mb
25. Conclusion: (for my use case) Fluentd is good enough. •One less server (yay!) •Trade-off of more ram used: acceptable •JVM is “scary”, because I’m ignorant •(But PacketBeat and TopBeat look really useful)
26. @fideloper Thanks! Chris Fidao