Upgrade to Pro — share decks privately, control downloads, hide ads and more …

FileBeat (Won't save you from the JVM)

Chris Fidao
January 24, 2016

FileBeat (Won't save you from the JVM)

A quick presentation about using Elastic's FileBeat for log aggregation.

Chris Fidao

January 24, 2016
Tweet

More Decks by Chris Fidao

Other Decks in Technology

Transcript

  1. FileBeat (Won’t save you from the JVM)

  2. Beats Data Shippers for Elasticsearch (written in Golang)

  3. PacketBeat TopBeat

  4. FileBeat

  5. Versus:

  6. !

  7. None
  8. None
  9. None
  10. Goal: "

  11. # Download Filebeat Package (Debian/Ubuntu) curl -L -O https://download.elastic.co/beats/filebeat/ filebeat_1.0.1_amd64.deb

    # Install from .deb file, # without worrying about dependencies, # because Golang ! sudo dpkg -i filebeat_1.0.1_amd64.deb #!/usr/bin/env bash
  12. filebeat: prospectors: - paths: - /var/log/nginx/*.log input_type: log - paths:

    - /var/log/php7.0-fpm.log input_type: log output: elasticsearch: hosts: [“https://search-sadevops.us-east-1.es.aws.com:443"] shipper: tags: ["web-service", "or-like-whatever"] /etc/filebeat/filebeat.yml
  13. useless (un-parsed message)

  14. Plaintext Log Message useless

  15. You Can’t Escape the JVM

  16. None
  17. Plaintext Log Message Parsed Log (JSON)

  18. # Install Java sudo apt-get install -y openjdk-7-jdk # Add

    ES Key wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add - # Add Repo echo "deb http://packages.elastic.co/logstash/2.1/debian stable main" | sudo tee /etc/apt/sources.list.d/logstash.list # Update and install package sudo apt-get update sudo apt-get install -y logstash # Install Filebeat Plugin sudo /opt/logstash/bin/plugin install logstash-input-beats On a new server…
  19. input { beats { type => beats port => 5044

    } } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } remove_tag => ["_grokparsefailure"] add_tag => ["nginx_access"] } } output { elasticsearch { hosts => ["search-sadevops.us-east-1.es.aws.com:80"] } } /etc/logstash/conf.d/filebeat.conf
  20. filebeat: prospectors: - paths: - /var/log/nginx/*.log input_type: log - paths:

    - /var/log/php7.0-fpm.log input_type: log output: logstash: hosts: ["172.31.28.187:5044"] shipper: tags: ["web-service", "or-like-whatever"] /etc/filebeat/filebeat.yml
  21. None
  22. None
  23. $ ab -n 50000 -c 2 localhost/ mehhhhhh

  24. 40% 250mb

  25. Conclusion: (for my use case) Fluentd is good enough. •One

    less server (yay!) •Trade-off of more ram used: acceptable •JVM is “scary”, because I’m ignorant •(But PacketBeat and TopBeat look really useful)
  26. @fideloper Thanks! Chris Fidao