Tweet
Tweet

Slide 1

Slide 1 text

FileBeat (Won’t save you from the JVM)

Slide 2

Slide 2 text

Beats Data Shippers for Elasticsearch (written in Golang)

Slide 3

Slide 3 text

PacketBeat TopBeat

Slide 4

Slide 4 text

FileBeat

Slide 5

Slide 5 text

Versus:

Slide 6

Slide 6 text

!

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

Goal: "

Slide 11

Slide 11 text

# Download Filebeat Package (Debian/Ubuntu) curl -L -O https://download.elastic.co/beats/filebeat/ filebeat_1.0.1_amd64.deb # Install from .deb file, # without worrying about dependencies, # because Golang ! sudo dpkg -i filebeat_1.0.1_amd64.deb #!/usr/bin/env bash

Slide 12

Slide 12 text

filebeat: prospectors: - paths: - /var/log/nginx/*.log input_type: log - paths: - /var/log/php7.0-fpm.log input_type: log output: elasticsearch: hosts: [“https://search-sadevops.us-east-1.es.aws.com:443"] shipper: tags: ["web-service", "or-like-whatever"] /etc/filebeat/filebeat.yml

Slide 13

Slide 13 text

useless (un-parsed message)

Slide 14

Slide 14 text

Plaintext Log Message useless

Slide 15

Slide 15 text

You Can’t Escape the JVM

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

Plaintext Log Message Parsed Log (JSON)

Slide 18

Slide 18 text

# Install Java sudo apt-get install -y openjdk-7-jdk # Add ES Key wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add - # Add Repo echo "deb http://packages.elastic.co/logstash/2.1/debian stable main" | sudo tee /etc/apt/sources.list.d/logstash.list # Update and install package sudo apt-get update sudo apt-get install -y logstash # Install Filebeat Plugin sudo /opt/logstash/bin/plugin install logstash-input-beats On a new server…

Slide 19

Slide 19 text

input { beats { type => beats port => 5044 } } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } remove_tag => ["_grokparsefailure"] add_tag => ["nginx_access"] } } output { elasticsearch { hosts => ["search-sadevops.us-east-1.es.aws.com:80"] } } /etc/logstash/conf.d/filebeat.conf

Slide 20

Slide 20 text

filebeat: prospectors: - paths: - /var/log/nginx/*.log input_type: log - paths: - /var/log/php7.0-fpm.log input_type: log output: logstash: hosts: ["172.31.28.187:5044"] shipper: tags: ["web-service", "or-like-whatever"] /etc/filebeat/filebeat.yml

Slide 21

Slide 21 text

No content

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

$ ab -n 50000 -c 2 localhost/ mehhhhhh

Slide 24

Slide 24 text

40% 250mb

Slide 25

Slide 25 text

Conclusion: (for my use case) Fluentd is good enough. •One less server (yay!) •Trade-off of more ram used: acceptable •JVM is “scary”, because I’m ignorant •(But PacketBeat and TopBeat look really useful)

Slide 26

Slide 26 text

@fideloper Thanks! Chris Fidao