Upgrade to Pro — share decks privately, control downloads, hide ads and more …

FileBeat (Won't save you from the JVM)

Chris Fidao
January 24, 2016

FileBeat (Won't save you from the JVM)

A quick presentation about using Elastic's FileBeat for log aggregation.

Chris Fidao

January 24, 2016
Tweet

More Decks by Chris Fidao

Other Decks in Technology

Transcript

  1. FileBeat
    (Won’t save you from the JVM)

    View full-size slide

  2. Beats
    Data Shippers for Elasticsearch
    (written in Golang)

    View full-size slide

  3. PacketBeat
    TopBeat

    View full-size slide

  4. # Download Filebeat Package (Debian/Ubuntu)
    curl -L -O https://download.elastic.co/beats/filebeat/
    filebeat_1.0.1_amd64.deb
    # Install from .deb file,
    # without worrying about dependencies,
    # because Golang !
    sudo dpkg -i filebeat_1.0.1_amd64.deb
    #!/usr/bin/env bash

    View full-size slide

  5. filebeat:
    prospectors:
    -
    paths:
    - /var/log/nginx/*.log
    input_type: log
    -
    paths:
    - /var/log/php7.0-fpm.log
    input_type: log
    output:
    elasticsearch:
    hosts: [“https://search-sadevops.us-east-1.es.aws.com:443"]
    shipper:
    tags: ["web-service", "or-like-whatever"]
    /etc/filebeat/filebeat.yml

    View full-size slide

  6. useless
    (un-parsed message)

    View full-size slide

  7. Plaintext Log
    Message
    useless

    View full-size slide

  8. You
    Can’t Escape
    the JVM

    View full-size slide

  9. Plaintext Log
    Message
    Parsed Log (JSON)

    View full-size slide

  10. # Install Java
    sudo apt-get install -y openjdk-7-jdk
    # Add ES Key
    wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch
    | sudo apt-key add -
    # Add Repo
    echo "deb http://packages.elastic.co/logstash/2.1/debian stable
    main" | sudo tee /etc/apt/sources.list.d/logstash.list
    # Update and install package
    sudo apt-get update
    sudo apt-get install -y logstash
    # Install Filebeat Plugin
    sudo /opt/logstash/bin/plugin install logstash-input-beats
    On a new server…

    View full-size slide

  11. input {
    beats {
    type => beats
    port => 5044
    }
    }
    filter {
    grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
    remove_tag => ["_grokparsefailure"]
    add_tag => ["nginx_access"]
    }
    }
    output {
    elasticsearch {
    hosts => ["search-sadevops.us-east-1.es.aws.com:80"]
    }
    }
    /etc/logstash/conf.d/filebeat.conf

    View full-size slide

  12. filebeat:
    prospectors:
    -
    paths:
    - /var/log/nginx/*.log
    input_type: log
    -
    paths:
    - /var/log/php7.0-fpm.log
    input_type: log
    output:
    logstash:
    hosts: ["172.31.28.187:5044"]
    shipper:
    tags: ["web-service", "or-like-whatever"]
    /etc/filebeat/filebeat.yml

    View full-size slide

  13. $ ab -n 50000 -c 2 localhost/
    mehhhhhh

    View full-size slide

  14. Conclusion:
    (for my use case)
    Fluentd is good enough.
    •One less server (yay!)
    •Trade-off of more ram used: acceptable
    •JVM is “scary”, because I’m ignorant
    •(But PacketBeat and TopBeat look really useful)

    View full-size slide

  15. @fideloper
    Thanks!
    Chris Fidao

    View full-size slide