Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
FileBeat (Won't save you from the JVM)
Search
Chris Fidao
January 24, 2016
Technology
1
330
FileBeat (Won't save you from the JVM)
A quick presentation about using Elastic's FileBeat for log aggregation.
Chris Fidao
January 24, 2016
Tweet
Share
More Decks by Chris Fidao
See All by Chris Fidao
Development Environments that Feel Local
fideloper
0
51
Refactoring Terraform - CloudCasts - Scaling EC2
fideloper
0
78
Scaling Laravel - Laracon.net 2018
fideloper
15
1.9k
Linux Environment
fideloper
1
11k
Server Survival
fideloper
29
23k
Powering Your Applications With Nginx
fideloper
9
7.7k
Hexagonal Architecture
fideloper
49
200k
Intro to etcd
fideloper
3
580
Service Oriented Architecture with a little help from NodeJS
fideloper
4
2.3k
Other Decks in Technology
See All in Technology
室長の逆襲 :データ活用の陣地を増やすためのヒント
masatoshi0205
0
180
Shadow DOMとセキュリティ - 光と影の境界を探る / Shibuya.XSS techtalk #13
masatokinugawa
0
270
公開初日に個人環境で試した Gemini CLI 体験記など / Gemini CLI実験レポート
you
PRO
3
290
データ戦略部門 紹介資料
sansan33
PRO
1
3.3k
Maintainer Meetupで「生の声」を聞く ~講演だけじゃないKubeCon
logica0419
1
160
Digitization部 紹介資料
sansan33
PRO
1
4.6k
ML Pipelineの開発と運用を OpenTelemetryで繋ぐ @ OpenTelemetry Meetup 2025-07
getty708
0
210
研究開発部メンバーの働き⽅ / Sansan R&D Profile
sansan33
PRO
3
18k
Microsoft Fabric ガバナンス設計の一歩目を考える
ryomaru0825
1
250
PHPからはじめるコンピュータアーキテクチャ / From Scripts to Silicon: A Journey Through the Layers of Computing
tomzoh
2
380
Ktor + Google Cloud Tasks/PubSub におけるOTel Messaging計装の実践
sansantech
PRO
1
230
Microsoft Defender XDRで疲弊しないためのインシデント対応
sophiakunii
3
400
Featured
See All Featured
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
357
30k
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
The Power of CSS Pseudo Elements
geoffreycrofte
77
5.9k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
35
2.5k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
367
26k
The Invisible Side of Design
smashingmag
301
51k
Done Done
chrislema
184
16k
Thoughts on Productivity
jonyablonski
69
4.7k
VelocityConf: Rendering Performance Case Studies
addyosmani
332
24k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
34
3.1k
Designing for humans not robots
tammielis
253
25k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.4k
Transcript
FileBeat (Won’t save you from the JVM)
Beats Data Shippers for Elasticsearch (written in Golang)
PacketBeat TopBeat
FileBeat
Versus:
!
None
None
None
Goal: "
# Download Filebeat Package (Debian/Ubuntu) curl -L -O https://download.elastic.co/beats/filebeat/ filebeat_1.0.1_amd64.deb
# Install from .deb file, # without worrying about dependencies, # because Golang ! sudo dpkg -i filebeat_1.0.1_amd64.deb #!/usr/bin/env bash
filebeat: prospectors: - paths: - /var/log/nginx/*.log input_type: log - paths:
- /var/log/php7.0-fpm.log input_type: log output: elasticsearch: hosts: [“https://search-sadevops.us-east-1.es.aws.com:443"] shipper: tags: ["web-service", "or-like-whatever"] /etc/filebeat/filebeat.yml
useless (un-parsed message)
Plaintext Log Message useless
You Can’t Escape the JVM
None
Plaintext Log Message Parsed Log (JSON)
# Install Java sudo apt-get install -y openjdk-7-jdk # Add
ES Key wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add - # Add Repo echo "deb http://packages.elastic.co/logstash/2.1/debian stable main" | sudo tee /etc/apt/sources.list.d/logstash.list # Update and install package sudo apt-get update sudo apt-get install -y logstash # Install Filebeat Plugin sudo /opt/logstash/bin/plugin install logstash-input-beats On a new server…
input { beats { type => beats port => 5044
} } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } remove_tag => ["_grokparsefailure"] add_tag => ["nginx_access"] } } output { elasticsearch { hosts => ["search-sadevops.us-east-1.es.aws.com:80"] } } /etc/logstash/conf.d/filebeat.conf
filebeat: prospectors: - paths: - /var/log/nginx/*.log input_type: log - paths:
- /var/log/php7.0-fpm.log input_type: log output: logstash: hosts: ["172.31.28.187:5044"] shipper: tags: ["web-service", "or-like-whatever"] /etc/filebeat/filebeat.yml
None
None
$ ab -n 50000 -c 2 localhost/ mehhhhhh
40% 250mb
Conclusion: (for my use case) Fluentd is good enough. •One
less server (yay!) •Trade-off of more ram used: acceptable •JVM is “scary”, because I’m ignorant •(But PacketBeat and TopBeat look really useful)
@fideloper Thanks! Chris Fidao