Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
FileBeat (Won't save you from the JVM)
Search
Chris Fidao
January 24, 2016
Technology
1
330
FileBeat (Won't save you from the JVM)
A quick presentation about using Elastic's FileBeat for log aggregation.
Chris Fidao
January 24, 2016
Tweet
Share
More Decks by Chris Fidao
See All by Chris Fidao
Development Environments that Feel Local
fideloper
0
50
Refactoring Terraform - CloudCasts - Scaling EC2
fideloper
0
77
Scaling Laravel - Laracon.net 2018
fideloper
15
1.9k
Linux Environment
fideloper
1
11k
Server Survival
fideloper
29
23k
Powering Your Applications With Nginx
fideloper
9
7.7k
Hexagonal Architecture
fideloper
49
200k
Intro to etcd
fideloper
3
580
Service Oriented Architecture with a little help from NodeJS
fideloper
4
2.3k
Other Decks in Technology
See All in Technology
AIでテストプロセス自動化に挑戦する
sakatakazunori
1
400
[SRE NEXT 2025] すみずみまで暖かく照らすあなたの太陽でありたい
carnappopper
2
390
QuickSight SPICE の効果的な運用戦略~S3 + Athena 構成での実践ノウハウ~/quicksight-spice-s3-athena-best-practices
emiki
0
280
モニタリング統一への道のり - 分散モニタリングツール統合のためのオブザーバビリティプロジェクト
niftycorp
PRO
1
450
研究開発部メンバーの働き⽅ / Sansan R&D Profile
sansan33
PRO
3
18k
AWS CDKの仕組み / how-aws-cdk-works
gotok365
10
1k
データ戦略部門 紹介資料
sansan33
PRO
1
3.3k
All About Sansan – for New Global Engineers
sansan33
PRO
1
1.2k
Rethinking Incident Response: Context-Aware AI in Practice
rrreeeyyy
1
830
ClaudeCodeにキレない技術
gtnao
1
820
american airlines®️ USA Contact Numbers: Complete 2025 Support Guide
supportflight
1
120
Introduction to Sansan, inc / Sansan Global Development Center, Inc.
sansan33
PRO
0
2.7k
Featured
See All Featured
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
2.9k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
50
5.5k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
31
1.3k
It's Worth the Effort
3n
185
28k
Raft: Consensus for Rubyists
vanstee
140
7k
Music & Morning Musume
bryan
46
6.7k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
Facilitating Awesome Meetings
lara
54
6.5k
Code Review Best Practice
trishagee
69
19k
Thoughts on Productivity
jonyablonski
69
4.7k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
667
120k
Transcript
FileBeat (Won’t save you from the JVM)
Beats Data Shippers for Elasticsearch (written in Golang)
PacketBeat TopBeat
FileBeat
Versus:
!
None
None
None
Goal: "
# Download Filebeat Package (Debian/Ubuntu) curl -L -O https://download.elastic.co/beats/filebeat/ filebeat_1.0.1_amd64.deb
# Install from .deb file, # without worrying about dependencies, # because Golang ! sudo dpkg -i filebeat_1.0.1_amd64.deb #!/usr/bin/env bash
filebeat: prospectors: - paths: - /var/log/nginx/*.log input_type: log - paths:
- /var/log/php7.0-fpm.log input_type: log output: elasticsearch: hosts: [“https://search-sadevops.us-east-1.es.aws.com:443"] shipper: tags: ["web-service", "or-like-whatever"] /etc/filebeat/filebeat.yml
useless (un-parsed message)
Plaintext Log Message useless
You Can’t Escape the JVM
None
Plaintext Log Message Parsed Log (JSON)
# Install Java sudo apt-get install -y openjdk-7-jdk # Add
ES Key wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add - # Add Repo echo "deb http://packages.elastic.co/logstash/2.1/debian stable main" | sudo tee /etc/apt/sources.list.d/logstash.list # Update and install package sudo apt-get update sudo apt-get install -y logstash # Install Filebeat Plugin sudo /opt/logstash/bin/plugin install logstash-input-beats On a new server…
input { beats { type => beats port => 5044
} } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } remove_tag => ["_grokparsefailure"] add_tag => ["nginx_access"] } } output { elasticsearch { hosts => ["search-sadevops.us-east-1.es.aws.com:80"] } } /etc/logstash/conf.d/filebeat.conf
filebeat: prospectors: - paths: - /var/log/nginx/*.log input_type: log - paths:
- /var/log/php7.0-fpm.log input_type: log output: logstash: hosts: ["172.31.28.187:5044"] shipper: tags: ["web-service", "or-like-whatever"] /etc/filebeat/filebeat.yml
None
None
$ ab -n 50000 -c 2 localhost/ mehhhhhh
40% 250mb
Conclusion: (for my use case) Fluentd is good enough. •One
less server (yay!) •Trade-off of more ram used: acceptable •JVM is “scary”, because I’m ignorant •(But PacketBeat and TopBeat look really useful)
@fideloper Thanks! Chris Fidao