Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
FileBeat (Won't save you from the JVM)
Chris Fidao
January 24, 2016
Technology
1
250
FileBeat (Won't save you from the JVM)
A quick presentation about using Elastic's FileBeat for log aggregation.
Chris Fidao
January 24, 2016
Tweet
Share
More Decks by Chris Fidao
See All by Chris Fidao
Development Environments that Feel Local
fideloper
0
10
Refactoring Terraform - CloudCasts - Scaling EC2
fideloper
0
17
Scaling Laravel - Laracon.net 2018
fideloper
15
1.6k
Linux Environment
fideloper
1
9.5k
Server Survival
fideloper
29
23k
Powering Your Applications With Nginx
fideloper
9
7.6k
Hexagonal Architecture
fideloper
49
190k
Intro to etcd
fideloper
3
470
Service Oriented Architecture with a little help from NodeJS
fideloper
4
2.2k
Other Decks in Technology
See All in Technology
ユーザーテストガイドライン VERSION 2.0
kouzoukaikaku
0
1.1k
オンプレk8sとEKSの並行運用の実際
ch1aki
0
260
01_ユーザーリサーチ実施の進め方
kouzoukaikaku
0
310
FlexScan HD2452Wの 後継を探して
tring
0
6.2k
SPA・SSGでSSRのようなOGP対応!
simo123
2
150
- Rでオブジェクト指向プログラミング- クラス設計入門の入門
kotatyamtema
1
720
MarvelClient Upgrade 64bit クライアントへの自動アップグレード設定
mitsuru_katoh
0
120
03_ユーザビリティテスト
kouzoukaikaku
0
300
データ分析基盤の要件分析の話(202201_JEDAI)
yabooun
0
240
IoT から見る AWS re:invent 2022 ― AWSのIoTの歴史を添えて/Point of view the AWS re:invent 2022 with IoT - with a history of IoT in AWS
ma2shita
0
250
ROS_Japan_UG_#49_LT
maeharakeisuke
0
220
AWS re:Invent 2022で発表された新機能を試してみた ~Cloud OperationとSecurity~ / New Cloud Operation and Security Features Announced at AWS reInvent 2022
yuj1osm
1
190
Featured
See All Featured
The Art of Programming - Codeland 2020
erikaheidi
35
11k
Making the Leap to Tech Lead
cromwellryan
117
7.7k
Building an army of robots
kneath
301
40k
Ruby is Unlike a Banana
tanoku
93
9.5k
A designer walks into a library…
pauljervisheath
199
16k
What's new in Ruby 2.0
geeforr
336
30k
How to Ace a Technical Interview
jacobian
270
21k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
44
14k
Building Adaptive Systems
keathley
27
1.3k
What's in a price? How to price your products and services
michaelherold
233
9.7k
Scaling GitHub
holman
453
140k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
7
570
Transcript
FileBeat (Won’t save you from the JVM)
Beats Data Shippers for Elasticsearch (written in Golang)
PacketBeat TopBeat
FileBeat
Versus:
!
None
None
None
Goal: "
# Download Filebeat Package (Debian/Ubuntu) curl -L -O https://download.elastic.co/beats/filebeat/ filebeat_1.0.1_amd64.deb
# Install from .deb file, # without worrying about dependencies, # because Golang ! sudo dpkg -i filebeat_1.0.1_amd64.deb #!/usr/bin/env bash
filebeat: prospectors: - paths: - /var/log/nginx/*.log input_type: log - paths:
- /var/log/php7.0-fpm.log input_type: log output: elasticsearch: hosts: [“https://search-sadevops.us-east-1.es.aws.com:443"] shipper: tags: ["web-service", "or-like-whatever"] /etc/filebeat/filebeat.yml
useless (un-parsed message)
Plaintext Log Message useless
You Can’t Escape the JVM
None
Plaintext Log Message Parsed Log (JSON)
# Install Java sudo apt-get install -y openjdk-7-jdk # Add
ES Key wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add - # Add Repo echo "deb http://packages.elastic.co/logstash/2.1/debian stable main" | sudo tee /etc/apt/sources.list.d/logstash.list # Update and install package sudo apt-get update sudo apt-get install -y logstash # Install Filebeat Plugin sudo /opt/logstash/bin/plugin install logstash-input-beats On a new server…
input { beats { type => beats port => 5044
} } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } remove_tag => ["_grokparsefailure"] add_tag => ["nginx_access"] } } output { elasticsearch { hosts => ["search-sadevops.us-east-1.es.aws.com:80"] } } /etc/logstash/conf.d/filebeat.conf
filebeat: prospectors: - paths: - /var/log/nginx/*.log input_type: log - paths:
- /var/log/php7.0-fpm.log input_type: log output: logstash: hosts: ["172.31.28.187:5044"] shipper: tags: ["web-service", "or-like-whatever"] /etc/filebeat/filebeat.yml
None
None
$ ab -n 50000 -c 2 localhost/ mehhhhhh
40% 250mb
Conclusion: (for my use case) Fluentd is good enough. •One
less server (yay!) •Trade-off of more ram used: acceptable •JVM is “scary”, because I’m ignorant •(But PacketBeat and TopBeat look really useful)
@fideloper Thanks! Chris Fidao