Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Berlin 2013 - Session - Radu Gheorghe
Search
Monitorama
September 20, 2013
690
0
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Berlin 2013 - Session - Radu Gheorghe
Monitorama
September 20, 2013
Transcript
39 slides
On Centralizing Logs Radu Gheorghe @radu0gheorghe
[email protected]
@sematext
Hello World! Logsene mlmoneu13cf for -44%
app app app app files files
app app app app files files Elasticsearch logstash Kibana
Elasticsearch Reason #1: Quick Search No indexing But... =>
...and other reasons good write speed lots of tools for
logging scales easily
Production Tips stability performance
Stability 1/4: Discovery multicast unicast vs cluster name list of
nodes + plugins: EC2, GCE
Stability 2/4: Preventing Split Brain minimum_master_nodes = N/2 + 1
Stability 3/4: No OOMs, pls! 1GB ½ total RAM Monitor
the requirements SPM for Elasticsearch 20% off with MONEU2013
Stability 4/4: Field Cache can be changed to index.cache.field.type: soft
indices.fielddata.cache.size: X%
Performance 1/4: Bulk Processing use Bulk API or Bulk UDP
API ...translog.flush_threshold_ops
Performance 2/4: Refresh Interval http://blog.sematext.com/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/ default: every second => but
every 5s +25% indexing* every 30s +70% indexing*
Performance 3/4: Timed Indices
Performance 4/4: Buffers ...index_buffer_size: 30% (YMMV) index.store.type: mmapfs (on 64-bit
machines) http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
Setting Up Kibana as Frontend servers you
Kibana: Search
Kibana: Visualize
Meet Some Syslog Daemons syslogd traditional everywhere syslog-ng OSE, PE
documentation++ config format++ rsyslog OSS only ES output* * http://blog.sematext.com/2013/07/01/recipe-rsyslog-elasticsearch-kibana/
X-ray of a Modern Syslog Daemon read+buffer file /dev/log …
parse syslog formats JSON unstructured data assemble conditionals formatting ... buffer+write file syslog Elasticsearch ...
2001's RFC3164: The Semi-Standard <10>Oct 11 22:14:15 host program:hello world
TCP + LF = no year, ms, nor TZ little structure
2009's RFC5424 <165>1 2003-10-11T22:14:15.003Z host program - - - [origin
ip="192.168.0.1"] hello world [ structured=data ] octet-count* + LF = * UDP (RFC5426), TCP (RFC6587), TLS (RFC5425)
Teaching Old Dog New Tricks RSYSLOG_ForwardFormat (ISO8601 over RFC3164) $MaxMessageSize
2048k log_message_size(2097152) @cee: {"message": "hello world"} @@(o)192.168.0.1 octet-counted framing
Reliable Transport? Encryption? TCP + TLS (RFC5425) RLTP + TLS
RELP + TLS
Logstash: The Swiss Army Knife inputs (+codecs) filters (parse, modify)
outputs (+codecs) lots of plugins => lots of options
Logstash: Example Lumberjack Logstash Elasticsearch
Logstash: Add Buffer Lumberjack Lumberjack
Logstash: Scale Everything Lumberjack Lumberjack Lumberjack Lumberjack
Back to the Beginning Lumberjack Lumberjack Lumberjack Lumberjack syslogd
Logsene Lumberjack Lumberjack Lumberjack Lumberjack syslogd Logsene http://sematext.com/logsene
(More) Alternatives files syslog
Alternatives Can Mix files syslog Logstash Elasticsearch Kibana
Thank you! Radu Gheorghe @radu0gheorghe
[email protected]
@sematext
None
rsyslog 1/4: Upgrade to 7.x RPMs or DEBs better performance
nicer config format omelasticsearch
rsyslog 2/4: Faster Inputs UDP increase TimeRequery TCP use imptcp
rsyslog 3/4: Main Message Queue $MainMsgQueueType FixedArray $MainMsgQueueSize 1000000.... ...or
LinkedList or Disk $...DequeueBatchSize 1000 $...WorkerThreads 3
rsyslog 4/4: Action Queue queue.type="linkedlist" queue.size="1000000" bulkmode="on" # ES specific
queue.dequeuebatchsize="1000" queue.workerthreads="3"
Thank you! Radu Gheorghe @radu0gheorghe
[email protected]
@sematext
More Decks by Monitorama
See All by Monitorama
Monitorama PDX 2017 - Ian Bennett
monitorama
1
650
PDX 2017 - Pedro Andrade
monitorama
0
860
PDX 2017 - Roy Rapoport
monitorama
4
1k
PDX 2017 - Julia Evans
monitorama
0
540
Berlin 2013 - Session - Brad Lhotsky
monitorama
5
780
Berlin 2013 - Session - Alex Petrov
monitorama
6
740
Berlin 2013 - Session - Jeff Weinstein
monitorama
2
690
Berlin 2013 - Session - Oliver Hankeln
monitorama
1
610
Berlin 2013 - Session - David Goodlad
monitorama
0
540
Featured
See All Featured
Ruling the World: When Life Gets Gamed
codingconduct
0
450
Build The Right Thing And Hit Your Dates
maggiecrowley
39
3.5k
Tell your own story through comics
letsgokoyo
1
1.1k
The Illustrated Guide to Node.js - THAT Conference 2024
reverentgeek
1
620
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
128
56k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
56
3.5k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
33
5.9k
How to optimise 3,500 product descriptions for ecommerce in one day using ChatGPT
katarinadahlin
PRO
3
3.9k
Into the Great Unknown - MozCon
thekraken
41
2.8k
GraphQLとの向き合い方2022年版
quramy
50
15k
How People are Using Generative and Agentic AI to Supercharge Their Products, Projects, Services and Value Streams Today
helenjbeal
2
400
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
3k