Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up
for free
Berlin 2013 - Session - Radu Gheorghe
Monitorama
September 20, 2013
0
480
Berlin 2013 - Session - Radu Gheorghe
Monitorama
September 20, 2013
Tweet
Share
More Decks by Monitorama
See All by Monitorama
monitorama
1
340
monitorama
0
260
monitorama
4
550
monitorama
0
210
monitorama
5
600
monitorama
6
570
monitorama
2
500
monitorama
1
430
monitorama
0
310
Featured
See All Featured
malarkey
119
16k
andyhume
62
3.4k
davidbonilla
69
3.5k
nonsquared
81
3.3k
jponch
103
5k
hannesfritz
27
930
jrom
114
7.1k
stephaniewalter
260
11k
vanstee
116
4.8k
garrettdimon
287
110k
philhawksworth
190
17k
jonrohan
1021
380k
Transcript
On Centralizing Logs Radu Gheorghe @radu0gheorghe radu.gheorghe@sematext.com @sematext
Hello World! Logsene mlmoneu13cf for -44%
app app app app files files
app app app app files files Elasticsearch logstash Kibana
Elasticsearch Reason #1: Quick Search No indexing But... =>
...and other reasons good write speed lots of tools for
logging scales easily
Production Tips stability performance
Stability 1/4: Discovery multicast unicast vs cluster name list of
nodes + plugins: EC2, GCE
Stability 2/4: Preventing Split Brain minimum_master_nodes = N/2 + 1
Stability 3/4: No OOMs, pls! 1GB ½ total RAM Monitor
the requirements SPM for Elasticsearch 20% off with MONEU2013
Stability 4/4: Field Cache can be changed to index.cache.field.type: soft
indices.fielddata.cache.size: X%
Performance 1/4: Bulk Processing use Bulk API or Bulk UDP
API ...translog.flush_threshold_ops
Performance 2/4: Refresh Interval http://blog.sematext.com/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/ default: every second => but
every 5s +25% indexing* every 30s +70% indexing*
Performance 3/4: Timed Indices
Performance 4/4: Buffers ...index_buffer_size: 30% (YMMV) index.store.type: mmapfs (on 64-bit
machines) http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
Setting Up Kibana as Frontend servers you
Kibana: Search
Kibana: Visualize
Meet Some Syslog Daemons syslogd traditional everywhere syslog-ng OSE, PE
documentation++ config format++ rsyslog OSS only ES output* * http://blog.sematext.com/2013/07/01/recipe-rsyslog-elasticsearch-kibana/
X-ray of a Modern Syslog Daemon read+buffer file /dev/log …
parse syslog formats JSON unstructured data assemble conditionals formatting ... buffer+write file syslog Elasticsearch ...
2001's RFC3164: The Semi-Standard <10>Oct 11 22:14:15 host program:hello world
TCP + LF = no year, ms, nor TZ little structure
2009's RFC5424 <165>1 2003-10-11T22:14:15.003Z host program - - - [origin
ip="192.168.0.1"] hello world [ structured=data ] octet-count* + LF = * UDP (RFC5426), TCP (RFC6587), TLS (RFC5425)
Teaching Old Dog New Tricks RSYSLOG_ForwardFormat (ISO8601 over RFC3164) $MaxMessageSize
2048k log_message_size(2097152) @cee: {"message": "hello world"} @@(o)192.168.0.1 octet-counted framing
Reliable Transport? Encryption? TCP + TLS (RFC5425) RLTP + TLS
RELP + TLS
Logstash: The Swiss Army Knife inputs (+codecs) filters (parse, modify)
outputs (+codecs) lots of plugins => lots of options
Logstash: Example Lumberjack Logstash Elasticsearch
Logstash: Add Buffer Lumberjack Lumberjack
Logstash: Scale Everything Lumberjack Lumberjack Lumberjack Lumberjack
Back to the Beginning Lumberjack Lumberjack Lumberjack Lumberjack syslogd
Logsene Lumberjack Lumberjack Lumberjack Lumberjack syslogd Logsene http://sematext.com/logsene
(More) Alternatives files syslog
Alternatives Can Mix files syslog Logstash Elasticsearch Kibana
Thank you! Radu Gheorghe @radu0gheorghe radu.gheorghe@sematext.com @sematext
None
rsyslog 1/4: Upgrade to 7.x RPMs or DEBs better performance
nicer config format omelasticsearch
rsyslog 2/4: Faster Inputs UDP increase TimeRequery TCP use imptcp
rsyslog 3/4: Main Message Queue $MainMsgQueueType FixedArray $MainMsgQueueSize 1000000.... ...or
LinkedList or Disk $...DequeueBatchSize 1000 $...WorkerThreads 3
rsyslog 4/4: Action Queue queue.type="linkedlist" queue.size="1000000" bulkmode="on" # ES specific
queue.dequeuebatchsize="1000" queue.workerthreads="3"
Thank you! Radu Gheorghe @radu0gheorghe radu.gheorghe@sematext.com @sematext