Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Berlin 2013 - Session - Radu Gheorghe
Search
Monitorama
September 20, 2013
0
620
Berlin 2013 - Session - Radu Gheorghe
Monitorama
September 20, 2013
Tweet
Share
More Decks by Monitorama
See All by Monitorama
Monitorama PDX 2017 - Ian Bennett
monitorama
1
580
PDX 2017 - Pedro Andrade
monitorama
0
730
PDX 2017 - Roy Rapoport
monitorama
4
940
PDX 2017 - Julia Evans
monitorama
0
470
Berlin 2013 - Session - Brad Lhotsky
monitorama
5
710
Berlin 2013 - Session - Alex Petrov
monitorama
6
680
Berlin 2013 - Session - Jeff Weinstein
monitorama
2
620
Berlin 2013 - Session - Oliver Hankeln
monitorama
1
540
Berlin 2013 - Session - David Goodlad
monitorama
0
450
Featured
See All Featured
What's in a price? How to price your products and services
michaelherold
246
12k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
32
2.4k
Build The Right Thing And Hit Your Dates
maggiecrowley
36
2.8k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
29
9.5k
The Language of Interfaces
destraynor
158
25k
Site-Speed That Sticks
csswizardry
10
670
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
229
22k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
20
1.3k
Java REST API Framework Comparison - PWX 2021
mraible
31
8.7k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
130
19k
Writing Fast Ruby
sferik
628
62k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
31
1.3k
Transcript
On Centralizing Logs Radu Gheorghe @radu0gheorghe
[email protected]
@sematext
Hello World! Logsene mlmoneu13cf for -44%
app app app app files files
app app app app files files Elasticsearch logstash Kibana
Elasticsearch Reason #1: Quick Search No indexing But... =>
...and other reasons good write speed lots of tools for
logging scales easily
Production Tips stability performance
Stability 1/4: Discovery multicast unicast vs cluster name list of
nodes + plugins: EC2, GCE
Stability 2/4: Preventing Split Brain minimum_master_nodes = N/2 + 1
Stability 3/4: No OOMs, pls! 1GB ½ total RAM Monitor
the requirements SPM for Elasticsearch 20% off with MONEU2013
Stability 4/4: Field Cache can be changed to index.cache.field.type: soft
indices.fielddata.cache.size: X%
Performance 1/4: Bulk Processing use Bulk API or Bulk UDP
API ...translog.flush_threshold_ops
Performance 2/4: Refresh Interval http://blog.sematext.com/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/ default: every second => but
every 5s +25% indexing* every 30s +70% indexing*
Performance 3/4: Timed Indices
Performance 4/4: Buffers ...index_buffer_size: 30% (YMMV) index.store.type: mmapfs (on 64-bit
machines) http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
Setting Up Kibana as Frontend servers you
Kibana: Search
Kibana: Visualize
Meet Some Syslog Daemons syslogd traditional everywhere syslog-ng OSE, PE
documentation++ config format++ rsyslog OSS only ES output* * http://blog.sematext.com/2013/07/01/recipe-rsyslog-elasticsearch-kibana/
X-ray of a Modern Syslog Daemon read+buffer file /dev/log …
parse syslog formats JSON unstructured data assemble conditionals formatting ... buffer+write file syslog Elasticsearch ...
2001's RFC3164: The Semi-Standard <10>Oct 11 22:14:15 host program:hello world
TCP + LF = no year, ms, nor TZ little structure
2009's RFC5424 <165>1 2003-10-11T22:14:15.003Z host program - - - [origin
ip="192.168.0.1"] hello world [ structured=data ] octet-count* + LF = * UDP (RFC5426), TCP (RFC6587), TLS (RFC5425)
Teaching Old Dog New Tricks RSYSLOG_ForwardFormat (ISO8601 over RFC3164) $MaxMessageSize
2048k log_message_size(2097152) @cee: {"message": "hello world"} @@(o)192.168.0.1 octet-counted framing
Reliable Transport? Encryption? TCP + TLS (RFC5425) RLTP + TLS
RELP + TLS
Logstash: The Swiss Army Knife inputs (+codecs) filters (parse, modify)
outputs (+codecs) lots of plugins => lots of options
Logstash: Example Lumberjack Logstash Elasticsearch
Logstash: Add Buffer Lumberjack Lumberjack
Logstash: Scale Everything Lumberjack Lumberjack Lumberjack Lumberjack
Back to the Beginning Lumberjack Lumberjack Lumberjack Lumberjack syslogd
Logsene Lumberjack Lumberjack Lumberjack Lumberjack syslogd Logsene http://sematext.com/logsene
(More) Alternatives files syslog
Alternatives Can Mix files syslog Logstash Elasticsearch Kibana
Thank you! Radu Gheorghe @radu0gheorghe
[email protected]
@sematext
None
rsyslog 1/4: Upgrade to 7.x RPMs or DEBs better performance
nicer config format omelasticsearch
rsyslog 2/4: Faster Inputs UDP increase TimeRequery TCP use imptcp
rsyslog 3/4: Main Message Queue $MainMsgQueueType FixedArray $MainMsgQueueSize 1000000.... ...or
LinkedList or Disk $...DequeueBatchSize 1000 $...WorkerThreads 3
rsyslog 4/4: Action Queue queue.type="linkedlist" queue.size="1000000" bulkmode="on" # ES specific
queue.dequeuebatchsize="1000" queue.workerthreads="3"
Thank you! Radu Gheorghe @radu0gheorghe
[email protected]
@sematext