Berlin 2013 - Session - Radu Gheorghe

Transcript

1. On Centralizing Logs Radu Gheorghe @radu0gheorghe radu.gheorghe@sematext.com @sematext
2. Hello World! Logsene mlmoneu13cf for -44%
3. app app app app files files
4. app app app app files files Elasticsearch logstash Kibana
5. Elasticsearch Reason #1: Quick Search No indexing But... =>
6. ...and other reasons good write speed lots of tools for logging scales easily
7. Production Tips stability performance
8. Stability 1/4: Discovery multicast unicast vs cluster name list of nodes + plugins: EC2, GCE
9. Stability 2/4: Preventing Split Brain minimum_master_nodes = N/2 + 1
10. Stability 3/4: No OOMs, pls! 1GB ½ total RAM Monitor the requirements SPM for Elasticsearch 20% off with MONEU2013
11. Stability 4/4: Field Cache can be changed to index.cache.field.type: soft indices.fielddata.cache.size: X%
12. Performance 1/4: Bulk Processing use Bulk API or Bulk UDP API ...translog.flush_threshold_ops
13. Performance 2/4: Refresh Interval http://blog.sematext.com/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/ default: every second => but every 5s +25% indexing* every 30s +70% indexing*
14. Performance 3/4: Timed Indices
15. Performance 4/4: Buffers ...index_buffer_size: 30% (YMMV) index.store.type: mmapfs (on 64-bit machines) http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
16. Setting Up Kibana as Frontend servers you
17. Kibana: Search
18. Kibana: Visualize
19. Meet Some Syslog Daemons syslogd traditional everywhere syslog-ng OSE, PE documentation++ config format++ rsyslog OSS only ES output* * http://blog.sematext.com/2013/07/01/recipe-rsyslog-elasticsearch-kibana/
20. X-ray of a Modern Syslog Daemon read+buffer file /dev/log … parse syslog formats JSON unstructured data assemble conditionals formatting ... buffer+write file syslog Elasticsearch ...
21. 2001's RFC3164: The Semi-Standard <10>Oct 11 22:14:15 host program:hello world TCP + LF = no year, ms, nor TZ little structure
22. 2009's RFC5424 <165>1 2003-10-11T22:14:15.003Z host program - - - [origin ip="192.168.0.1"] hello world [ structured=data ] octet-count* + LF = * UDP (RFC5426), TCP (RFC6587), TLS (RFC5425)
23. Teaching Old Dog New Tricks RSYSLOG_ForwardFormat (ISO8601 over RFC3164) $MaxMessageSize 2048k log_message_size(2097152) @cee: {"message": "hello world"} @@(o)192.168.0.1 octet-counted framing
24. Reliable Transport? Encryption? TCP + TLS (RFC5425) RLTP + TLS RELP + TLS
25. Logstash: The Swiss Army Knife inputs (+codecs) filters (parse, modify) outputs (+codecs) lots of plugins => lots of options
26. Logstash: Example Lumberjack Logstash Elasticsearch
27. Logstash: Add Buffer Lumberjack Lumberjack
28. Logstash: Scale Everything Lumberjack Lumberjack Lumberjack Lumberjack
29. Back to the Beginning Lumberjack Lumberjack Lumberjack Lumberjack syslogd
30. Logsene Lumberjack Lumberjack Lumberjack Lumberjack syslogd Logsene http://sematext.com/logsene
31. (More) Alternatives files syslog
32. Alternatives Can Mix files syslog Logstash Elasticsearch Kibana
33. Thank you! Radu Gheorghe @radu0gheorghe radu.gheorghe@sematext.com @sematext
35. rsyslog 1/4: Upgrade to 7.x RPMs or DEBs better performance nicer config format omelasticsearch
36. rsyslog 2/4: Faster Inputs UDP increase TimeRequery TCP use imptcp
37. rsyslog 3/4: Main Message Queue $MainMsgQueueType FixedArray $MainMsgQueueSize 1000000.... ...or LinkedList or Disk $...DequeueBatchSize 1000 $...WorkerThreads 3
38. rsyslog 4/4: Action Queue queue.type="linkedlist" queue.size="1000000" bulkmode="on" # ES specific queue.dequeuebatchsize="1000" queue.workerthreads="3"
39. Thank you! Radu Gheorghe @radu0gheorghe radu.gheorghe@sematext.com @sematext