Berlin 2013 - Session - Radu Gheorghe

Transcript

1. On Centralizing Logs Radu Gheorghe @radu0gheorghe radu.gheorghe@sematext.com @sematext

2. Hello World! Logsene mlmoneu13cf for -44%

3. app app app app files files

4. app app app app files files Elasticsearch logstash Kibana

5. Elasticsearch Reason #1: Quick Search No indexing But... =>

6. ...and other reasons good write speed lots of tools for

   logging scales easily

7. Production Tips stability performance

8. Stability 1/4: Discovery multicast unicast vs cluster name list of

   nodes + plugins: EC2, GCE

9. Stability 2/4: Preventing Split Brain minimum_master_nodes = N/2 + 1

10. Stability 3/4: No OOMs, pls! 1GB ½ total RAM Monitor

    the requirements SPM for Elasticsearch 20% off with MONEU2013

11. Stability 4/4: Field Cache can be changed to index.cache.field.type: soft

    indices.fielddata.cache.size: X%

12. Performance 1/4: Bulk Processing use Bulk API or Bulk UDP

    API ...translog.flush_threshold_ops

13. Performance 2/4: Refresh Interval http://blog.sematext.com/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/ default: every second => but

    every 5s +25% indexing* every 30s +70% indexing*

14. Performance 3/4: Timed Indices

15. Performance 4/4: Buffers ...index_buffer_size: 30% (YMMV) index.store.type: mmapfs (on 64-bit

    machines) http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html

16. Setting Up Kibana as Frontend servers you

17. Kibana: Search

18. Kibana: Visualize

19. Meet Some Syslog Daemons syslogd traditional everywhere syslog-ng OSE, PE

    documentation++ config format++ rsyslog OSS only ES output* * http://blog.sematext.com/2013/07/01/recipe-rsyslog-elasticsearch-kibana/

20. X-ray of a Modern Syslog Daemon read+buffer file /dev/log …

    parse syslog formats JSON unstructured data assemble conditionals formatting ... buffer+write file syslog Elasticsearch ...

21. 2001's RFC3164: The Semi-Standard <10>Oct 11 22:14:15 host program:hello world

    TCP + LF = no year, ms, nor TZ little structure

22. 2009's RFC5424 <165>1 2003-10-11T22:14:15.003Z host program - - - [origin

    ip="192.168.0.1"] hello world [ structured=data ] octet-count* + LF = * UDP (RFC5426), TCP (RFC6587), TLS (RFC5425)

23. Teaching Old Dog New Tricks RSYSLOG_ForwardFormat (ISO8601 over RFC3164) $MaxMessageSize

    2048k log_message_size(2097152) @cee: {"message": "hello world"} @@(o)192.168.0.1 octet-counted framing

24. Reliable Transport? Encryption? TCP + TLS (RFC5425) RLTP + TLS

    RELP + TLS

25. Logstash: The Swiss Army Knife inputs (+codecs) filters (parse, modify)

    outputs (+codecs) lots of plugins => lots of options

26. Logstash: Example Lumberjack Logstash Elasticsearch

27. Logstash: Add Buffer Lumberjack Lumberjack

28. Logstash: Scale Everything Lumberjack Lumberjack Lumberjack Lumberjack

29. Back to the Beginning Lumberjack Lumberjack Lumberjack Lumberjack syslogd

30. Logsene Lumberjack Lumberjack Lumberjack Lumberjack syslogd Logsene http://sematext.com/logsene

31. (More) Alternatives files syslog

32. Alternatives Can Mix files syslog Logstash Elasticsearch Kibana

33. Thank you! Radu Gheorghe @radu0gheorghe radu.gheorghe@sematext.com @sematext

34. None

35. rsyslog 1/4: Upgrade to 7.x RPMs or DEBs better performance

    nicer config format omelasticsearch

36. rsyslog 2/4: Faster Inputs UDP increase TimeRequery TCP use imptcp

37. rsyslog 3/4: Main Message Queue $MainMsgQueueType FixedArray $MainMsgQueueSize 1000000.... ...or

    LinkedList or Disk $...DequeueBatchSize 1000 $...WorkerThreads 3

38. rsyslog 4/4: Action Queue queue.type="linkedlist" queue.size="1000000" bulkmode="on" # ES specific

    queue.dequeuebatchsize="1000" queue.workerthreads="3"

39. Thank you! Radu Gheorghe @radu0gheorghe radu.gheorghe@sematext.com @sematext