Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Berlin 2013 - Session - Radu Gheorghe

0580d500edfdb2e5e80e4732ac8df1ea?s=47 Monitorama
September 20, 2013
500

Berlin 2013 - Session - Radu Gheorghe

0580d500edfdb2e5e80e4732ac8df1ea?s=128

Monitorama

September 20, 2013
Tweet

Transcript

  1. On Centralizing Logs Radu Gheorghe @radu0gheorghe radu.gheorghe@sematext.com @sematext

  2. Hello World! Logsene mlmoneu13cf for -44%

  3. app app app app files files

  4. app app app app files files Elasticsearch logstash Kibana

  5. Elasticsearch Reason #1: Quick Search No indexing But... =>

  6. ...and other reasons good write speed lots of tools for

    logging scales easily
  7. Production Tips stability performance

  8. Stability 1/4: Discovery multicast unicast vs cluster name list of

    nodes + plugins: EC2, GCE
  9. Stability 2/4: Preventing Split Brain minimum_master_nodes = N/2 + 1

  10. Stability 3/4: No OOMs, pls! 1GB ½ total RAM Monitor

    the requirements SPM for Elasticsearch 20% off with MONEU2013
  11. Stability 4/4: Field Cache can be changed to index.cache.field.type: soft

    indices.fielddata.cache.size: X%
  12. Performance 1/4: Bulk Processing use Bulk API or Bulk UDP

    API ...translog.flush_threshold_ops
  13. Performance 2/4: Refresh Interval http://blog.sematext.com/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/ default: every second => but

    every 5s +25% indexing* every 30s +70% indexing*
  14. Performance 3/4: Timed Indices

  15. Performance 4/4: Buffers ...index_buffer_size: 30% (YMMV) index.store.type: mmapfs (on 64-bit

    machines) http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
  16. Setting Up Kibana as Frontend servers you

  17. Kibana: Search

  18. Kibana: Visualize

  19. Meet Some Syslog Daemons syslogd traditional everywhere syslog-ng OSE, PE

    documentation++ config format++ rsyslog OSS only ES output* * http://blog.sematext.com/2013/07/01/recipe-rsyslog-elasticsearch-kibana/
  20. X-ray of a Modern Syslog Daemon read+buffer file /dev/log …

    parse syslog formats JSON unstructured data assemble conditionals formatting ... buffer+write file syslog Elasticsearch ...
  21. 2001's RFC3164: The Semi-Standard <10>Oct 11 22:14:15 host program:hello world

    TCP + LF = no year, ms, nor TZ little structure
  22. 2009's RFC5424 <165>1 2003-10-11T22:14:15.003Z host program - - - [origin

    ip="192.168.0.1"] hello world [ structured=data ] octet-count* + LF = * UDP (RFC5426), TCP (RFC6587), TLS (RFC5425)
  23. Teaching Old Dog New Tricks RSYSLOG_ForwardFormat (ISO8601 over RFC3164) $MaxMessageSize

    2048k log_message_size(2097152) @cee: {"message": "hello world"} @@(o)192.168.0.1 octet-counted framing
  24. Reliable Transport? Encryption? TCP + TLS (RFC5425) RLTP + TLS

    RELP + TLS
  25. Logstash: The Swiss Army Knife inputs (+codecs) filters (parse, modify)

    outputs (+codecs) lots of plugins => lots of options
  26. Logstash: Example Lumberjack Logstash Elasticsearch

  27. Logstash: Add Buffer Lumberjack Lumberjack

  28. Logstash: Scale Everything Lumberjack Lumberjack Lumberjack Lumberjack

  29. Back to the Beginning Lumberjack Lumberjack Lumberjack Lumberjack syslogd

  30. Logsene Lumberjack Lumberjack Lumberjack Lumberjack syslogd Logsene http://sematext.com/logsene

  31. (More) Alternatives files syslog

  32. Alternatives Can Mix files syslog Logstash Elasticsearch Kibana

  33. Thank you! Radu Gheorghe @radu0gheorghe radu.gheorghe@sematext.com @sematext

  34. None
  35. rsyslog 1/4: Upgrade to 7.x RPMs or DEBs better performance

    nicer config format omelasticsearch
  36. rsyslog 2/4: Faster Inputs UDP increase TimeRequery TCP use imptcp

  37. rsyslog 3/4: Main Message Queue $MainMsgQueueType FixedArray $MainMsgQueueSize 1000000.... ...or

    LinkedList or Disk $...DequeueBatchSize 1000 $...WorkerThreads 3
  38. rsyslog 4/4: Action Queue queue.type="linkedlist" queue.size="1000000" bulkmode="on" # ES specific

    queue.dequeuebatchsize="1000" queue.workerthreads="3"
  39. Thank you! Radu Gheorghe @radu0gheorghe radu.gheorghe@sematext.com @sematext