Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Berlin 2013 - Session - Radu Gheorghe

Monitorama
September 20, 2013
560

Berlin 2013 - Session - Radu Gheorghe

Monitorama

September 20, 2013
Tweet

Transcript

  1. Stability 3/4: No OOMs, pls! 1GB ½ total RAM Monitor

    the requirements SPM for Elasticsearch 20% off with MONEU2013
  2. Performance 1/4: Bulk Processing use Bulk API or Bulk UDP

    API ...translog.flush_threshold_ops
  3. Performance 4/4: Buffers ...index_buffer_size: 30% (YMMV) index.store.type: mmapfs (on 64-bit

    machines) http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
  4. Meet Some Syslog Daemons syslogd traditional everywhere syslog-ng OSE, PE

    documentation++ config format++ rsyslog OSS only ES output* * http://blog.sematext.com/2013/07/01/recipe-rsyslog-elasticsearch-kibana/
  5. X-ray of a Modern Syslog Daemon read+buffer file /dev/log …

    parse syslog formats JSON unstructured data assemble conditionals formatting ... buffer+write file syslog Elasticsearch ...
  6. 2009's RFC5424 <165>1 2003-10-11T22:14:15.003Z host program - - - [origin

    ip="192.168.0.1"] hello world [ structured=data ] octet-count* + LF = * UDP (RFC5426), TCP (RFC6587), TLS (RFC5425)
  7. Teaching Old Dog New Tricks RSYSLOG_ForwardFormat (ISO8601 over RFC3164) $MaxMessageSize

    2048k log_message_size(2097152) @cee: {"message": "hello world"} @@(o)192.168.0.1 octet-counted framing
  8. Logstash: The Swiss Army Knife inputs (+codecs) filters (parse, modify)

    outputs (+codecs) lots of plugins => lots of options
  9. rsyslog 3/4: Main Message Queue $MainMsgQueueType FixedArray $MainMsgQueueSize 1000000.... ...or

    LinkedList or Disk $...DequeueBatchSize 1000 $...WorkerThreads 3