$30 off During Our Annual Pro Sale. View Details »

Berlin 2013 - Session - Radu Gheorghe

Monitorama
September 20, 2013
540

Berlin 2013 - Session - Radu Gheorghe

Monitorama

September 20, 2013
Tweet

Transcript

  1. On Centralizing Logs
    Radu Gheorghe
    @radu0gheorghe
    [email protected]
    @sematext

    View Slide

  2. Hello World!
    Logsene
    mlmoneu13cf for -44%

    View Slide

  3. app
    app
    app
    app
    files
    files

    View Slide

  4. app
    app
    app
    app
    files
    files
    Elasticsearch
    logstash
    Kibana

    View Slide

  5. Elasticsearch Reason #1: Quick Search
    No indexing
    But...
    =>

    View Slide

  6. ...and other reasons
    good write speed lots of tools for logging
    scales easily

    View Slide

  7. Production Tips
    stability performance

    View Slide

  8. Stability 1/4: Discovery
    multicast unicast
    vs
    cluster name list of nodes
    + plugins: EC2, GCE

    View Slide

  9. Stability 2/4: Preventing Split Brain
    minimum_master_nodes = N/2 + 1

    View Slide

  10. Stability 3/4: No OOMs, pls!
    1GB
    ½ total RAM
    Monitor the requirements
    SPM for Elasticsearch
    20% off with MONEU2013

    View Slide

  11. Stability 4/4: Field Cache
    can be changed to
    index.cache.field.type: soft
    indices.fielddata.cache.size: X%

    View Slide

  12. Performance 1/4: Bulk Processing
    use Bulk API
    or Bulk UDP API
    ...translog.flush_threshold_ops

    View Slide

  13. Performance 2/4: Refresh Interval
    http://blog.sematext.com/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/
    default:
    every second => but
    every 5s
    +25% indexing*
    every 30s
    +70% indexing*

    View Slide

  14. Performance 3/4: Timed Indices

    View Slide

  15. Performance 4/4: Buffers
    ...index_buffer_size: 30%
    (YMMV)
    index.store.type: mmapfs
    (on 64-bit machines)
    http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html

    View Slide

  16. Setting Up Kibana as Frontend
    servers you

    View Slide

  17. Kibana: Search

    View Slide

  18. Kibana: Visualize

    View Slide

  19. Meet Some Syslog Daemons
    syslogd
    traditional
    everywhere
    syslog-ng
    OSE, PE
    documentation++
    config format++
    rsyslog
    OSS only
    ES output*
    * http://blog.sematext.com/2013/07/01/recipe-rsyslog-elasticsearch-kibana/

    View Slide

  20. X-ray of a Modern Syslog Daemon
    read+buffer
    file
    /dev/log

    parse
    syslog formats
    JSON
    unstructured data
    assemble
    conditionals
    formatting
    ...
    buffer+write
    file
    syslog
    Elasticsearch
    ...

    View Slide

  21. 2001's RFC3164: The Semi-Standard
    <10>Oct 11 22:14:15 host program:hello world
    TCP + LF =
    no year, ms, nor TZ
    little structure

    View Slide

  22. 2009's RFC5424
    <165>1 2003-10-11T22:14:15.003Z host program - - -
    [origin ip="192.168.0.1"] hello world
    [ structured=data ] octet-count* + LF =
    * UDP (RFC5426), TCP (RFC6587), TLS (RFC5425)

    View Slide

  23. Teaching Old Dog New Tricks
    RSYSLOG_ForwardFormat
    (ISO8601 over RFC3164)
    $MaxMessageSize 2048k
    log_message_size(2097152)
    @cee: {"message": "hello world"} @@(o)192.168.0.1
    octet-counted framing

    View Slide

  24. Reliable Transport? Encryption?
    TCP + TLS (RFC5425)
    RLTP + TLS RELP + TLS

    View Slide

  25. Logstash: The Swiss Army Knife
    inputs
    (+codecs)
    filters
    (parse, modify)
    outputs
    (+codecs)
    lots of plugins => lots of options

    View Slide

  26. Logstash: Example
    Lumberjack
    Logstash Elasticsearch

    View Slide

  27. Logstash: Add Buffer
    Lumberjack
    Lumberjack

    View Slide

  28. Logstash: Scale Everything
    Lumberjack
    Lumberjack
    Lumberjack
    Lumberjack

    View Slide

  29. Back to the Beginning
    Lumberjack
    Lumberjack
    Lumberjack
    Lumberjack
    syslogd

    View Slide

  30. Logsene
    Lumberjack
    Lumberjack
    Lumberjack
    Lumberjack
    syslogd
    Logsene
    http://sematext.com/logsene

    View Slide

  31. (More) Alternatives
    files
    syslog

    View Slide

  32. Alternatives Can Mix
    files
    syslog
    Logstash
    Elasticsearch Kibana

    View Slide

  33. Thank you!
    Radu Gheorghe
    @radu0gheorghe
    [email protected]
    @sematext

    View Slide

  34. View Slide

  35. rsyslog 1/4: Upgrade to 7.x
    RPMs or DEBs better performance
    nicer config format omelasticsearch

    View Slide

  36. rsyslog 2/4: Faster Inputs
    UDP
    increase TimeRequery
    TCP
    use imptcp

    View Slide

  37. rsyslog 3/4: Main Message Queue
    $MainMsgQueueType FixedArray
    $MainMsgQueueSize 1000000....
    ...or LinkedList or Disk
    $...DequeueBatchSize 1000 $...WorkerThreads 3

    View Slide

  38. rsyslog 4/4: Action Queue
    queue.type="linkedlist"
    queue.size="1000000"
    bulkmode="on" # ES specific
    queue.dequeuebatchsize="1000"
    queue.workerthreads="3"

    View Slide

  39. Thank you!
    Radu Gheorghe
    @radu0gheorghe
    [email protected]
    @sematext

    View Slide