Berlin 2013 - Session - Radu Gheorghe

Transcript

1. On Centralizing Logs
   Radu Gheorghe
   @radu0gheorghe
   radu.gheorghe@sematext.com
   @sematext
   

   View Slide

2. Hello World!
   Logsene
   mlmoneu13cf for -44%
   

   View Slide

3. app
   app
   app
   app
   files
   files
   

   View Slide

4. app
   app
   app
   app
   files
   files
   Elasticsearch
   logstash
   Kibana
   

   View Slide

5. Elasticsearch Reason #1: Quick Search
   No indexing
   But...
   =>
   

   View Slide

6. ...and other reasons
   good write speed lots of tools for logging
   scales easily
   

   View Slide

7. Production Tips
   stability performance
   

   View Slide

8. Stability 1/4: Discovery
   multicast unicast
   vs
   cluster name list of nodes
   + plugins: EC2, GCE
   

   View Slide

9. Stability 2/4: Preventing Split Brain
   minimum_master_nodes = N/2 + 1
   

   View Slide

10. Stability 3/4: No OOMs, pls!
    1GB
    ½ total RAM
    Monitor the requirements
    SPM for Elasticsearch
    20% off with MONEU2013
    

    View Slide

11. Stability 4/4: Field Cache
    can be changed to
    index.cache.field.type: soft
    indices.fielddata.cache.size: X%
    

    View Slide

12. Performance 1/4: Bulk Processing
    use Bulk API
    or Bulk UDP API
    ...translog.flush_threshold_ops
    

    View Slide

13. Performance 2/4: Refresh Interval
    http://blog.sematext.com/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/
    default:
    every second => but
    every 5s
    +25% indexing*
    every 30s
    +70% indexing*
    

    View Slide

14. Performance 3/4: Timed Indices
    

    View Slide

15. Performance 4/4: Buffers
    ...index_buffer_size: 30%
    (YMMV)
    index.store.type: mmapfs
    (on 64-bit machines)
    http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
    

    View Slide

16. Setting Up Kibana as Frontend
    servers you
    

    View Slide

17. Kibana: Search
    

    View Slide

18. Kibana: Visualize
    

    View Slide

19. Meet Some Syslog Daemons
    syslogd
    traditional
    everywhere
    syslog-ng
    OSE, PE
    documentation++
    config format++
    rsyslog
    OSS only
    ES output*
    * http://blog.sematext.com/2013/07/01/recipe-rsyslog-elasticsearch-kibana/
    

    View Slide

20. X-ray of a Modern Syslog Daemon
    read+buffer
    file
    /dev/log
    …
    parse
    syslog formats
    JSON
    unstructured data
    assemble
    conditionals
    formatting
    ...
    buffer+write
    file
    syslog
    Elasticsearch
    ...
    

    View Slide

21. 2001's RFC3164: The Semi-Standard
    <10>Oct 11 22:14:15 host program:hello world
    TCP + LF =
    no year, ms, nor TZ
    little structure
    

    View Slide

22. 2009's RFC5424
    <165>1 2003-10-11T22:14:15.003Z host program - - -
    [origin ip="192.168.0.1"] hello world
    [ structured=data ] octet-count* + LF =
    * UDP (RFC5426), TCP (RFC6587), TLS (RFC5425)
    

    View Slide

23. Teaching Old Dog New Tricks
    RSYSLOG_ForwardFormat
    (ISO8601 over RFC3164)
    $MaxMessageSize 2048k
    log_message_size(2097152)
    @cee: {"message": "hello world"} @@(o)192.168.0.1
    octet-counted framing
    

    View Slide

24. Reliable Transport? Encryption?
    TCP + TLS (RFC5425)
    RLTP + TLS RELP + TLS
    

    View Slide

25. Logstash: The Swiss Army Knife
    inputs
    (+codecs)
    filters
    (parse, modify)
    outputs
    (+codecs)
    lots of plugins => lots of options
    

    View Slide

26. Logstash: Example
    Lumberjack
    Logstash Elasticsearch
    

    View Slide

27. Logstash: Add Buffer
    Lumberjack
    Lumberjack
    

    View Slide

28. Logstash: Scale Everything
    Lumberjack
    Lumberjack
    Lumberjack
    Lumberjack
    

    View Slide

29. Back to the Beginning
    Lumberjack
    Lumberjack
    Lumberjack
    Lumberjack
    syslogd
    

    View Slide

30. Logsene
    Lumberjack
    Lumberjack
    Lumberjack
    Lumberjack
    syslogd
    Logsene
    http://sematext.com/logsene
    

    View Slide

31. (More) Alternatives
    files
    syslog
    

    View Slide

32. Alternatives Can Mix
    files
    syslog
    Logstash
    Elasticsearch Kibana
    

    View Slide

33. Thank you!
    Radu Gheorghe
    @radu0gheorghe
    radu.gheorghe@sematext.com
    @sematext
    

    View Slide

34. View Slide

35. rsyslog 1/4: Upgrade to 7.x
    RPMs or DEBs better performance
    nicer config format omelasticsearch
    

    View Slide

36. rsyslog 2/4: Faster Inputs
    UDP
    increase TimeRequery
    TCP
    use imptcp
    

    View Slide

37. rsyslog 3/4: Main Message Queue
    $MainMsgQueueType FixedArray
    $MainMsgQueueSize 1000000....
    ...or LinkedList or Disk
    $...DequeueBatchSize 1000 $...WorkerThreads 3
    

    View Slide

38. rsyslog 4/4: Action Queue
    queue.type="linkedlist"
    queue.size="1000000"
    bulkmode="on" # ES specific
    queue.dequeuebatchsize="1000"
    queue.workerthreads="3"
    

    View Slide

39. Thank you!
    Radu Gheorghe
    @radu0gheorghe
    radu.gheorghe@sematext.com
    @sematext
    

    View Slide