Centralized logging

Vorrag von Jan-Piet Mens

Transcript

1. Centralized logging
   Jan-Piet Mens
   November 2012
   @jpmens
   

   View Slide

2. @jpmens: consultant,
   author, architect, part-time
   admin, small-scale fiddler,
   loves LDAP, DNS,
   plain text, and things
   that work.
   

   View Slide

3. some logs
   179.44.34.142 - - [13/Sep/2012:02:32:49 -0400]"GET /
   files/logstash/logstash-1.1.0-monolithic.jar HTTP/1.1"
   200 40923996 "-" "Chef Client/0.10.10"
   11-Nov-2012 09:00:33.604 transfer of 'ww.mens.de/IN'
   from 192.168.1.20#53: Transfer completed: 1 messages, 6
   records, 320 bytes, 0.091 secs (3516 bytes/sec)
   Oct 22 14:54:22 hippo slapd[6829]: conn=1011 op=252 MOD
   attr=krbLastSuccessfulAuth krbExtraData
   

   View Slide

4. what time is it?
   1304060505 29/Apr/2011:07:05:26 +0000
   Fri, 21 Nov 1997 09:55:06 -0600
   Oct 11 20:21:47 020805 13:51:24
   110429.071055,118
   @4000000037c219bf2ef02e94
   

   View Slide

5. Tools
   • logstash
   • graylog2
   • elasticsearch
   • lots more ...
   

   View Slide

6. it's really easy...
   

   View Slide

7. input: file
   filter: grok
   output: elasticsearch
   logstash
   

   View Slide

8. configuration
   input {
   file {
   type => "apache"
   path => [ "/var/log/apache.log" ]
   }
   }
   filter {
   grok {
   type => "apache"
   pattern => "%{COMBINEDAPACHELOG}"
   }
   }
   output {
   elasticsearch { host => 'localhost' }
   }
   

   View Slide

9. logstash inputs
   amqp, eventlog, exec, file, ganglia, gelf, gemfire,
   generator, heroku, irc, log4j, lumberjack, pipe,
   redis, relp, sqs, stdin, stomp, syslog, tcp, twitter,
   udp, xmpp, zenoss, zeromq
   

   View Slide

10. logstash filters
    alter, checksum, csv, date, dns, environment, gelfify, geoip,
    grep, grok, grokdiscovery, json, kv, multiline, mutate, noop,
    split, syslog_pri, urldecode, xml, zeromq
    

    View Slide

11. logstash outputs
    amqp, boundary, circonus, datadog, elasticsearch,
    elasticsearch_http, elasticsearch_river, email,
    exec, file, ganglia, gelf, gemfire, graphite,
    graphtastic, http, internal, irc, juggernaut, librato,
    loggly, lumberjack, metriccatcher, mongodb, nagios,
    nagios_nsca, null, opentsdb, pagerduty, pipe, redis,
    riak, riemann, sns, sqs, statsd, stdout, stomp,
    tcp, websocket, xmpp, zabbix, zeromq
    

    View Slide

12. grokking grok
    

    View Slide

13. scaling logstash
    • redis
    • zeromq
    • amqp
    • irc
    • xmpp
    

    View Slide

14. storage
    • based on Lucene
    • schema-free (JSON)
    • elasticsearch scales horizontally
    

    View Slide

15. Kibana
    

    View Slide

16. Graylog2
    

    View Slide

17. Graylog2
    

    View Slide

18. log shipping
    • syslog-ng
    • rsyslog
    • Redis / 0mq
    • lumberjack
    • Beaver
    • ...
    

    View Slide

19. Graphite
    

    View Slide

20. credits
    @jordansissel
    http://semicomplete.com/presentations/logstash-metrics-sf-2012.10/
    

    View Slide