Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Centralized logging

08734031c152fb64f70dc38a4f632cee?s=47 fraosug
March 17, 2013

Centralized logging

Vorrag von Jan-Piet Mens

08734031c152fb64f70dc38a4f632cee?s=128

fraosug

March 17, 2013
Tweet

Transcript

  1. Centralized logging Jan-Piet Mens November 2012 @jpmens

  2. @jpmens: consultant, author, architect, part-time admin, small-scale fiddler, loves LDAP,

    DNS, plain text, and things that work.
  3. some logs 179.44.34.142 - - [13/Sep/2012:02:32:49 -0400]"GET / files/logstash/logstash-1.1.0-monolithic.jar HTTP/1.1"

    200 40923996 "-" "Chef Client/0.10.10" 11-Nov-2012 09:00:33.604 transfer of 'ww.mens.de/IN' from 192.168.1.20#53: Transfer completed: 1 messages, 6 records, 320 bytes, 0.091 secs (3516 bytes/sec) Oct 22 14:54:22 hippo slapd[6829]: conn=1011 op=252 MOD attr=krbLastSuccessfulAuth krbExtraData
  4. what time is it? 1304060505 29/Apr/2011:07:05:26 +0000 Fri, 21 Nov

    1997 09:55:06 -0600 Oct 11 20:21:47 020805 13:51:24 110429.071055,118 @4000000037c219bf2ef02e94
  5. Tools • logstash • graylog2 • elasticsearch • lots more

    ...
  6. it's really easy...

  7. input: file filter: grok output: elasticsearch logstash

  8. configuration input { file { type => "apache" path =>

    [ "/var/log/apache.log" ] } } filter { grok { type => "apache" pattern => "%{COMBINEDAPACHELOG}" } } output { elasticsearch { host => 'localhost' } }
  9. logstash inputs amqp, eventlog, exec, file, ganglia, gelf, gemfire, generator,

    heroku, irc, log4j, lumberjack, pipe, redis, relp, sqs, stdin, stomp, syslog, tcp, twitter, udp, xmpp, zenoss, zeromq
  10. logstash filters alter, checksum, csv, date, dns, environment, gelfify, geoip,

    grep, grok, grokdiscovery, json, kv, multiline, mutate, noop, split, syslog_pri, urldecode, xml, zeromq
  11. logstash outputs amqp, boundary, circonus, datadog, elasticsearch, elasticsearch_http, elasticsearch_river, email,

    exec, file, ganglia, gelf, gemfire, graphite, graphtastic, http, internal, irc, juggernaut, librato, loggly, lumberjack, metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty, pipe, redis, riak, riemann, sns, sqs, statsd, stdout, stomp, tcp, websocket, xmpp, zabbix, zeromq
  12. grokking grok

  13. scaling logstash • redis • zeromq • amqp • irc

    • xmpp
  14. storage • based on Lucene • schema-free (JSON) • elasticsearch

    scales horizontally
  15. Kibana

  16. Graylog2

  17. Graylog2

  18. log shipping • syslog-ng • rsyslog • Redis / 0mq

    • lumberjack • Beaver • ...
  19. Graphite

  20. credits @jordansissel http://semicomplete.com/presentations/logstash-metrics-sf-2012.10/