Centralized logging

Vorrag von Jan-Piet Mens

Transcript

1. Centralized logging Jan-Piet Mens November 2012 @jpmens
2. @jpmens: consultant, author, architect, part-time admin, small-scale fiddler, loves LDAP, DNS, plain text, and things that work.
3. some logs 179.44.34.142 - - [13/Sep/2012:02:32:49 -0400]"GET / files/logstash/logstash-1.1.0-monolithic.jar HTTP/1.1" 200 40923996 "-" "Chef Client/0.10.10" 11-Nov-2012 09:00:33.604 transfer of 'ww.mens.de/IN' from 192.168.1.20#53: Transfer completed: 1 messages, 6 records, 320 bytes, 0.091 secs (3516 bytes/sec) Oct 22 14:54:22 hippo slapd[6829]: conn=1011 op=252 MOD attr=krbLastSuccessfulAuth krbExtraData
4. what time is it? 1304060505 29/Apr/2011:07:05:26 +0000 Fri, 21 Nov 1997 09:55:06 -0600 Oct 11 20:21:47 020805 13:51:24 110429.071055,118 @4000000037c219bf2ef02e94
5. Tools • logstash • graylog2 • elasticsearch • lots more ...
6. it's really easy...
7. input: file filter: grok output: elasticsearch logstash
8. configuration input { file { type => "apache" path => [ "/var/log/apache.log" ] } } filter { grok { type => "apache" pattern => "%{COMBINEDAPACHELOG}" } } output { elasticsearch { host => 'localhost' } }
9. logstash inputs amqp, eventlog, exec, file, ganglia, gelf, gemfire, generator, heroku, irc, log4j, lumberjack, pipe, redis, relp, sqs, stdin, stomp, syslog, tcp, twitter, udp, xmpp, zenoss, zeromq
10. logstash filters alter, checksum, csv, date, dns, environment, gelfify, geoip, grep, grok, grokdiscovery, json, kv, multiline, mutate, noop, split, syslog_pri, urldecode, xml, zeromq
11. logstash outputs amqp, boundary, circonus, datadog, elasticsearch, elasticsearch_http, elasticsearch_river, email, exec, file, ganglia, gelf, gemfire, graphite, graphtastic, http, internal, irc, juggernaut, librato, loggly, lumberjack, metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty, pipe, redis, riak, riemann, sns, sqs, statsd, stdout, stomp, tcp, websocket, xmpp, zabbix, zeromq
12. grokking grok
13. scaling logstash • redis • zeromq • amqp • irc • xmpp
14. storage • based on Lucene • schema-free (JSON) • elasticsearch scales horizontally
15. Kibana
16. Graylog2
17. Graylog2
18. log shipping • syslog-ng • rsyslog • Redis / 0mq • lumberjack • Beaver • ...
19. Graphite
20. credits @jordansissel http://semicomplete.com/presentations/logstash-metrics-sf-2012.10/