The Truth is in the Logs (de)

my Slides from the #osdc 2013 in Nuremberg about Logfile Management and Processing

Video from the presentation can be found on Youtube: http://youtu.be/vLdJJ14qezc

Transcript

1. The Truth is in the Logs Nuremberg, 2013-04-17
2. 2  Capgemini Outsourcing Services • Infrastructur Engineer  wer-kennt-wen.de GmbH • System Operator Linux  1blu AG • System Engineer (Shared Hosting) • Teamlead System Operations  HostEurope GmbH • Teamlead Shared Hosting Operations • Postmaster Jan Doberstein #OSS #bash #vi
3. 3 Eine Logdatei … enthält das automatisch geführte Protokoll aller oder bestimmter Aktionen von Prozessen auf einem Computersystem … Wichtige Anwendungen finden sich vor allem bei der Prozesskontrolle und Automatisierung. Prinzipiell werden alle Aktionen mitgeschrieben, die für eine spätere Untersuchung erforderlich sind oder sein könnten. https://de.wikipedia.org/wiki/Logdatei
4. 4 10.41.148.32 - - [18/Mar/2013:12:43:43 +0100] "GET /health/currentthroughput HTTP/1.1" 200 21 "http://10.43.248.109/users/new" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.172 Safari/537.22" 10.41.148.32 - - [18/Mar/2013:13:06:17 +0100] "GET /visuals/fetch/streamgraph?hours=2&stream_id=514 6bccd772ae72c95000006 HTTP/1.1" 200 2162 "http://10.43.248.109/streams/5146bccd772ae72c95 000006-user/messages" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.172 Safari/537.22" 10.41.148.32 - - [18/Mar/2013:13:06:22 +0100] "GET /messages HTTP/1.1" 200 57786 "http://10.43.248.109/streams/5146bccd772ae72c95 000006-user/messages" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.172 Safari/537.22" Mar 19 09:30:01 sv004972 /usr/sbin/cron[7657]: (root) CMD (/root/bin/http_connection >> /tmp/http_connection.log) Mar 19 09:30:02 sv004972 sshd[7761]: Accepted publickey for root from 192.168.97.41 port 37209 ssh2 Mar 19 09:31:01 sv004972 /usr/sbin/cron[8098]: (/root/bin/http_connection >> /tmp/http_connection.log) Mar 19 09:30:02 sv004972 sshd[7761]: Accepted publickey for root from 192.168.97.41 port 37209 ssh2 Mar 19 09:30:03 sv004972 sshd[7795]: Accepted publickey for root from 192.168.97.41 port 37212 ssh2 Mar 19 09:31:01 sv004972 /usr/sbin/cron[8098]: (root) CMD (/root/bin/http_connection >> /tmp/http_connection.log)
7. 7 https://xkcd.com/208/
8. 8 https://xkcd.com/208/
10. 10 syslog ist ein De-facto-Standard zur Übermittlung von Log- Meldungen [...] Der Begriff „syslog“ wird oft sowohl für das eigentliche syslog- Netzwerkprotokoll als auch für die Anwendung oder Bibliothek benutzt, die syslog-Meldungen sendet oder empfängt. http://de.wikipedia.org/wiki/Syslog
12. 12 https://xkcd.com/208/
15. 15 Graylog2 enables you to unleash the power that lays inside your logs. Use it to run analytics, alerting, monitoring and powerful searches over your whole log base. […] Create streams for every consumer and have them always only one click away. http://graylog2.org
17. 17 http://graylog2.org/
18. 18 http://graylog2.org/
19. 19 - Syslog - GELF The Graylog Extended Log Format
22. 22 logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). http://logstash.net
23. 23 Input Filter Output amqp drupal_dblog eventlog exec file ganglia gelf gemfire generator heroku irc log4j lumberjack pipe redis relp sqs stdin stomp syslog tcp twitter udp xmpp zenoss zeromq alter anonymize checksum csv date dns environment gelfify geoip grep grok grokdiscovery json kv metrics multiline mutate noop split syslog_pri urldecode xml zeromq amqp boundary circonus cloudwatch datadog elasticsearch elasticsearch_http elasticsearch_river email exec file ganglia gelf gemfire graphite graphtastic http internal irc juggernaut librato loggly lumberjack metriccatcher mongodb nagios nagios_nsca opentsdb pagerduty pipe redis riak riemann sns sqs statsd stdout stomp syslog tcp websocket xmpp zabbix zeromq
27. 27 graphite statsd
28. 28 graphite statsd Zentrale Filterung Metriken Monitoring Zentrale Einsicht
29. 29 graphite statsd
30. 30 http://kibana.org/
31. 31 http://www.elasticsearch.org/
32. 32 Keine Raketenwissenschaft Jim Parsons in The Big Bang Theory http://de.wikipedia.org/wiki/The_Big_Bang_Theory
33. 33 • Analyse • Monitoring • Reporting
34. www.capgemini.com Jan Doberstein Infrastructur Engineer jan.doberstein@capgemini.com http://about.me/jandoberstein Credits: -Icons "Nerdy-Lines" by Christoph Brill (egore@gmx.de) - Graylog2 Gorilla from the oatmeal (http://theoatmeal.com/comics/semicolon) - Logstash Logo from http://logstash.net/ by Jordan Sissel @jordansissel Font : Source Sans Pro by Paul D. Hunt
35. Start: 19:00, Hotel Lobby INGRESS for Beginners by Kris Köhntopp 18:15 Ofenbar – neben Rezeption