The Truth is in the Logs (de)

The Truth is in the Logs (de)

my Slides from the #osdc 2013 in Nuremberg about Logfile Management and Processing

Video from the presentation can be found on Youtube: http://youtu.be/vLdJJ14qezc

8bf1b7d831c3818f6f3997e260f9ad3d?s=128

Jan Doberstein

April 17, 2013
Tweet

Transcript

  1. The Truth is in the Logs Nuremberg, 2013-04-17

  2. 2  Capgemini Outsourcing Services • Infrastructur Engineer  wer-kennt-wen.de

    GmbH • System Operator Linux  1blu AG • System Engineer (Shared Hosting) • Teamlead System Operations  HostEurope GmbH • Teamlead Shared Hosting Operations • Postmaster Jan Doberstein #OSS #bash #vi
  3. 3 Eine Logdatei … enthält das automatisch geführte Protokoll aller

    oder bestimmter Aktionen von Prozessen auf einem Computersystem … Wichtige Anwendungen finden sich vor allem bei der Prozesskontrolle und Automatisierung. Prinzipiell werden alle Aktionen mitgeschrieben, die für eine spätere Untersuchung erforderlich sind oder sein könnten. https://de.wikipedia.org/wiki/Logdatei
  4. 4 10.41.148.32 - - [18/Mar/2013:12:43:43 +0100] "GET /health/currentthroughput HTTP/1.1" 200

    21 "http://10.43.248.109/users/new" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.172 Safari/537.22" 10.41.148.32 - - [18/Mar/2013:13:06:17 +0100] "GET /visuals/fetch/streamgraph?hours=2&stream_id=514 6bccd772ae72c95000006 HTTP/1.1" 200 2162 "http://10.43.248.109/streams/5146bccd772ae72c95 000006-user/messages" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.172 Safari/537.22" 10.41.148.32 - - [18/Mar/2013:13:06:22 +0100] "GET /messages HTTP/1.1" 200 57786 "http://10.43.248.109/streams/5146bccd772ae72c95 000006-user/messages" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.172 Safari/537.22" Mar 19 09:30:01 sv004972 /usr/sbin/cron[7657]: (root) CMD (/root/bin/http_connection >> /tmp/http_connection.log) Mar 19 09:30:02 sv004972 sshd[7761]: Accepted publickey for root from 192.168.97.41 port 37209 ssh2 Mar 19 09:31:01 sv004972 /usr/sbin/cron[8098]: (/root/bin/http_connection >> /tmp/http_connection.log) Mar 19 09:30:02 sv004972 sshd[7761]: Accepted publickey for root from 192.168.97.41 port 37209 ssh2 Mar 19 09:30:03 sv004972 sshd[7795]: Accepted publickey for root from 192.168.97.41 port 37212 ssh2 Mar 19 09:31:01 sv004972 /usr/sbin/cron[8098]: (root) CMD (/root/bin/http_connection >> /tmp/http_connection.log)
  5. 5

  6. 6

  7. 7 https://xkcd.com/208/

  8. 8 https://xkcd.com/208/

  9. 9

  10. 10 syslog ist ein De-facto-Standard zur Übermittlung von Log- Meldungen

    [...] Der Begriff „syslog“ wird oft sowohl für das eigentliche syslog- Netzwerkprotokoll als auch für die Anwendung oder Bibliothek benutzt, die syslog-Meldungen sendet oder empfängt. http://de.wikipedia.org/wiki/Syslog
  11. 11

  12. 12 https://xkcd.com/208/

  13. 13

  14. 14

  15. 15 Graylog2 enables you to unleash the power that lays

    inside your logs. Use it to run analytics, alerting, monitoring and powerful searches over your whole log base. […] Create streams for every consumer and have them always only one click away. http://graylog2.org
  16. 16

  17. 17 http://graylog2.org/

  18. 18 http://graylog2.org/

  19. 19 - Syslog - GELF The Graylog Extended Log Format

  20. 20

  21. 21

  22. 22 logstash is a tool for managing events and logs.

    You can use it to collect logs, parse them, and store them for later use (like, for searching). http://logstash.net
  23. 23 Input Filter Output amqp drupal_dblog eventlog exec file ganglia

    gelf gemfire generator heroku irc log4j lumberjack pipe redis relp sqs stdin stomp syslog tcp twitter udp xmpp zenoss zeromq alter anonymize checksum csv date dns environment gelfify geoip grep grok grokdiscovery json kv metrics multiline mutate noop split syslog_pri urldecode xml zeromq amqp boundary circonus cloudwatch datadog elasticsearch elasticsearch_http elasticsearch_river email exec file ganglia gelf gemfire graphite graphtastic http internal irc juggernaut librato loggly lumberjack metriccatcher mongodb nagios nagios_nsca opentsdb pagerduty pipe redis riak riemann sns sqs statsd stdout stomp syslog tcp websocket xmpp zabbix zeromq
  24. 24

  25. 25

  26. 26

  27. 27 graphite statsd

  28. 28 graphite statsd Zentrale Filterung Metriken Monitoring Zentrale Einsicht

  29. 29 graphite statsd

  30. 30 http://kibana.org/

  31. 31 http://www.elasticsearch.org/

  32. 32 Keine Raketenwissenschaft Jim Parsons in The Big Bang Theory

    http://de.wikipedia.org/wiki/The_Big_Bang_Theory
  33. 33 • Analyse • Monitoring • Reporting

  34. www.capgemini.com Jan Doberstein Infrastructur Engineer jan.doberstein@capgemini.com http://about.me/jandoberstein Credits: -Icons "Nerdy-Lines"

    by Christoph Brill (egore@gmx.de) - Graylog2 Gorilla from the oatmeal (http://theoatmeal.com/comics/semicolon) - Logstash Logo from http://logstash.net/ by Jordan Sissel @jordansissel Font : Source Sans Pro by Paul D. Hunt
  35. Start: 19:00, Hotel Lobby INGRESS for Beginners by Kris Köhntopp

    18:15 Ofenbar – neben Rezeption