The Truth is in the Logs (de)

my Slides from the #osdc 2013 in Nuremberg about Logfile Management and Processing

Video from the presentation can be found on Youtube: http://youtu.be/vLdJJ14qezc

Transcript

1. The Truth is in the Logs
   Nuremberg, 2013-04-17
   

   View Slide

2. 2
    Capgemini Outsourcing Services
   • Infrastructur Engineer
    wer-kennt-wen.de GmbH
   • System Operator Linux
    1blu AG
   • System Engineer (Shared
   Hosting)
   • Teamlead System Operations
    HostEurope GmbH
   • Teamlead Shared Hosting
   Operations
   • Postmaster
   Jan Doberstein
   #OSS #bash #vi
   

   View Slide

3. 3
   Eine Logdatei … enthält das
   automatisch geführte Protokoll aller
   oder bestimmter Aktionen von
   Prozessen auf einem Computersystem
   …
   Wichtige Anwendungen finden sich
   vor allem bei der Prozesskontrolle und
   Automatisierung. Prinzipiell werden
   alle Aktionen mitgeschrieben, die für
   eine spätere Untersuchung
   erforderlich sind oder sein könnten.
   https://de.wikipedia.org/wiki/Logdatei
   

   View Slide

4. 4
   10.41.148.32 - - [18/Mar/2013:12:43:43 +0100] "GET
   /health/currentthroughput HTTP/1.1" 200 21
   "http://10.43.248.109/users/new" "Mozilla/5.0
   (Windows NT 6.1) AppleWebKit/537.22 (KHTML, like
   Gecko) Chrome/25.0.1364.172 Safari/537.22"
   10.41.148.32 - - [18/Mar/2013:13:06:17 +0100] "GET
   /visuals/fetch/streamgraph?hours=2&stream_id=514
   6bccd772ae72c95000006 HTTP/1.1" 200 2162
   "http://10.43.248.109/streams/5146bccd772ae72c95
   000006-user/messages" "Mozilla/5.0 (Windows NT
   6.1) AppleWebKit/537.22 (KHTML, like Gecko)
   Chrome/25.0.1364.172 Safari/537.22"
   10.41.148.32 - - [18/Mar/2013:13:06:22 +0100] "GET
   /messages HTTP/1.1" 200 57786
   "http://10.43.248.109/streams/5146bccd772ae72c95
   000006-user/messages" "Mozilla/5.0 (Windows NT
   6.1) AppleWebKit/537.22 (KHTML, like Gecko)
   Chrome/25.0.1364.172 Safari/537.22"
   Mar 19 09:30:01 sv004972 /usr/sbin/cron[7657]: (root)
   CMD (/root/bin/http_connection >>
   /tmp/http_connection.log)
   Mar 19 09:30:02 sv004972 sshd[7761]: Accepted
   publickey for root from 192.168.97.41 port 37209
   ssh2
   Mar 19 09:31:01 sv004972 /usr/sbin/cron[8098]:
   (/root/bin/http_connection >>
   /tmp/http_connection.log)
   Mar 19 09:30:02 sv004972 sshd[7761]: Accepted
   publickey for root from 192.168.97.41 port 37209
   ssh2
   Mar 19 09:30:03 sv004972 sshd[7795]: Accepted
   publickey for root from 192.168.97.41 port 37212
   ssh2
   Mar 19 09:31:01 sv004972 /usr/sbin/cron[8098]: (root)
   CMD (/root/bin/http_connection >>
   /tmp/http_connection.log)
   

   View Slide

5. View Slide

6. View Slide

7. 7
   https://xkcd.com/208/
   

   View Slide

8. 8
   https://xkcd.com/208/
   

   View Slide

9. View Slide

10. 10
    syslog ist ein De-facto-Standard
    zur Übermittlung von Log-
    Meldungen [...] Der Begriff
    „syslog“ wird oft sowohl für das
    eigentliche syslog-
    Netzwerkprotokoll als auch für
    die Anwendung oder Bibliothek
    benutzt, die syslog-Meldungen
    sendet oder empfängt.
    http://de.wikipedia.org/wiki/Syslog
    

    View Slide

11. View Slide

12. 12
    https://xkcd.com/208/
    

    View Slide

13. View Slide

14. View Slide

15. 15
    Graylog2 enables you to unleash the power that
    lays inside your logs. Use it to run analytics,
    alerting, monitoring and powerful searches over
    your whole log base. […] Create streams for
    every consumer and have them always only one
    click away.
    http://graylog2.org
    

    View Slide

16. View Slide

17. 17
    http://graylog2.org/
    

    View Slide

18. 18
    http://graylog2.org/
    

    View Slide

19. 19
    - Syslog
    - GELF
    The Graylog Extended Log Format
    

    View Slide

20. View Slide

21. View Slide

22. 22
    logstash is a tool for managing events and
    logs. You can use it to collect logs, parse them,
    and store them for later use (like, for
    searching).
    http://logstash.net
    

    View Slide

23. 23
    Input Filter Output
    amqp
    drupal_dblog
    eventlog
    exec
    file
    ganglia
    gelf
    gemfire
    generator
    heroku
    irc
    log4j
    lumberjack
    pipe
    redis
    relp
    sqs
    stdin
    stomp
    syslog
    tcp
    twitter
    udp
    xmpp
    zenoss
    zeromq
    alter
    anonymize
    checksum
    csv
    date
    dns
    environment
    gelfify
    geoip
    grep
    grok
    grokdiscovery
    json
    kv
    metrics
    multiline
    mutate
    noop
    split
    syslog_pri
    urldecode
    xml
    zeromq
    amqp
    boundary
    circonus
    cloudwatch
    datadog
    elasticsearch
    elasticsearch_http
    elasticsearch_river
    email
    exec
    file
    ganglia
    gelf
    gemfire
    graphite
    graphtastic
    http
    internal
    irc
    juggernaut
    librato
    loggly
    lumberjack
    metriccatcher
    mongodb
    nagios
    nagios_nsca
    opentsdb
    pagerduty
    pipe
    redis
    riak
    riemann
    sns
    sqs
    statsd
    stdout
    stomp
    syslog
    tcp
    websocket
    xmpp
    zabbix
    zeromq
    

    View Slide

24. View Slide

25. View Slide

26. View Slide

27. 27
    graphite
    statsd
    

    View Slide

28. 28
    graphite
    statsd
    Zentrale Filterung
    Metriken
    Monitoring
    Zentrale Einsicht
    

    View Slide

29. 29
    graphite
    statsd
    

    View Slide

30. 30
    http://kibana.org/
    

    View Slide

31. 31
    http://www.elasticsearch.org/
    

    View Slide

32. 32
    Keine Raketenwissenschaft
    Jim Parsons in The Big Bang Theory http://de.wikipedia.org/wiki/The_Big_Bang_Theory
    

    View Slide

33. 33
    • Analyse
    • Monitoring
    • Reporting
    

    View Slide

34. www.capgemini.com
    Jan Doberstein
    Infrastructur Engineer
    jan.doberstein@capgemini.com
    http://about.me/jandoberstein
    Credits:
    -Icons "Nerdy-Lines" by Christoph Brill (egore@gmx.de)
    - Graylog2 Gorilla from the oatmeal
    (http://theoatmeal.com/comics/semicolon)
    - Logstash Logo from http://logstash.net/ by Jordan Sissel @jordansissel
    Font : Source Sans Pro by Paul D. Hunt
    

    View Slide

35. Start: 19:00, Hotel Lobby
    INGRESS for Beginners by Kris Köhntopp
    18:15 Ofenbar – neben Rezeption
    

    View Slide