$30 off During Our Annual Pro Sale. View Details »

Structured Logging & Introduction to Graylog Collector

Bernd Ahlers
November 20, 2015
410

Structured Logging & Introduction to Graylog Collector

Structured logging and an introduction to Graylog Collector. OSMC 2015 (open source monitoring conference)

Bernd Ahlers

November 20, 2015
Tweet

Transcript

  1. Bernd Ahlers – Graylog, Inc. [email protected]
    Monitoring Linux and Windows Logs
    with Graylog Collector
    Bernd Ahlers
    Graylog, Inc.

    View Slide

  2. Bernd Ahlers – Graylog, Inc. [email protected]
    Structured Logging & Introduction to
    Graylog Collector
    Bernd Ahlers
    Graylog, Inc.

    View Slide

  3. Bernd Ahlers – Graylog, Inc. [email protected]
    Introduction: Graylog

    Open source log management platform

    Collect, index and analyze structured and
    unstructured log data

    Alerts based on log data

    Extensible via custom plugins

    View Slide

  4. Bernd Ahlers – Graylog, Inc. [email protected]

    View Slide

  5. Bernd Ahlers – Graylog, Inc. [email protected]

    View Slide

  6. Bernd Ahlers – Graylog, Inc. [email protected]

    View Slide

  7. Bernd Ahlers – Graylog, Inc. [email protected]

    View Slide

  8. Bernd Ahlers – Graylog, Inc. [email protected]

    View Slide

  9. Bernd Ahlers – Graylog, Inc. [email protected]

    View Slide

  10. Bernd Ahlers – Graylog, Inc. [email protected]

    View Slide

  11. Bernd Ahlers – Graylog, Inc. [email protected]

    View Slide

  12. Bernd Ahlers – Graylog, Inc. [email protected]
    More about Graylog

    www.graylog.org

    marketplace.graylog.org

    docs.graylog.org

    github.com/Graylog2

    View Slide

  13. Bernd Ahlers – Graylog, Inc. [email protected]
    Why are we writing logs?

    Getting insight & collecting business metrics

    Debugging problems

    Building an audit trail

    Monitoring

    View Slide

  14. Bernd Ahlers – Graylog, Inc. [email protected]
    How do we access our logs?

    Applications write to local files

    SSH into machines

    tail, grep, awk

    If lucky: central log management

    View Slide

  15. Bernd Ahlers – Graylog, Inc. [email protected]
    What do they look like?

    Syslog RFC 3164 (BSD)

    Syslog RFC 5424

    View Slide

  16. Bernd Ahlers – Graylog, Inc. [email protected]
    Syslog RFC 3164 (BSD)
    Nov 10 15:55:01 tumbler CRON[2684]: (root) CMD
    (command -v debian-sa1 > /dev/null && debian-sa1
    1 1)

    View Slide

  17. Bernd Ahlers – Graylog, Inc. [email protected]
    Syslog RFC 5424
    2003-10-11T22:14:15.003Z mymachine.example.com
    evntslog - ID47 [exampleSDID@32473 iut="3"
    eventSource="Application" eventID="1011"] BOMAn
    application event log entry...

    View Slide

  18. Bernd Ahlers – Graylog, Inc. [email protected]
    Apache
    127.0.0.1 - bernd [28/Dec/2014:06:43:15 +0100]
    "PROPFIND /remote.php/webdav/ HTTP/1.1" 207 910
    "-" "Mozilla/5.0 (Linux) mirall/1.7.1"

    View Slide

  19. Bernd Ahlers – Graylog, Inc. [email protected]
    Postfix
    Aug 5 17:05:26 hostname postfix/qmgr[308]:
    A44F828C71: from=, size=153136,
    nrcpt=1 (queue active)

    View Slide

  20. Bernd Ahlers – Graylog, Inc. [email protected]
    Squid
    sq18.wikimedia.org 1715898 2010-12-
    01T21:57:22.331 0 1.2.3.4 TCP_MEM_HIT/200
    13208 GET
    http://en.wikipedia.org/wiki/Main_Page NONE/-
    text/html - - Mozilla/4.0%20(compatible;%20MSIE
    %206.0;%20Windows%20NT%205.1;%20.NET%20CLR
    %201.1.4322) en-US -

    View Slide

  21. Bernd Ahlers – Graylog, Inc. [email protected]
    log4j
    0 [main] INFO MyApp - Entering application.
    36 [main] DEBUG com.foo.Bar - Did it again!
    51 [main] INFO MyApp - Exiting application.

    View Slide

  22. Bernd Ahlers – Graylog, Inc. [email protected]
    Ruby Logger
    I, [2015-11-18T00:16:27.723972 #3609] INFO -- :
    Hello world!

    View Slide

  23. Bernd Ahlers – Graylog, Inc. [email protected]
    #1 Problem: Timestamps

    Everyone likes to invent one

    Missing most of the time: timezone, year

    View Slide

  24. Bernd Ahlers – Graylog, Inc. [email protected]
    How to get value out of unstructured logs?

    Regex

    More regex

    Even more regex

    View Slide

  25. Bernd Ahlers – Graylog, Inc. [email protected]
    ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:
    [0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|
    1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4})
    {1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-
    9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:
    [0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-
    4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]
    {1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-
    9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]
    {1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-
    5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d))
    {3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-
    Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|
    1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]
    {1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|
    1\d\d|[1-9]?\d)){3}))|:)))(%.+)?

    View Slide

  26. Bernd Ahlers – Graylog, Inc. [email protected]
    Grok
    IPV6 ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9...
    USERNAME [a-zA-Z0-9._-]+
    USER %{USERNAME}
    HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-
    Za-z-]{0,62}))*(\.?|\b)
    EMAILLOCALPART [a-zA-Z][a-zA-Z0-9_.+-=:]+
    EMAILADDRESS %{EMAILLOCALPART}@%{HOSTNAME}
    ...
    COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}

    View Slide

  27. Bernd Ahlers – Graylog, Inc. [email protected]
    Graylog: Extractors

    Regular expressions based

    Extracts data into message fields

    View Slide

  28. Bernd Ahlers – Graylog, Inc. [email protected]

    View Slide

  29. Bernd Ahlers – Graylog, Inc. [email protected]
    How to fix this?

    Central log collection (Graylog, ELK, others)

    Use structured log formats
    – Structured Syslog RFC 5424
    – CEF Format
    – GELF
    – JSON

    View Slide

  30. Bernd Ahlers – Graylog, Inc. [email protected]
    Structured Syslog RFC 5424
    2003-10-11T22:14:15.003Z mymachine.example.com
    evntslog - ID47 [exampleSDID@32473 iut="3"
    eventSource="Application" eventID="1011"] BOMAn
    application event log entry...

    View Slide

  31. Bernd Ahlers – Graylog, Inc. [email protected]
    CEF by ArcSight/HP
    Sep 19 08:26:10 host CEF:0|HP|siem|
    1.0|100|service
    successfully stopped|10|
    src=10.0.0.1 dst=2.1.2.2 spt=1232

    View Slide

  32. Bernd Ahlers – Graylog, Inc. [email protected]
    GELF
    { "version": "1.1",
    "timestamp": 1385053862.3072,
    "host": "example.org",
    "short_message": "A short message",
    "full_message": "Backtrace here\n\nmore stuff",
    "level": 1,
    "_user_id": 9001,
    "_some_info": "foo",
    "_some_env_var": "bar"}

    View Slide

  33. Bernd Ahlers – Graylog, Inc. [email protected]
    JSON
    { "source": "example.org",
    "message": "A log message",
    "timestamp": "2015-11-15T10:43:21Z",
    "user_id": 9001,
    "http_method": "GET"}

    View Slide

  34. Bernd Ahlers – Graylog, Inc. [email protected]
    How we try to improve the ecosystem

    Icinga2 GELF output for events

    Docker GELF logging driver (since Docker 1.8)

    apache-mod_log_gelf (beta)

    log4j2-gelf

    gelfclient Java library

    svloggelfd (log forwarding for runit)

    View Slide

  35. Bernd Ahlers – Graylog, Inc. [email protected]
    We at Graylog <3 structured data
    and you should too!

    View Slide

  36. Bernd Ahlers – Graylog, Inc. [email protected]
    Introduction: Graylog Collector

    Reads local log files and ships them to Graylog

    Windows EventLog support (limited for now)

    Transport encryption via TLS

    Runs on Linux, Windows, Mac OS X and AIX

    View Slide

  37. Bernd Ahlers – Graylog, Inc. [email protected]
    Why another Collector?

    There are lots of others: nxlog, fluentd, heka,
    filebeat, rsyslog, syslog-ng

    We want integration and centralized
    management of collectors in Graylog

    View Slide

  38. Bernd Ahlers – Graylog, Inc. [email protected]

    View Slide

  39. Bernd Ahlers – Graylog, Inc. [email protected]
    Collector Installation

    OS packages for Linux distributions

    Manual installation on Windows via ZIP file
    (MSI upcoming)
    Runs as Windows service

    View Slide

  40. Bernd Ahlers – Graylog, Inc. [email protected]
    Collector Configuration
    server-url = "http://your-graylog-server:12900"
    inputs {
    windows-application-log {
    type = "windows-eventlog"
    source-name = "Application"
    }
    }
    outputs {
    gelf-tcp {
    type = "gelf"
    host = "your-graylog-server"
    port = 12201
    }
    }

    View Slide

  41. Bernd Ahlers – Graylog, Inc. [email protected]
    Collector: Current State

    Windows EventLog support needs update to
    support new Windows APIs

    File reading needs improvement

    Centralized management needs to be
    implemented

    :-(

    View Slide

  42. Bernd Ahlers – Graylog, Inc. [email protected]
    Tomorrow: Hackathon

    View Slide

  43. Bernd Ahlers – Graylog, Inc. [email protected]
    Thank you!
    Thank you for your time!

    View Slide

  44. Bernd Ahlers – Graylog, Inc. [email protected]
    QA
    Ask me anything!
    Bernd Ahlers / Graylog, Inc.
    [email protected]
    @berndahlers
    www.graylog.org
    github.com/Graylog2

    View Slide