$30 off During Our Annual Pro Sale. View Details »

Structured Logging & Introduction to Graylog Collector

Bernd Ahlers
November 20, 2015
0
410

Structured Logging & Introduction to Graylog Collector

Structured logging and an introduction to Graylog Collector. OSMC 2015 (open source monitoring conference)

Bernd Ahlers

November 20, 2015
Tweet

More Decks by Bernd Ahlers

See All by Bernd Ahlers
Zentrales Log-Management mit Graylog (german)
bernd
0
500
What is your configuration management doing?
bernd
0
55
Get the best out of Graylog2 & Icinga 2
bernd
0
710
fpm-cookery RUGHH Oct 2011
bernd
0
67

Featured

See All Featured
How STYLIGHT went responsive
nonsquared
89
4.5k
The MySQL Ecosystem @ GitHub 2015
samlambert
241
11k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
16
1.5k
A designer walks into a library…
pauljervisheath
199
19k
Web Components: a chance to create the future
zenorocha
304
41k
The Straight Up "How To Draw Better" Workshop
denniskardys
226
130k
Music & Morning Musume
bryan
38
5.2k
Debugging Ruby Performance
tmm1
68
11k
Intergalactic Javascript Robots from Outer Space
tanoku
264
26k
Faster Mobile Websites
deanohume
296
29k
How to name files
jennybc
59
87k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
52
19k

Transcript

  1. Bernd Ahlers – Graylog, Inc. [email protected]
    Monitoring Linux and Windows Logs
    with Graylog Collector
    Bernd Ahlers
    Graylog, Inc.

    View Slide

  2. Bernd Ahlers – Graylog, Inc. [email protected]
    Structured Logging & Introduction to
    Graylog Collector
    Bernd Ahlers
    Graylog, Inc.

    View Slide

  3. Bernd Ahlers – Graylog, Inc. [email protected]
    Introduction: Graylog
    ●
    Open source log management platform
    ●
    Collect, index and analyze structured and
    unstructured log data
    ●
    Alerts based on log data
    ●
    Extensible via custom plugins

    View Slide

  4. Bernd Ahlers – Graylog, Inc. [email protected]

    View Slide

  5. Bernd Ahlers – Graylog, Inc. [email protected]

    View Slide

  6. Bernd Ahlers – Graylog, Inc. [email protected]

    View Slide

  7. Bernd Ahlers – Graylog, Inc. [email protected]

    View Slide

  8. Bernd Ahlers – Graylog, Inc. [email protected]

    View Slide

  9. Bernd Ahlers – Graylog, Inc. [email protected]

    View Slide

  10. Bernd Ahlers – Graylog, Inc. [email protected]

    View Slide

  11. Bernd Ahlers – Graylog, Inc. [email protected]

    View Slide

  12. Bernd Ahlers – Graylog, Inc. [email protected]
    More about Graylog
    ●
    www.graylog.org
    ●
    marketplace.graylog.org
    ●
    docs.graylog.org
    ●
    github.com/Graylog2

    View Slide

  13. Bernd Ahlers – Graylog, Inc. [email protected]
    Why are we writing logs?
    ●
    Getting insight & collecting business metrics
    ●
    Debugging problems
    ●
    Building an audit trail
    ●
    Monitoring

    View Slide

  14. Bernd Ahlers – Graylog, Inc. [email protected]
    How do we access our logs?
    ●
    Applications write to local files
    ●
    SSH into machines
    ●
    tail, grep, awk
    ●
    If lucky: central log management

    View Slide

  15. Bernd Ahlers – Graylog, Inc. [email protected]
    What do they look like?
    ●
    Syslog RFC 3164 (BSD)
    ●
    Syslog RFC 5424

    View Slide

  16. Bernd Ahlers – Graylog, Inc. [email protected]
    Syslog RFC 3164 (BSD)
    Nov 10 15:55:01 tumbler CRON[2684]: (root) CMD
    (command -v debian-sa1 > /dev/null && debian-sa1
    1 1)

    View Slide

  17. Bernd Ahlers – Graylog, Inc. [email protected]
    Syslog RFC 5424
    2003-10-11T22:14:15.003Z mymachine.example.com
    evntslog - ID47 [exampleSDID@32473 iut="3"
    eventSource="Application" eventID="1011"] BOMAn
    application event log entry...

    View Slide

  18. Bernd Ahlers – Graylog, Inc. [email protected]
    Apache
    127.0.0.1 - bernd [28/Dec/2014:06:43:15 +0100]
    "PROPFIND /remote.php/webdav/ HTTP/1.1" 207 910
    "-" "Mozilla/5.0 (Linux) mirall/1.7.1"

    View Slide

  19. Bernd Ahlers – Graylog, Inc. [email protected]
    Postfix
    Aug 5 17:05:26 hostname postfix/qmgr[308]:
    A44F828C71: from=, size=153136,
    nrcpt=1 (queue active)

    View Slide

  20. Bernd Ahlers – Graylog, Inc. [email protected]
    Squid
    sq18.wikimedia.org 1715898 2010-12-
    01T21:57:22.331 0 1.2.3.4 TCP_MEM_HIT/200
    13208 GET
    http://en.wikipedia.org/wiki/Main_Page NONE/-
    text/html - - Mozilla/4.0%20(compatible;%20MSIE
    %206.0;%20Windows%20NT%205.1;%20.NET%20CLR
    %201.1.4322) en-US -

    View Slide

  21. Bernd Ahlers – Graylog, Inc. [email protected]
    log4j
    0 [main] INFO MyApp - Entering application.
    36 [main] DEBUG com.foo.Bar - Did it again!
    51 [main] INFO MyApp - Exiting application.

    View Slide

  22. Bernd Ahlers – Graylog, Inc. [email protected]
    Ruby Logger
    I, [2015-11-18T00:16:27.723972 #3609] INFO -- :
    Hello world!

    View Slide

  23. Bernd Ahlers – Graylog, Inc. [email protected]
    #1 Problem: Timestamps
    ●
    Everyone likes to invent one
    ●
    Missing most of the time: timezone, year

    View Slide

  24. Bernd Ahlers – Graylog, Inc. [email protected]
    How to get value out of unstructured logs?
    ●
    Regex
    ●
    More regex
    ●
    Even more regex

    View Slide

  25. Bernd Ahlers – Graylog, Inc. [email protected]
    ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:
    [0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|
    1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4})
    {1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-
    9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:
    [0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-
    4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]
    {1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-
    9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]
    {1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-
    5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d))
    {3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-
    Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|
    1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]
    {1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|
    1\d\d|[1-9]?\d)){3}))|:)))(%.+)?

    View Slide

  26. Bernd Ahlers – Graylog, Inc. [email protected]
    Grok
    IPV6 ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9...
    USERNAME [a-zA-Z0-9._-]+
    USER %{USERNAME}
    HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-
    Za-z-]{0,62}))*(\.?|\b)
    EMAILLOCALPART [a-zA-Z][a-zA-Z0-9_.+-=:]+
    EMAILADDRESS %{EMAILLOCALPART}@%{HOSTNAME}
    ...
    COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}

    View Slide

  27. Bernd Ahlers – Graylog, Inc. [email protected]
    Graylog: Extractors
    ●
    Regular expressions based
    ●
    Extracts data into message fields

    View Slide

  28. Bernd Ahlers – Graylog, Inc. [email protected]

    View Slide

  29. Bernd Ahlers – Graylog, Inc. [email protected]
    How to fix this?
    ●
    Central log collection (Graylog, ELK, others)
    ●
    Use structured log formats
    – Structured Syslog RFC 5424
    – CEF Format
    – GELF
    – JSON

    View Slide

  30. Bernd Ahlers – Graylog, Inc. [email protected]
    Structured Syslog RFC 5424
    2003-10-11T22:14:15.003Z mymachine.example.com
    evntslog - ID47 [exampleSDID@32473 iut="3"
    eventSource="Application" eventID="1011"] BOMAn
    application event log entry...

    View Slide

  31. Bernd Ahlers – Graylog, Inc. [email protected]
    CEF by ArcSight/HP
    Sep 19 08:26:10 host CEF:0|HP|siem|
    1.0|100|service
    successfully stopped|10|
    src=10.0.0.1 dst=2.1.2.2 spt=1232

    View Slide

  32. Bernd Ahlers – Graylog, Inc. [email protected]
    GELF
    { "version": "1.1",
    "timestamp": 1385053862.3072,
    "host": "example.org",
    "short_message": "A short message",
    "full_message": "Backtrace here\n\nmore stuff",
    "level": 1,
    "_user_id": 9001,
    "_some_info": "foo",
    "_some_env_var": "bar"}

    View Slide

  33. Bernd Ahlers – Graylog, Inc. [email protected]
    JSON
    { "source": "example.org",
    "message": "A log message",
    "timestamp": "2015-11-15T10:43:21Z",
    "user_id": 9001,
    "http_method": "GET"}

    View Slide

  34. Bernd Ahlers – Graylog, Inc. [email protected]
    How we try to improve the ecosystem
    ●
    Icinga2 GELF output for events
    ●
    Docker GELF logging driver (since Docker 1.8)
    ●
    apache-mod_log_gelf (beta)
    ●
    log4j2-gelf
    ●
    gelfclient Java library
    ●
    svloggelfd (log forwarding for runit)

    View Slide

  35. Bernd Ahlers – Graylog, Inc. [email protected]
    We at Graylog <3 structured data
    and you should too!

    View Slide

  36. Bernd Ahlers – Graylog, Inc. [email protected]
    Introduction: Graylog Collector
    ●
    Reads local log files and ships them to Graylog
    ●
    Windows EventLog support (limited for now)
    ●
    Transport encryption via TLS
    ●
    Runs on Linux, Windows, Mac OS X and AIX

    View Slide

  37. Bernd Ahlers – Graylog, Inc. [email protected]
    Why another Collector?
    ●
    There are lots of others: nxlog, fluentd, heka,
    filebeat, rsyslog, syslog-ng
    ●
    We want integration and centralized
    management of collectors in Graylog

    View Slide

  38. Bernd Ahlers – Graylog, Inc. [email protected]

    View Slide

  39. Bernd Ahlers – Graylog, Inc. [email protected]
    Collector Installation
    ●
    OS packages for Linux distributions
    ●
    Manual installation on Windows via ZIP file
    (MSI upcoming)
    Runs as Windows service

    View Slide

  40. Bernd Ahlers – Graylog, Inc. [email protected]
    Collector Configuration
    server-url = "http://your-graylog-server:12900"
    inputs {
    windows-application-log {
    type = "windows-eventlog"
    source-name = "Application"
    }
    }
    outputs {
    gelf-tcp {
    type = "gelf"
    host = "your-graylog-server"
    port = 12201
    }
    }

    View Slide

  41. Bernd Ahlers – Graylog, Inc. [email protected]
    Collector: Current State
    ●
    Windows EventLog support needs update to
    support new Windows APIs
    ●
    File reading needs improvement
    ●
    Centralized management needs to be
    implemented
    ●
    :-(

    View Slide

  42. Bernd Ahlers – Graylog, Inc. [email protected]
    Tomorrow: Hackathon

    View Slide

  43. Bernd Ahlers – Graylog, Inc. [email protected]
    Thank you!
    Thank you for your time!

    View Slide

  44. Bernd Ahlers – Graylog, Inc. [email protected]
    QA
    Ask me anything!
    Bernd Ahlers / Graylog, Inc.
    [email protected]
    @berndahlers
    www.graylog.org
    github.com/Graylog2

    View Slide