Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Structured Logging & Introduction to Graylog Co...

Bernd Ahlers
November 20, 2015
0
440

Structured Logging & Introduction to Graylog Collector

Structured logging and an introduction to Graylog Collector. OSMC 2015 (open source monitoring conference)

Bernd Ahlers

November 20, 2015
Tweet

More Decks by Bernd Ahlers

See All by Bernd Ahlers
Zentrales Log-Management mit Graylog (german)
bernd
0
580
What is your configuration management doing?
bernd
0
65
Get the best out of Graylog2 & Icinga 2
bernd
0
780
fpm-cookery RUGHH Oct 2011
bernd
0
110

Featured

See All Featured
How to train your dragon (web standard)
notwaldorf
88
5.7k
Facilitating Awesome Meetings
lara
50
6.1k
Rebuilding a faster, lazier Slack
samanthasiow
79
8.7k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
8
1.2k
The Cult of Friendly URLs
andyhume
78
6.1k
Building Your Own Lightsaber
phodgson
103
6.1k
Intergalactic Javascript Robots from Outer Space
tanoku
270
27k
A Philosophy of Restraint
colly
203
16k
Bash Introduction
62gerente
608
210k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
127
18k
KATA
mclloyd
29
14k
The Straight Up "How To Draw Better" Workshop
denniskardys
232
140k

Transcript

  1. Bernd Ahlers – Graylog, Inc. [email protected] Monitoring Linux and Windows

    Logs with Graylog Collector Bernd Ahlers Graylog, Inc.

  2. Bernd Ahlers – Graylog, Inc. [email protected] Structured Logging & Introduction

    to Graylog Collector Bernd Ahlers Graylog, Inc.

  3. Bernd Ahlers – Graylog, Inc. [email protected] Introduction: Graylog • Open

    source log management platform • Collect, index and analyze structured and unstructured log data • Alerts based on log data • Extensible via custom plugins

  4. Bernd Ahlers – Graylog, Inc. [email protected]

  5. Bernd Ahlers – Graylog, Inc. [email protected]

  6. Bernd Ahlers – Graylog, Inc. [email protected]

  7. Bernd Ahlers – Graylog, Inc. [email protected]

  8. Bernd Ahlers – Graylog, Inc. [email protected]

  9. Bernd Ahlers – Graylog, Inc. [email protected]

  10. Bernd Ahlers – Graylog, Inc. [email protected]

  11. Bernd Ahlers – Graylog, Inc. [email protected]

  12. Bernd Ahlers – Graylog, Inc. [email protected] More about Graylog •

    www.graylog.org • marketplace.graylog.org • docs.graylog.org • github.com/Graylog2

  13. Bernd Ahlers – Graylog, Inc. [email protected] Why are we writing

    logs? • Getting insight & collecting business metrics • Debugging problems • Building an audit trail • Monitoring

  14. Bernd Ahlers – Graylog, Inc. [email protected] How do we access

    our logs? • Applications write to local files • SSH into machines • tail, grep, awk • If lucky: central log management

  15. Bernd Ahlers – Graylog, Inc. [email protected] What do they look

    like? • Syslog RFC 3164 (BSD) • Syslog RFC 5424

  16. Bernd Ahlers – Graylog, Inc. [email protected] Syslog RFC 3164 (BSD)

    Nov 10 15:55:01 tumbler CRON[2684]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)

  17. Bernd Ahlers – Graylog, Inc. [email protected] Syslog RFC 5424 2003-10-11T22:14:15.003Z

    mymachine.example.com evntslog - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] BOMAn application event log entry...

  18. Bernd Ahlers – Graylog, Inc. [email protected] Apache 127.0.0.1 - bernd

    [28/Dec/2014:06:43:15 +0100] "PROPFIND /remote.php/webdav/ HTTP/1.1" 207 910 "-" "Mozilla/5.0 (Linux) mirall/1.7.1"

  19. Bernd Ahlers – Graylog, Inc. [email protected] Postfix Aug 5 17:05:26

    hostname postfix/qmgr[308]: A44F828C71: from=<[email protected]>, size=153136, nrcpt=1 (queue active)

  20. Bernd Ahlers – Graylog, Inc. [email protected] Squid sq18.wikimedia.org 1715898 2010-12-

    01T21:57:22.331 0 1.2.3.4 TCP_MEM_HIT/200 13208 GET http://en.wikipedia.org/wiki/Main_Page NONE/- text/html - - Mozilla/4.0%20(compatible;%20MSIE %206.0;%20Windows%20NT%205.1;%20.NET%20CLR %201.1.4322) en-US -

  21. Bernd Ahlers – Graylog, Inc. [email protected] log4j 0 [main] INFO

    MyApp - Entering application. 36 [main] DEBUG com.foo.Bar - Did it again! 51 [main] INFO MyApp - Exiting application.

  22. Bernd Ahlers – Graylog, Inc. [email protected] Ruby Logger I, [2015-11-18T00:16:27.723972

    #3609] INFO -- : Hello world!

  23. Bernd Ahlers – Graylog, Inc. [email protected] #1 Problem: Timestamps •

    Everyone likes to invent one • Missing most of the time: timezone, year

  24. Bernd Ahlers – Graylog, Inc. [email protected] How to get value

    out of unstructured logs? • Regex • More regex • Even more regex

  25. Bernd Ahlers – Graylog, Inc. [email protected] ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(: [0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d| 1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}) {1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-

    9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((: [0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0- 4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f] {1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1- 9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f] {1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0- 5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)) {3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A- Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d| 1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f] {1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d| 1\d\d|[1-9]?\d)){3}))|:)))(%.+)?

  26. Bernd Ahlers – Graylog, Inc. [email protected] Grok IPV6 ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9... USERNAME

    [a-zA-Z0-9._-]+ USER %{USERNAME} HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A- Za-z-]{0,62}))*(\.?|\b) EMAILLOCALPART [a-zA-Z][a-zA-Z0-9_.+-=:]+ EMAILADDRESS %{EMAILLOCALPART}@%{HOSTNAME} ... COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}

  27. Bernd Ahlers – Graylog, Inc. [email protected] Graylog: Extractors • Regular

    expressions based • Extracts data into message fields

  28. Bernd Ahlers – Graylog, Inc. [email protected]

  29. Bernd Ahlers – Graylog, Inc. [email protected] How to fix this?

    • Central log collection (Graylog, ELK, others) • Use structured log formats – Structured Syslog RFC 5424 – CEF Format – GELF – JSON

  30. Bernd Ahlers – Graylog, Inc. [email protected] Structured Syslog RFC 5424

    2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] BOMAn application event log entry...

  31. Bernd Ahlers – Graylog, Inc. [email protected] CEF by ArcSight/HP Sep

    19 08:26:10 host CEF:0|HP|siem| 1.0|100|service successfully stopped|10| src=10.0.0.1 dst=2.1.2.2 spt=1232

  32. Bernd Ahlers – Graylog, Inc. [email protected] GELF { "version": "1.1",

    "timestamp": 1385053862.3072, "host": "example.org", "short_message": "A short message", "full_message": "Backtrace here\n\nmore stuff", "level": 1, "_user_id": 9001, "_some_info": "foo", "_some_env_var": "bar"}

  33. Bernd Ahlers – Graylog, Inc. [email protected] JSON { "source": "example.org",

    "message": "A log message", "timestamp": "2015-11-15T10:43:21Z", "user_id": 9001, "http_method": "GET"}

  34. Bernd Ahlers – Graylog, Inc. [email protected] How we try to

    improve the ecosystem • Icinga2 GELF output for events • Docker GELF logging driver (since Docker 1.8) • apache-mod_log_gelf (beta) • log4j2-gelf • gelfclient Java library • svloggelfd (log forwarding for runit)

  35. Bernd Ahlers – Graylog, Inc. [email protected] We at Graylog <3

    structured data and you should too!

  36. Bernd Ahlers – Graylog, Inc. [email protected] Introduction: Graylog Collector •

    Reads local log files and ships them to Graylog • Windows EventLog support (limited for now) • Transport encryption via TLS • Runs on Linux, Windows, Mac OS X and AIX

  37. Bernd Ahlers – Graylog, Inc. [email protected] Why another Collector? •

    There are lots of others: nxlog, fluentd, heka, filebeat, rsyslog, syslog-ng • We want integration and centralized management of collectors in Graylog

  38. Bernd Ahlers – Graylog, Inc. [email protected]

  39. Bernd Ahlers – Graylog, Inc. [email protected] Collector Installation • OS

    packages for Linux distributions • Manual installation on Windows via ZIP file (MSI upcoming) Runs as Windows service

  40. Bernd Ahlers – Graylog, Inc. [email protected] Collector Configuration server-url =

    "http://your-graylog-server:12900" inputs { windows-application-log { type = "windows-eventlog" source-name = "Application" } } outputs { gelf-tcp { type = "gelf" host = "your-graylog-server" port = 12201 } }

  41. Bernd Ahlers – Graylog, Inc. [email protected] Collector: Current State •

    Windows EventLog support needs update to support new Windows APIs • File reading needs improvement • Centralized management needs to be implemented • :-(

  42. Bernd Ahlers – Graylog, Inc. [email protected] Tomorrow: Hackathon

  43. Bernd Ahlers – Graylog, Inc. [email protected] Thank you! Thank you

    for your time!

  44. Bernd Ahlers – Graylog, Inc. [email protected] QA Ask me anything!

    Bernd Ahlers / Graylog, Inc. [email protected] @berndahlers www.graylog.org github.com/Graylog2