Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Structured Logging & Introduction to Graylog Collector

Bernd Ahlers
November 20, 2015
0
430

Structured Logging & Introduction to Graylog Collector

Structured logging and an introduction to Graylog Collector. OSMC 2015 (open source monitoring conference)

Bernd Ahlers

November 20, 2015
Tweet

More Decks by Bernd Ahlers

See All by Bernd Ahlers
Zentrales Log-Management mit Graylog (german)
bernd
0
560
What is your configuration management doing?
bernd
0
58
Get the best out of Graylog2 & Icinga 2
bernd
0
750
fpm-cookery RUGHH Oct 2011
bernd
0
110

Featured

See All Featured
Rebuilding a faster, lazier Slack
samanthasiow
78
8.5k
A Philosophy of Restraint
colly
200
16k
Become a Pro
speakerdeck
PRO
15
4.8k
It's Worth the Effort
3n
181
27k
Principles of Awesome APIs and How to Build Them.
keavy
124
16k
Clear Off the Table
cherdarchuk
89
320k
Agile that works and the tools we love
rasmusluckow
325
20k
Unsuck your backbone
ammeep
666
57k
Thoughts on Productivity
jonyablonski
64
4.1k
Building Better People: How to give real-time feedback that sticks.
wjessup
357
18k
In The Pink: A Labor of Love
frogandcode
139
22k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
155
14k

Transcript

  1. Bernd Ahlers – Graylog, Inc. [email protected] Monitoring Linux and Windows

    Logs with Graylog Collector Bernd Ahlers Graylog, Inc.

  2. Bernd Ahlers – Graylog, Inc. [email protected] Structured Logging & Introduction

    to Graylog Collector Bernd Ahlers Graylog, Inc.

  3. Bernd Ahlers – Graylog, Inc. [email protected] Introduction: Graylog • Open

    source log management platform • Collect, index and analyze structured and unstructured log data • Alerts based on log data • Extensible via custom plugins

  4. Bernd Ahlers – Graylog, Inc. [email protected]

  5. Bernd Ahlers – Graylog, Inc. [email protected]

  6. Bernd Ahlers – Graylog, Inc. [email protected]

  7. Bernd Ahlers – Graylog, Inc. [email protected]

  8. Bernd Ahlers – Graylog, Inc. [email protected]

  9. Bernd Ahlers – Graylog, Inc. [email protected]

  10. Bernd Ahlers – Graylog, Inc. [email protected]

  11. Bernd Ahlers – Graylog, Inc. [email protected]

  12. Bernd Ahlers – Graylog, Inc. [email protected] More about Graylog •

    www.graylog.org • marketplace.graylog.org • docs.graylog.org • github.com/Graylog2

  13. Bernd Ahlers – Graylog, Inc. [email protected] Why are we writing

    logs? • Getting insight & collecting business metrics • Debugging problems • Building an audit trail • Monitoring

  14. Bernd Ahlers – Graylog, Inc. [email protected] How do we access

    our logs? • Applications write to local files • SSH into machines • tail, grep, awk • If lucky: central log management

  15. Bernd Ahlers – Graylog, Inc. [email protected] What do they look

    like? • Syslog RFC 3164 (BSD) • Syslog RFC 5424

  16. Bernd Ahlers – Graylog, Inc. [email protected] Syslog RFC 3164 (BSD)

    Nov 10 15:55:01 tumbler CRON[2684]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)

  17. Bernd Ahlers – Graylog, Inc. [email protected] Syslog RFC 5424 2003-10-11T22:14:15.003Z

    mymachine.example.com evntslog - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] BOMAn application event log entry...

  18. Bernd Ahlers – Graylog, Inc. [email protected] Apache 127.0.0.1 - bernd

    [28/Dec/2014:06:43:15 +0100] "PROPFIND /remote.php/webdav/ HTTP/1.1" 207 910 "-" "Mozilla/5.0 (Linux) mirall/1.7.1"

  19. Bernd Ahlers – Graylog, Inc. [email protected] Postfix Aug 5 17:05:26

    hostname postfix/qmgr[308]: A44F828C71: from=<[email protected]>, size=153136, nrcpt=1 (queue active)

  20. Bernd Ahlers – Graylog, Inc. [email protected] Squid sq18.wikimedia.org 1715898 2010-12-

    01T21:57:22.331 0 1.2.3.4 TCP_MEM_HIT/200 13208 GET http://en.wikipedia.org/wiki/Main_Page NONE/- text/html - - Mozilla/4.0%20(compatible;%20MSIE %206.0;%20Windows%20NT%205.1;%20.NET%20CLR %201.1.4322) en-US -

  21. Bernd Ahlers – Graylog, Inc. [email protected] log4j 0 [main] INFO

    MyApp - Entering application. 36 [main] DEBUG com.foo.Bar - Did it again! 51 [main] INFO MyApp - Exiting application.

  22. Bernd Ahlers – Graylog, Inc. [email protected] Ruby Logger I, [2015-11-18T00:16:27.723972

    #3609] INFO -- : Hello world!

  23. Bernd Ahlers – Graylog, Inc. [email protected] #1 Problem: Timestamps •

    Everyone likes to invent one • Missing most of the time: timezone, year

  24. Bernd Ahlers – Graylog, Inc. [email protected] How to get value

    out of unstructured logs? • Regex • More regex • Even more regex

  25. Bernd Ahlers – Graylog, Inc. [email protected] ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(: [0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d| 1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}) {1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-

    9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((: [0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0- 4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f] {1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1- 9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f] {1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0- 5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)) {3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A- Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d| 1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f] {1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d| 1\d\d|[1-9]?\d)){3}))|:)))(%.+)?

  26. Bernd Ahlers – Graylog, Inc. [email protected] Grok IPV6 ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9... USERNAME

    [a-zA-Z0-9._-]+ USER %{USERNAME} HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A- Za-z-]{0,62}))*(\.?|\b) EMAILLOCALPART [a-zA-Z][a-zA-Z0-9_.+-=:]+ EMAILADDRESS %{EMAILLOCALPART}@%{HOSTNAME} ... COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}

  27. Bernd Ahlers – Graylog, Inc. [email protected] Graylog: Extractors • Regular

    expressions based • Extracts data into message fields

  28. Bernd Ahlers – Graylog, Inc. [email protected]

  29. Bernd Ahlers – Graylog, Inc. [email protected] How to fix this?

    • Central log collection (Graylog, ELK, others) • Use structured log formats – Structured Syslog RFC 5424 – CEF Format – GELF – JSON

  30. Bernd Ahlers – Graylog, Inc. [email protected] Structured Syslog RFC 5424

    2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] BOMAn application event log entry...

  31. Bernd Ahlers – Graylog, Inc. [email protected] CEF by ArcSight/HP Sep

    19 08:26:10 host CEF:0|HP|siem| 1.0|100|service successfully stopped|10| src=10.0.0.1 dst=2.1.2.2 spt=1232

  32. Bernd Ahlers – Graylog, Inc. [email protected] GELF { "version": "1.1",

    "timestamp": 1385053862.3072, "host": "example.org", "short_message": "A short message", "full_message": "Backtrace here\n\nmore stuff", "level": 1, "_user_id": 9001, "_some_info": "foo", "_some_env_var": "bar"}

  33. Bernd Ahlers – Graylog, Inc. [email protected] JSON { "source": "example.org",

    "message": "A log message", "timestamp": "2015-11-15T10:43:21Z", "user_id": 9001, "http_method": "GET"}

  34. Bernd Ahlers – Graylog, Inc. [email protected] How we try to

    improve the ecosystem • Icinga2 GELF output for events • Docker GELF logging driver (since Docker 1.8) • apache-mod_log_gelf (beta) • log4j2-gelf • gelfclient Java library • svloggelfd (log forwarding for runit)

  35. Bernd Ahlers – Graylog, Inc. [email protected] We at Graylog <3

    structured data and you should too!

  36. Bernd Ahlers – Graylog, Inc. [email protected] Introduction: Graylog Collector •

    Reads local log files and ships them to Graylog • Windows EventLog support (limited for now) • Transport encryption via TLS • Runs on Linux, Windows, Mac OS X and AIX

  37. Bernd Ahlers – Graylog, Inc. [email protected] Why another Collector? •

    There are lots of others: nxlog, fluentd, heka, filebeat, rsyslog, syslog-ng • We want integration and centralized management of collectors in Graylog

  38. Bernd Ahlers – Graylog, Inc. [email protected]

  39. Bernd Ahlers – Graylog, Inc. [email protected] Collector Installation • OS

    packages for Linux distributions • Manual installation on Windows via ZIP file (MSI upcoming) Runs as Windows service

  40. Bernd Ahlers – Graylog, Inc. [email protected] Collector Configuration server-url =

    "http://your-graylog-server:12900" inputs { windows-application-log { type = "windows-eventlog" source-name = "Application" } } outputs { gelf-tcp { type = "gelf" host = "your-graylog-server" port = 12201 } }

  41. Bernd Ahlers – Graylog, Inc. [email protected] Collector: Current State •

    Windows EventLog support needs update to support new Windows APIs • File reading needs improvement • Centralized management needs to be implemented • :-(

  42. Bernd Ahlers – Graylog, Inc. [email protected] Tomorrow: Hackathon

  43. Bernd Ahlers – Graylog, Inc. [email protected] Thank you! Thank you

    for your time!

  44. Bernd Ahlers – Graylog, Inc. [email protected] QA Ask me anything!

    Bernd Ahlers / Graylog, Inc. [email protected] @berndahlers www.graylog.org github.com/Graylog2