Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Structured Logging & Introduction to Graylog Co...

Avatar for Bernd Ahlers Bernd Ahlers
November 20, 2015
0
460

Structured Logging & Introduction to Graylog Collector

Structured logging and an introduction to Graylog Collector. OSMC 2015 (open source monitoring conference)
Avatar for Bernd Ahlers

Bernd Ahlers

November 20, 2015
Tweet

More Decks by Bernd Ahlers

See All by Bernd Ahlers
Zentrales Log-Management mit Graylog (german)
Avatar for Bernd Ahlers bernd
0
630
What is your configuration management doing?
Avatar for Bernd Ahlers bernd
0
68
Get the best out of Graylog2 & Icinga 2
Avatar for Bernd Ahlers bernd
0
800
fpm-cookery RUGHH Oct 2011
Avatar for Bernd Ahlers bernd
0
110

Featured

See All Featured
Fireside Chat
Avatar for paigeccino paigeccino
37
3.5k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
Avatar for Vitaly Friedman smashingmag
252
21k
Learning to Love Humans: Emotional Interface Design
Avatar for Aarron Walter aarron
273
40k
JavaScript: Past, Present, and Future - NDC Porto 2020
Avatar for David Neal reverentgeek
48
5.4k
Embracing the Ebb and Flow
Avatar for Simon Collison colly
86
4.7k
Raft: Consensus for Rubyists
Avatar for Patrick Van Stee vanstee
140
7k
Rebuilding a faster, lazier Slack
Avatar for samanthasiow samanthasiow
82
9.1k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
Avatar for Eileen M. Uchitelle eileencodes
138
34k
Building Adaptive Systems
Avatar for Chris Keathley keathley
43
2.6k
Designing Experiences People Love
Avatar for Jonathan Moore moore
142
24k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
Avatar for Marc Duiker marcduiker
30
2.1k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
Avatar for Rebecca Miller-Webster rmw
34
3.1k

Transcript

  1. Bernd Ahlers – Graylog, Inc. [email protected] Monitoring Linux and Windows

    Logs with Graylog Collector Bernd Ahlers Graylog, Inc.

  2. Bernd Ahlers – Graylog, Inc. [email protected] Structured Logging & Introduction

    to Graylog Collector Bernd Ahlers Graylog, Inc.

  3. Bernd Ahlers – Graylog, Inc. [email protected] Introduction: Graylog • Open

    source log management platform • Collect, index and analyze structured and unstructured log data • Alerts based on log data • Extensible via custom plugins

  4. Bernd Ahlers – Graylog, Inc. [email protected]

  5. Bernd Ahlers – Graylog, Inc. [email protected]

  6. Bernd Ahlers – Graylog, Inc. [email protected]

  7. Bernd Ahlers – Graylog, Inc. [email protected]

  8. Bernd Ahlers – Graylog, Inc. [email protected]

  9. Bernd Ahlers – Graylog, Inc. [email protected]

  10. Bernd Ahlers – Graylog, Inc. [email protected]

  11. Bernd Ahlers – Graylog, Inc. [email protected]

  12. Bernd Ahlers – Graylog, Inc. [email protected] More about Graylog •

    www.graylog.org • marketplace.graylog.org • docs.graylog.org • github.com/Graylog2

  13. Bernd Ahlers – Graylog, Inc. [email protected] Why are we writing

    logs? • Getting insight & collecting business metrics • Debugging problems • Building an audit trail • Monitoring

  14. Bernd Ahlers – Graylog, Inc. [email protected] How do we access

    our logs? • Applications write to local files • SSH into machines • tail, grep, awk • If lucky: central log management

  15. Bernd Ahlers – Graylog, Inc. [email protected] What do they look

    like? • Syslog RFC 3164 (BSD) • Syslog RFC 5424

  16. Bernd Ahlers – Graylog, Inc. [email protected] Syslog RFC 3164 (BSD)

    Nov 10 15:55:01 tumbler CRON[2684]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)

  17. Bernd Ahlers – Graylog, Inc. [email protected] Syslog RFC 5424 2003-10-11T22:14:15.003Z

    mymachine.example.com evntslog - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] BOMAn application event log entry...

  18. Bernd Ahlers – Graylog, Inc. [email protected] Apache 127.0.0.1 - bernd

    [28/Dec/2014:06:43:15 +0100] "PROPFIND /remote.php/webdav/ HTTP/1.1" 207 910 "-" "Mozilla/5.0 (Linux) mirall/1.7.1"

  19. Bernd Ahlers – Graylog, Inc. [email protected] Postfix Aug 5 17:05:26

    hostname postfix/qmgr[308]: A44F828C71: from=<[email protected]>, size=153136, nrcpt=1 (queue active)

  20. Bernd Ahlers – Graylog, Inc. [email protected] Squid sq18.wikimedia.org 1715898 2010-12-

    01T21:57:22.331 0 1.2.3.4 TCP_MEM_HIT/200 13208 GET http://en.wikipedia.org/wiki/Main_Page NONE/- text/html - - Mozilla/4.0%20(compatible;%20MSIE %206.0;%20Windows%20NT%205.1;%20.NET%20CLR %201.1.4322) en-US -

  21. Bernd Ahlers – Graylog, Inc. [email protected] log4j 0 [main] INFO

    MyApp - Entering application. 36 [main] DEBUG com.foo.Bar - Did it again! 51 [main] INFO MyApp - Exiting application.

  22. Bernd Ahlers – Graylog, Inc. [email protected] Ruby Logger I, [2015-11-18T00:16:27.723972

    #3609] INFO -- : Hello world!

  23. Bernd Ahlers – Graylog, Inc. [email protected] #1 Problem: Timestamps •

    Everyone likes to invent one • Missing most of the time: timezone, year

  24. Bernd Ahlers – Graylog, Inc. [email protected] How to get value

    out of unstructured logs? • Regex • More regex • Even more regex

  25. Bernd Ahlers – Graylog, Inc. [email protected] ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(: [0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d| 1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}) {1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-

    9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((: [0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0- 4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f] {1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1- 9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f] {1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0- 5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)) {3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A- Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d| 1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f] {1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d| 1\d\d|[1-9]?\d)){3}))|:)))(%.+)?

  26. Bernd Ahlers – Graylog, Inc. [email protected] Grok IPV6 ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9... USERNAME

    [a-zA-Z0-9._-]+ USER %{USERNAME} HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A- Za-z-]{0,62}))*(\.?|\b) EMAILLOCALPART [a-zA-Z][a-zA-Z0-9_.+-=:]+ EMAILADDRESS %{EMAILLOCALPART}@%{HOSTNAME} ... COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}

  27. Bernd Ahlers – Graylog, Inc. [email protected] Graylog: Extractors • Regular

    expressions based • Extracts data into message fields

  28. Bernd Ahlers – Graylog, Inc. [email protected]

  29. Bernd Ahlers – Graylog, Inc. [email protected] How to fix this?

    • Central log collection (Graylog, ELK, others) • Use structured log formats – Structured Syslog RFC 5424 – CEF Format – GELF – JSON

  30. Bernd Ahlers – Graylog, Inc. [email protected] Structured Syslog RFC 5424

    2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] BOMAn application event log entry...

  31. Bernd Ahlers – Graylog, Inc. [email protected] CEF by ArcSight/HP Sep

    19 08:26:10 host CEF:0|HP|siem| 1.0|100|service successfully stopped|10| src=10.0.0.1 dst=2.1.2.2 spt=1232

  32. Bernd Ahlers – Graylog, Inc. [email protected] GELF { "version": "1.1",

    "timestamp": 1385053862.3072, "host": "example.org", "short_message": "A short message", "full_message": "Backtrace here\n\nmore stuff", "level": 1, "_user_id": 9001, "_some_info": "foo", "_some_env_var": "bar"}

  33. Bernd Ahlers – Graylog, Inc. [email protected] JSON { "source": "example.org",

    "message": "A log message", "timestamp": "2015-11-15T10:43:21Z", "user_id": 9001, "http_method": "GET"}

  34. Bernd Ahlers – Graylog, Inc. [email protected] How we try to

    improve the ecosystem • Icinga2 GELF output for events • Docker GELF logging driver (since Docker 1.8) • apache-mod_log_gelf (beta) • log4j2-gelf • gelfclient Java library • svloggelfd (log forwarding for runit)

  35. Bernd Ahlers – Graylog, Inc. [email protected] We at Graylog <3

    structured data and you should too!

  36. Bernd Ahlers – Graylog, Inc. [email protected] Introduction: Graylog Collector •

    Reads local log files and ships them to Graylog • Windows EventLog support (limited for now) • Transport encryption via TLS • Runs on Linux, Windows, Mac OS X and AIX

  37. Bernd Ahlers – Graylog, Inc. [email protected] Why another Collector? •

    There are lots of others: nxlog, fluentd, heka, filebeat, rsyslog, syslog-ng • We want integration and centralized management of collectors in Graylog

  38. Bernd Ahlers – Graylog, Inc. [email protected]

  39. Bernd Ahlers – Graylog, Inc. [email protected] Collector Installation • OS

    packages for Linux distributions • Manual installation on Windows via ZIP file (MSI upcoming) Runs as Windows service

  40. Bernd Ahlers – Graylog, Inc. [email protected] Collector Configuration server-url =

    "http://your-graylog-server:12900" inputs { windows-application-log { type = "windows-eventlog" source-name = "Application" } } outputs { gelf-tcp { type = "gelf" host = "your-graylog-server" port = 12201 } }

  41. Bernd Ahlers – Graylog, Inc. [email protected] Collector: Current State •

    Windows EventLog support needs update to support new Windows APIs • File reading needs improvement • Centralized management needs to be implemented • :-(

  42. Bernd Ahlers – Graylog, Inc. [email protected] Tomorrow: Hackathon

  43. Bernd Ahlers – Graylog, Inc. [email protected] Thank you! Thank you

    for your time!

  44. Bernd Ahlers – Graylog, Inc. [email protected] QA Ask me anything!

    Bernd Ahlers / Graylog, Inc. [email protected] @berndahlers www.graylog.org github.com/Graylog2