Structured Logging & Introduction to Graylog Collector

Transcript

1. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Monitoring Linux and Windows

   Logs with Graylog Collector Bernd Ahlers Graylog, Inc.

2. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Structured Logging & Introduction

   to Graylog Collector Bernd Ahlers Graylog, Inc.

3. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Introduction: Graylog • Open

   source log management platform • Collect, index and analyze structured and unstructured log data • Alerts based on log data • Extensible via custom plugins

4. Bernd Ahlers – Graylog, Inc. bernd@graylog.com

5. Bernd Ahlers – Graylog, Inc. bernd@graylog.com

6. Bernd Ahlers – Graylog, Inc. bernd@graylog.com

7. Bernd Ahlers – Graylog, Inc. bernd@graylog.com

8. Bernd Ahlers – Graylog, Inc. bernd@graylog.com

9. Bernd Ahlers – Graylog, Inc. bernd@graylog.com

10. Bernd Ahlers – Graylog, Inc. bernd@graylog.com

11. Bernd Ahlers – Graylog, Inc. bernd@graylog.com

12. Bernd Ahlers – Graylog, Inc. bernd@graylog.com More about Graylog •

    www.graylog.org • marketplace.graylog.org • docs.graylog.org • github.com/Graylog2

13. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Why are we writing

    logs? • Getting insight & collecting business metrics • Debugging problems • Building an audit trail • Monitoring

14. Bernd Ahlers – Graylog, Inc. bernd@graylog.com How do we access

    our logs? • Applications write to local files • SSH into machines • tail, grep, awk • If lucky: central log management

15. Bernd Ahlers – Graylog, Inc. bernd@graylog.com What do they look

    like? • Syslog RFC 3164 (BSD) • Syslog RFC 5424

16. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Syslog RFC 3164 (BSD)

    Nov 10 15:55:01 tumbler CRON[2684]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)

17. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Syslog RFC 5424 2003-10-11T22:14:15.003Z

    mymachine.example.com evntslog - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] BOMAn application event log entry...

18. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Apache 127.0.0.1 - bernd

    [28/Dec/2014:06:43:15 +0100] "PROPFIND /remote.php/webdav/ HTTP/1.1" 207 910 "-" "Mozilla/5.0 (Linux) mirall/1.7.1"

19. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Postfix Aug 5 17:05:26

    hostname postfix/qmgr[308]: A44F828C71: from=<bamm@example.com>, size=153136, nrcpt=1 (queue active)

20. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Squid sq18.wikimedia.org 1715898 2010-12-

    01T21:57:22.331 0 1.2.3.4 TCP_MEM_HIT/200 13208 GET http://en.wikipedia.org/wiki/Main_Page NONE/- text/html - - Mozilla/4.0%20(compatible;%20MSIE %206.0;%20Windows%20NT%205.1;%20.NET%20CLR %201.1.4322) en-US -

21. Bernd Ahlers – Graylog, Inc. bernd@graylog.com log4j 0 [main] INFO

    MyApp - Entering application. 36 [main] DEBUG com.foo.Bar - Did it again! 51 [main] INFO MyApp - Exiting application.

22. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Ruby Logger I, [2015-11-18T00:16:27.723972

    #3609] INFO -- : Hello world!

23. Bernd Ahlers – Graylog, Inc. bernd@graylog.com #1 Problem: Timestamps •

    Everyone likes to invent one • Missing most of the time: timezone, year

24. Bernd Ahlers – Graylog, Inc. bernd@graylog.com How to get value

    out of unstructured logs? • Regex • More regex • Even more regex

25. Bernd Ahlers – Graylog, Inc. bernd@graylog.com ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(: [0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d| 1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}) {1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-

    9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((: [0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0- 4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f] {1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1- 9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f] {1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0- 5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)) {3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A- Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d| 1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f] {1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d| 1\d\d|[1-9]?\d)){3}))|:)))(%.+)?

26. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Grok IPV6 ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9... USERNAME

    [a-zA-Z0-9._-]+ USER %{USERNAME} HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A- Za-z-]{0,62}))*(\.?|\b) EMAILLOCALPART [a-zA-Z][a-zA-Z0-9_.+-=:]+ EMAILADDRESS %{EMAILLOCALPART}@%{HOSTNAME} ... COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}

27. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Graylog: Extractors • Regular

    expressions based • Extracts data into message fields

28. Bernd Ahlers – Graylog, Inc. bernd@graylog.com

29. Bernd Ahlers – Graylog, Inc. bernd@graylog.com How to fix this?

    • Central log collection (Graylog, ELK, others) • Use structured log formats – Structured Syslog RFC 5424 – CEF Format – GELF – JSON

30. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Structured Syslog RFC 5424

    2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] BOMAn application event log entry...

31. Bernd Ahlers – Graylog, Inc. bernd@graylog.com CEF by ArcSight/HP Sep

    19 08:26:10 host CEF:0|HP|siem| 1.0|100|service successfully stopped|10| src=10.0.0.1 dst=2.1.2.2 spt=1232

32. Bernd Ahlers – Graylog, Inc. bernd@graylog.com GELF { "version": "1.1",

    "timestamp": 1385053862.3072, "host": "example.org", "short_message": "A short message", "full_message": "Backtrace here\n\nmore stuff", "level": 1, "_user_id": 9001, "_some_info": "foo", "_some_env_var": "bar"}

33. Bernd Ahlers – Graylog, Inc. bernd@graylog.com JSON { "source": "example.org",

    "message": "A log message", "timestamp": "2015-11-15T10:43:21Z", "user_id": 9001, "http_method": "GET"}

34. Bernd Ahlers – Graylog, Inc. bernd@graylog.com How we try to

    improve the ecosystem • Icinga2 GELF output for events • Docker GELF logging driver (since Docker 1.8) • apache-mod_log_gelf (beta) • log4j2-gelf • gelfclient Java library • svloggelfd (log forwarding for runit)

35. Bernd Ahlers – Graylog, Inc. bernd@graylog.com We at Graylog <3

    structured data and you should too!

36. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Introduction: Graylog Collector •

    Reads local log files and ships them to Graylog • Windows EventLog support (limited for now) • Transport encryption via TLS • Runs on Linux, Windows, Mac OS X and AIX

37. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Why another Collector? •

    There are lots of others: nxlog, fluentd, heka, filebeat, rsyslog, syslog-ng • We want integration and centralized management of collectors in Graylog

38. Bernd Ahlers – Graylog, Inc. bernd@graylog.com

39. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Collector Installation • OS

    packages for Linux distributions • Manual installation on Windows via ZIP file (MSI upcoming) Runs as Windows service

40. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Collector Configuration server-url =

    "http://your-graylog-server:12900" inputs { windows-application-log { type = "windows-eventlog" source-name = "Application" } } outputs { gelf-tcp { type = "gelf" host = "your-graylog-server" port = 12201 } }

41. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Collector: Current State •

    Windows EventLog support needs update to support new Windows APIs • File reading needs improvement • Centralized management needs to be implemented • :-(

42. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Tomorrow: Hackathon

43. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Thank you! Thank you

    for your time!

44. Bernd Ahlers – Graylog, Inc. bernd@graylog.com QA Ask me anything!

    Bernd Ahlers / Graylog, Inc. bernd@graylog.com @berndahlers www.graylog.org github.com/Graylog2