Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Intro to Log Management

Intro to Log Management

This presentation was co-presented by Jordan Sissel (logstash author) and Lennart Koopman (graylog2 author) to teach the basic ideas and problems in the log management space.

It was presented at OSDC 2014 in Berlin.

Elasticsearch Inc

April 09, 2014

More Decks by Elasticsearch Inc

Other Decks in Programming


  1. Log Management An Introduction Lennart Koopmann Jordan Sissel

  2. What is a Log?

  3. time + data What is a Log?

  4. - - [08/Apr/2014:15:31:30 -0400] "GET /images/web/2009/banner.png HTTP/1.1" 304 -

    "http://semicomplete.com/style2.css" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0" What is a Log? time + data
  5. 120707 0:40:34 4 Connect [email protected] on 4 Query select @@version_comment

    limit 1 120707 0:40:45 4 Query select * from mysql.user What is a Log? time + data ?????? ???????
  6. Kinds of Logs

  7. Trace and Debug Kinds of Logs

  8. Accounting Kinds of Logs

  9. Transaction Kinds of Logs

  10. Problems

  11. Difficult to Access Problems

  12. Too Many Logs Problems Difficult to Access

  13. Too Many Servers Problems Difficult to Access

  14. No Permissions :( Problems Difficult to Access

  15. Difficult to Consume Problems

  16. Unstructured Problems Difficult to Consume

  17. Requires Expertise Problems Difficult to Consume

  18. Requires Maintenance Problems

  19. Configuration Problems Requires Maintenance

  20. Log Retention Problems Requires Maintenance

  21. Bad Tooling Problems

  22. grep, ssh, awk Problems Bad Tooling

  23. Life of a Log

  24. Life of a Log Record Transport Search & Analyze Archive

  25. Sources of Logs

  26. Vendor Hardware Sources of Logs Routers, VPNs, Printers, Phones, AWS

    CloudTrail, etc
  27. Vendor Software Sources of Logs Nginx, Wordpress, Jira

  28. In-house Software Sources of Logs Your company controls it

  29. Solutions (Open Source!)

  30. Solutions Logstash from Elasticsearch Graylog2 from Torch