Intro to Log Management

This presentation was co-presented by Jordan Sissel (logstash author) and Lennart Koopman (graylog2 author) to teach the basic ideas and problems in the log management space.

It was presented at OSDC 2014 in Berlin.

Transcript

1. Log Management An Introduction Lennart Koopmann Jordan Sissel
2. What is a Log?
3. time + data What is a Log?
4. 37.5.55.31 - - [08/Apr/2014:15:31:30 -0400] "GET /images/web/2009/banner.png HTTP/1.1" 304 - "http://semicomplete.com/style2.css" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0" What is a Log? time + data
5. 120707 0:40:34 4 Connect root@localhost on 4 Query select @@version_comment limit 1 120707 0:40:45 4 Query select * from mysql.user What is a Log? time + data ?????? ???????
6. Kinds of Logs
7. Trace and Debug Kinds of Logs
8. Accounting Kinds of Logs
9. Transaction Kinds of Logs
10. Problems
11. Difficult to Access Problems
12. Too Many Logs Problems Difficult to Access
13. Too Many Servers Problems Difficult to Access
14. No Permissions :( Problems Difficult to Access
15. Difficult to Consume Problems
16. Unstructured Problems Difficult to Consume
17. Requires Expertise Problems Difficult to Consume
18. Requires Maintenance Problems
19. Configuration Problems Requires Maintenance
20. Log Retention Problems Requires Maintenance
21. Bad Tooling Problems
22. grep, ssh, awk Problems Bad Tooling
23. Life of a Log
24. Life of a Log Record Transport Search & Analyze Archive Delete
25. Sources of Logs
26. Vendor Hardware Sources of Logs Routers, VPNs, Printers, Phones, AWS CloudTrail, etc
27. Vendor Software Sources of Logs Nginx, Wordpress, Jira
28. In-house Software Sources of Logs Your company controls it
29. Solutions (Open Source!)
30. Solutions Logstash from Elasticsearch Graylog2 from Torch