Splunk: Interrogate your logs

Introduction to Splunk search, extraction, visualization and alerting for developers.

Transcript

1. Splunk Interrogate your logs Friday, August 30, 13
2. What is Splunk ? Aggregates all your logs in one place Search Extract meaning Group, join, pivot and format results Visualization Email alerts Friday, August 30, 13
3. OH, SHINY HTTPS://SPLUNK.PROD.HEARSAYLABS.COM/EN-US/APP/SEARCH/BROWNBAG Friday, August 30, 13
4. Architecture How does it work? Friday, August 30, 13
5. IT’S ALL SETUP ALREADY - THANKS ADAM BRING YOUR OWN TIE Friday, August 30, 13
6. Hosts and Sources Same bits, different way Friday, August 30, 13
7. Friday, August 30, 13
8. Search Grep meets Mint.com Friday, August 30, 13
9. SEARCH DASHBOARD FIND ALL THE THINGS Friday, August 30, 13
10. Search Syntax sourcetype="hsl-prod-fe" "Chase Seibert" sourcetype="hsl-prod-fe" e186f85c914261eec9e54d3767fdd3cc BEGIN sourcetype="hsl-prod-crawl" |regex _raw="fanmgmt\.(analytics|metrics)" sourcetype="hsl-prod-crawl" facebook OR twitter NOT linkedin sourcetype="hsl-prod-crawl" facebook OR linkedin OR twitter earliest=-24h It’s easier to use the UI for this --> Friday, August 30, 13
11. Example: find publishes Friday, August 30, 13
12. Extract Bring order to chaos Friday, August 30, 13
13. Example: extract org_id Friday, August 30, 13
14. Lazy Mode! Just format values in your log like: foo org_id=1 asset_id=2 bar It also helps to add something unique to filter on. Friday, August 30, 13
15. Pipe Syntax sourcetype="hsl-prod-crawl" succeeded |stats count perc95(task_seconds) by python_module |sort count desc |head 10 Friday, August 30, 13
16. More Pipe Commands | uniq | script python myscript myarg1 myarg2 | bucket _time span=5m | eval name=coalesce(firstName, lastName) | rare, | anomalous | spath output=commit_author path=commits.author.name # extract xml/json values Friday, August 30, 13
17. More Functions avg() | eval description=case(error == 404, "Not found", error == 200, "OK") floor(), ceiling() len() isbool(), isint(), etc upper(), lower() trim(), ltrim(), rtrim() md5() now() random() replace() split() substr() Friday, August 30, 13
18. Example: gaps in publishes Friday, August 30, 13
19. Visualizations The goggles they do nothing! Friday, August 30, 13
20. Point and click graphs Explore top field values over time Friday, August 30, 13
21. Example: publishes by org Friday, August 30, 13
22. Alerts Who doesn’t want to get more email? Friday, August 30, 13
23. Email Alerts Starts with a regular search Give the search a time window Schedule it (supports cron syntax) Can alert always, or when num rows > N Can remember not to re-alert on the same items Sends an email Friday, August 30, 13
24. EXAMPLE ALERT EMAIL SEARCH STRING, LINK TO RESULTS AND ACTUAL RESULT ROWS Friday, August 30, 13
25. Example: publishing gaps Friday, August 30, 13
26. Next Steps Get a login (ask Adam) http://docs.splunk.com/Documentation/Splunk http://answers.splunk.com/ http://www.splunk.com/view/education-videos Friday, August 30, 13