Browse Raw Logs in One Place: Open Source Plug-in for Kibana

Kibana lets you visualize your Elasticsearch log data, but being able to effectively browse, scroll, and sift through the raw log files is critical for IT teams and developers to identify errors, perform root cause analysis, and troubleshoot. A new open source Kibana plug-in from Search Technologies lets you do just this.

Originally created for one of the United States’ largest retailers, Paul Nelson will discuss and demo the advantage of this open source plug-in that improves security and performance, and enables log browsing across multiple machines in a single view.

Paul Nelson l Chief Architect l Search Technologies

Transcript

1. 1 Browse all your Logs in one place: Open Source Log Browser UI Paul Nelson (with thanks to Esteban Alvarado) Chief Architect, Search Technologies pnelson@searchtechnologies.com June 4, 2016
2. 2 205+ Search & Big Data Consultants Worldwide San Diego San Jose, CR Cincinnati Manila, PH Washington (HQ) • Founded 2005 • Deep search expertise • 900+ customers worldwide • Consistent profitability • Search engines & Big Data • Vendor independent London, UK Frankfurt, DE Prague, CZ
3. 3 Elasticsearch Services • Planning, Installation, Configuration • Extreme Scaling • Accuracy Improvements / Engine Scoring • Data preparation • Log Analytics • New Search Operators (plug-ins) • Query Business Rules • Kibana Plug-in Development • Log Browsing
4. 4 Mostly Services – But We have Core Engineering Too
5. 5 Log Browser
6. 6 Why? • Log Lines in Context • Replicate Unix-Style browsing in a web browser – Browse log lines like you would using “more” or “vi” • More friendly for developers • More secure: – No need to log onto the servers and browse logs locally • Browse Combined Logs – Browse multiple logs from multiple servers together – Especially important for clustered arrays of machines • Intended for the troubleshooting use case – Find a log line, browse it in context, scroll up and down
7. 7 Open Source! • For more information: http://www.searchtechnologies.com/kibana-log-browser-plugin • GitHub URL: [WAIT UNTIL THE END OF THE PRESENTATION] • Requirements: – Elasticsearch 5.2 • Older versions may work, not tested – Kibana 5.2 • Older versions may work, not tested • This is Version 1.0.0 !! – And all that entails J
8. 8 Index Field Requirements • “message” (text) – Holds the text of the log line – Can be multi-line, tokenized for easy search • “message40” (keyword) – Holds the first 40 characters of the message – For sorting • “@timestamp” (date) – The ingestion time of the log line – Automatically added by logstash • “host” (keyword)– The server host which gathered the log – IP or hostname • “log_time” (date)– The time parsed out of the log line • “type” (keyword)– The type of log file – Used to browse log files of a particular type – Examples: apache, nginx, myapp1, myapp2, etc. • “source” (keyword) - The filename path of the log file on the server
9. 9 Installation • Shutdown Kibana • Install from Github: • Restart Kibana • Go to the plug-in: kibana-plugin install https://github.com/[WAIT-UNTIL-THE-END-OF-THE-PRESENTATION]/releases/download/1.0.0/log_browser.zip
10. 10 Demo
11. 11 Choose Log Files
14. 14 What’s Next? Future Work Latest updates: http://www.searchtechnologies.com/kibana-log-browser-plugin • “Browse right away” – Browse results at the top of the list right away – Acquire additional results only as needed • Jump back and forth to Kibana Dashboards – Show log lines in dashboard, then click to jump to log browser • Space bar to page, better paging controls – Similar to “more” command • Dynamic tail
15. 15 For More Information Is there a feature you want on the log browser UI? Come to our Booth! We’ll be happy to add it.
16. 16 For More Information For the latest updates, roadmap: http://www.searchtechnologies.com/kibana-log-browser-plugin For development help on the log browser UI: http://www.searchtechnologies.com/contacts The Github Repository: https://github.com/searchtechnologies/kibana-logbrowser
17. 17 Thank you!! Questions? Answers!