Elk Wrestling

I cover how we've been using Logstash, elasticsearch and Kibana to take control of our logging and start making it work for us. Spoke at DevOps Manchester 29th April 2014.

Transcript

1. { title: ‘ELK Wrestling’, author: ‘Steve Elliott’, company: ‘LateRooms.com’, type: ‘DevOpsManc’, @timestamp: ‘2014-04-29T17:30Z’ }
2. Home growing a metrics culture Needed visibility of live issues Had trialled off the shelf before (Splunk) Hadn’t gained traction Wanted the data still
3. It started with Badger...
4. Logging and Monitoring Project Locate and implement the tools we needed Started with Cube for metrics (wouldn’t recommend) Moved onto Logging
5. Current tooling... ...Lacking
6. “But it works”
7. What can we log? Pretty much anything with a timestamp Error log Web logs Proxy logs Releases? Tweets?
8. Logstash ELK
9. High level architectural design Web servers Queue Elasticsearch Dashboards Rest of Badger
10. Real time search and analytics database
11. Who’s using it? ...Clever people Certain other hotel website...
12. Working with Elasticsearch • RESTful API • JSON • Many libraries to deal with it (new on ElasticLinq for C#)
13. Sense Chrome Extension
14. Clustering Excellent distributed features Clusters like a pro Node Self discovery
15. More in depth architecture IIS Logs Errors WMI Collector (e.g. Live Server) Queue Forwarder Cube Search Analytics Rabbit MQ Filter & Forward
16. Logstash Inputs Filters Outputs e.g.HTTP logs, UDP, error logs, tweets. e.g. UDP, elasticsearch, graphite, IRC (e.g. Filter, grok, lookup IP, magic…)
17. Filter Configuration Grok Example… Patterns (stored in files): grok { match => [ "url", "%{URIPATH:url_path}%{URIPARAM:url_querystring}?" ] } Regex: grok { match => [ "url_path", "/(?<url_language>[a-zA-Z]{2})(?:/p(?<url_partner>[0-9]+))?(?:/pv(? <url_partner_value>[0-9a-zA-Z]+))?(?<url_page>/.+)" ] }
18. Why the Queue? • Resiliancy • Single source of data for everyone • Logstash used to recommend RabbitMQ, now they recommend Redis • We still use RabbitMQ, works for us
19. Kibana • Easy to build dashboards • Gateway drug to ElasticSearch queries • Examples!
23. Demo
24. Mistake #1 Should have used “normal” logstash more
25. More node More awesome??
26. Node Logstash vs Logstash Classic • Node logstash lends itself to easier installs - easiest to get up and running first • Ruby logstash used on varnish servers • Switching to Ruby logstash for infrastructure, will continue to use node logstash for collectors
27. Mistake #2 Should have automated sooner
28. High Barrier to Entry • No automation, meant you needed to know what to configure/where • Working to combat configuration • Also worked to lower barrier to integration - Nuget Package for everyone to use
29. Chef! (or puppet) I’ve used chef, sure puppet works as well • https://github.com/elasticsearch/cookbook- elasticsearch (It’s a bit “funny” about setting version) • https://github.com/lusis/chef-logstash • https://github.com/lusis/chef-kibana (Though installing kibana via remote file and nginx is easy enough)
31. Thanks for Listening! More: elasticsearch.org, logstash.net Blog: www.tegud.net Twitter: @tegud Github: www.github.com/tegud Come say hi!