Progression of Elasticsearch Usage @ Wayfair

Short Talk given at the Boston Elasticsearch Meetup
http://www.meetup.com/Elasticsearch-Boston/events/159201312/

Transcript

1. Elasticsearch @ Wayfair February 6, 2014
2. Name: Dan Rowe Company: Wayfair Team: Infrastructure Tools
3. Progression of integrating Elasticsearch @ Wayfair
4. 4 Elasticsearch integration Syslog
5. 5 Elasticsearch integration Syslog App Logs (gelf)
6. 6 Elasticsearch integration Syslog App Logs (gelf) Kibana 3
7. 7 Elasticsearch integration Syslog App Logs (gelf) HUD Dashboards Zabbix Graphite Others Elasticsearch
8. 8 Elasticsearch integration Syslog App Logs (gelf) Zabbix Graphite Others Elasticsearc h Zabbix
9. 9 Elasticsearch integration Syslog App Logs (gelf) Zabbix Graphite Others Elasticsearc h Commits
10. 10 Elasticsearch integration Syslog App Logs (gelf) Zabbix Graphite Others Elasticsearc h Commits Ad Hoc Query Reports
11. 11 Elasticsearch integration Syslog App Logs (gelf) Commits Network Traffic Zabbix Graphite Others Elasticsearc h Ad Hoc Query Reports
12. Usage of data in Elasticsearch @ Wayfair
13. 13 Usages : Deployments Who’s Code is going out? HUD Dashboard after Deploy
14. 14 Usages : Deployments Deploy Alert Email
15. 15 Usages: Ad Hoc Query Alerts /** Set up adhoc alerting based off what you use to locate messages in kibana. * * item_name - a short name. Examples: elastic_search.syslog.sum.5min.int and elastic_search.syslog.sum.5min.string * query - the search terms you are using * threshold - number of hits in the last frequency of minutes * type - the type of message either 'gelf' or 'syslog'. php messages are gelf and logstash messages are syslog */ /** Set up adhoc facet query alerting based off what you use to locate messages in kibana. * * item_name - a short name. Examples: elastic_search.syslog.sum.5min.int and elastic_search.syslog.sum.5min.string * query - the search terms you are using * short_name - added to the item name to distinguish multiple items from multiple queries that you want in the same email * threshold - number of hits in the last frequency of minutes * type - the type of message either 'gelf' or 'syslog'. php messages are gelf and logstash messages are syslog * facet - the field you want to facet on * recipients - the email(s) * template - the name of the zabbix template you'd like these items in * additional_fields - if you need to see detial on fields other than the facet field in the email you can add them here. * additional_facet_fields - if you need to see all of another field that this error contians for example all the hosts an error is on * latest_timestamp_within - will check that the latest timestamp is within this window before includeing any hits in the email * window - defaults to frequency. This is the number of minutes to search. * email_subject - the subject to appear in the email * es_host_array - an array of es_hosts to query */
16. 16 Usages: Issue/Error Investigation Syslog Commits Network Traffic App Logs (gelf) Unique Request ID Customer ID Files Involved Traffic Involved