Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Progression of Elasticsearch Usage @ Wayfair

Dan
February 06, 2014

Progression of Elasticsearch Usage @ Wayfair

Short Talk given at the Boston Elasticsearch Meetup
http://www.meetup.com/Elasticsearch-Boston/events/159201312/

Dan

February 06, 2014
Tweet

More Decks by Dan

Other Decks in Technology

Transcript

  1. Elasticsearch @ Wayfair February 6, 2014

  2. Name: Dan Rowe Company: Wayfair Team: Infrastructure Tools

  3. Progression of integrating Elasticsearch @ Wayfair

  4. 4 Elasticsearch integration Syslog

  5. 5 Elasticsearch integration Syslog App Logs (gelf)

  6. 6 Elasticsearch integration Syslog App Logs (gelf) Kibana 3

  7. 7 Elasticsearch integration Syslog App Logs (gelf) HUD Dashboards Zabbix

    Graphite Others Elasticsearch
  8. 8 Elasticsearch integration Syslog App Logs (gelf) Zabbix Graphite Others

    Elasticsearc h Zabbix
  9. 9 Elasticsearch integration Syslog App Logs (gelf) Zabbix Graphite Others

    Elasticsearc h Commits
  10. 10 Elasticsearch integration Syslog App Logs (gelf) Zabbix Graphite Others

    Elasticsearc h Commits Ad Hoc Query Reports
  11. 11 Elasticsearch integration Syslog App Logs (gelf) Commits Network Traffic

    Zabbix Graphite Others Elasticsearc h Ad Hoc Query Reports
  12. Usage of data in Elasticsearch @ Wayfair

  13. 13 Usages : Deployments Who’s Code is going out? HUD

    Dashboard after Deploy
  14. 14 Usages : Deployments Deploy Alert Email

  15. 15 Usages: Ad Hoc Query Alerts /** Set up adhoc

    alerting based off what you use to locate messages in kibana. * * item_name - a short name. Examples: elastic_search.syslog.sum.5min.int and elastic_search.syslog.sum.5min.string * query - the search terms you are using * threshold - number of hits in the last frequency of minutes * type - the type of message either 'gelf' or 'syslog'. php messages are gelf and logstash messages are syslog */ /** Set up adhoc facet query alerting based off what you use to locate messages in kibana. * * item_name - a short name. Examples: elastic_search.syslog.sum.5min.int and elastic_search.syslog.sum.5min.string * query - the search terms you are using * short_name - added to the item name to distinguish multiple items from multiple queries that you want in the same email * threshold - number of hits in the last frequency of minutes * type - the type of message either 'gelf' or 'syslog'. php messages are gelf and logstash messages are syslog * facet - the field you want to facet on * recipients - the email(s) * template - the name of the zabbix template you'd like these items in * additional_fields - if you need to see detial on fields other than the facet field in the email you can add them here. * additional_facet_fields - if you need to see all of another field that this error contians for example all the hosts an error is on * latest_timestamp_within - will check that the latest timestamp is within this window before includeing any hits in the email * window - defaults to frequency. This is the number of minutes to search. * email_subject - the subject to appear in the email * es_host_array - an array of es_hosts to query */
  16. 16 Usages: Issue/Error Investigation Syslog Commits Network Traffic App Logs

    (gelf) Unique Request ID Customer ID Files Involved Traffic Involved
  17. None