What's the 411? Building Alerts on Elasticsearch at Etsy

When working with a web application, you may find yourself drowning in logs. Some of this data is vital for debugging your application, and some can be a rich source of data for alerting. This talk will cover how Etsy constructs and responds to alert queries, as well as offer up ideas for additional types of alerts.

Transcript

1. Kai Zhong Security Engineer @ Etsy @sixhundredns 411: Automated alerts on Elasticsearch
2. •  LAMP stack •  Continuous Deployment –  Tests –  Feature flags –  Logs
3. Main Elasticsearch cluster •  Types –  Access logs –  Application logs –  Error logs •  3,000,000,000 lines/day •  30 day log retention •  211TB data LOG ALL THE THINGS
4. •  Heavily depend on alerting •  Moved to ES in mid 2014 •  We wanted –  Concise query syntax –  Automatic query scheduling •  No good options at the time ALERT ON ALL THE THINGS
5. Search scheduling Alert management
6. Scheduling
7. Searches: Automatically query a data source and return information Types •  Ping –  Check the reachability of a host •  HTTP –  Check the response code of an URL •  Logstash –  Retrieve results from Elasticsearch
8. Filters: Remove matching Alerts Types •  Regex –  Filter Alerts matching a regex •  Dedupe –  Filter Alerts that have been seen recently •  Throttle –  Filter Alerts that occur frequently
9. Targets: Send Alerts to external services Types •  WebHook –  Send Alerts to an HTTP endpoint •  Notification –  Send Alerts to an (extra) email address
10. Alert Pipeline Search Targets Filters ./search  |  filter1  |  filter2  |  tee  target1  target2   Alerts Alerts
11. Searches Fields •  Query –  The query to execute •  Frequency –  How often to schedule the query •  Assignee –  User/Group responsible for these Alerts •  Priority –  How important these Alerts are
12. Alert Emails Priority Email Frequency High Immediately Medium Hourly Rollup Low Never
13. Frontend
14. Dashboard •  Summary of active alerts •  Historical alert information
15. User management •  Manage users –  Create –  Modify –  Delete
16. Group management •  Manage groups –  Create –  Modify –  Delete
17. Searches page •  Manage searches –  Create –  Enable/Disable –  View Health
18. Search page •  Manage a search –  Modify –  Delete –  Test –  Execute –  Configure Filters/Targets •  View statistics •  Changelog
19. Alerts: Are actionable events Actions •  Escalation –  Promotes an Alert to high priority •  Assignment –  Sets a new Assignee for an Alert •  Resolution –  Marks an Alert as finished
20. •  Filters Alerts •  Manage Alerts –  Escalate/De-escalate –  Assign –  Mark New/In Progress/Resolved –  Add Note Alerts page
21. Alert page •  Manage Alert –  Escalate/De-escalate –  Assign –  Mark New/In Progress/Resolved –  Add Note •  View changelog
22. ES_Proxy Pipelined Lucene shorthand
23. Command Syntax Joins *  |  join  source:src_ip  target:dst_ip   Aggregations *  |  agg:terms  field:src_ip      |  agg:terms  field:user_id   Transactions *  |  trans  field:request_uuid   Lists src_ip:@internal_ips   Features
24. Logstash Search page Fields: •  Time Range –  How far back to query •  Result Type –  The type of data to return •  Result Filter –  Only return results if the result set matches a condition
25. Demo
26. Search Ideas •  Spike in HTTP 500 responses •  POSTs with a referrer from another site •  Odd HTTP verbs •  Googlebot useragent from non-Google IP •  Requests from known bad IPs •  Sign-ins from unusual locations
27. Thanks Emily Sommer Ken Lee Avleen Vig Security Operations
29. Questions? https://github.com/Etsy/411 Kai Zhong kzhong@etsy.com @sixhundredns