Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building Effective Security Alerting

Kennysan
August 05, 2016

Building Effective Security Alerting

Code available here: fouroneone.io

Kennysan

August 05, 2016
Tweet

More Decks by Kennysan

Other Decks in Technology

Transcript

  1. KZER Kai Zhong Product Security Engineer @ Etsy Loves tea,

    cats and netbooks Twitter: @sixhundredns
  2. KLEE Ken Lee Senior Product Security Engineer @ Etsy Spoke

    at Defcon 21 about Content Security Policy Loves funny cat gifs Twitter: @Kennysan
  3. Logstash Data processor and log shipper Allows you to break

    out your log data into separate fields
  4. Logstash Data processor and log shipper Allows you to break

    out your log data into separate fields We use it to ship logs into Elasticsearch!
  5. Elasticsearch Distributed, real-time search engine Allows storing complex, nested documents

    Allows generating statistics over your data We use it for analyzing logs!
  6. Switching to ELK Work started in mid 2014 Finished in

    mid 2015 We learned a lot from the migration
  7. Switching to ELK Work started in mid 2014 Finished in

    mid 2015 We learned a lot from the migration And got a bunch of great tools out of it
  8. It Was A Bumpy Road Hiccups are expected when moving

    to a new technology Had to deal with annoying, performance-impacting bugs
  9. It Was A Bumpy Road Hiccups are expected when moving

    to a new technology Had to deal with annoying, performance-impacting bugs Issues with SSDs, kernel-level bugs
  10. It Was A Bumpy Road Hiccups are expected when moving

    to a new technology Had to deal with annoying, performance-impacting bugs Issues with SSDs, kernel-level bugs Security needed an alerting solution
  11. Features Superset of the standard Lucene syntax Syntactically similar to

    SPL! Supports all the functionality we need!!!
  12. Syntax Command Syntax Inline params $size:20 $sort:user_id $fields:[a,b,c] Joins *

    | join source:src_ip target:dst_ip Aggregations * | agg:terms field:src_ip | agg:terms field:user_id Variable substitution src_ip:@internal_ips
  13. { "query": { "filtered": { "query": { "bool": { "minimum_number_should_match":

    1, "should": [ { "query_string": { "query": "type:web_info_log log_namespace:login logdata.reason:\"wrong password\" response:403 ", "default_operator": "AND", "lowercase_expanded_terms": false, "allow_leading_wildcard": false }}]}}, "filter": { "bool": { "must": [ { "range": { "event_timestamp": { "from": 1468294422783, "to": 1468295322783 }}}]}}}}, "size": 0, "sort": [ { "event_timestamp": { "order": "desc", "ignore_unmapped": true }}, { "event_timestamp": { "order": "desc", "ignore_unmapped": true }}], "aggs": { "terms_bucket": { "terms": { "field": "logdata.remote_host", "size": 10 }}}}
  14. 411

  15. Alert Generation & Management Write queries to be periodically executed

    Receive email alerts with results Manage alerts via the web interface
  16. Sensitivity For a given event, how often a search modelled

    on that event will alert True Positive Rate
  17. Sensitivity For a given event, how often a search modelled

    on that event will alert True Positive Rate Avoid creating searches that are too specific
  18. Sensitivity For a given event, how often a search modelled

    on that event will alert True Positive Rate Avoid creating searches that are too specific Minimize False Negatives
  19. Sensitivity For a given event, how often a search modelled

    on that event will alert True Positive Rate Avoid creating searches that are too specific Minimize False Negatives E.g. IP address AND user agent AND user id
  20. Specificity For a given event, how often a search modelled

    on that event will correctly not fire
  21. Specificity For a given event, how often a search modelled

    on that event will correctly not fire True Negative Rate
  22. Specificity For a given event, how often a search modelled

    on that event will correctly not fire True Negative Rate Avoid creating searches that are overly broad
  23. Specificity For a given event, how often a search modelled

    on that event will correctly not fire True Negative Rate Avoid creating searches that are overly broad Minimize False Positives
  24. Specificity For a given event, how often a search modelled

    on that event will correctly not fire True Negative Rate Avoid creating searches that are overly broad Minimize False Positives E.g. Numerous POST requests to /login
  25. Incident Response High specificity alerts Low priority alerts don’t generate

    notification e-mails Medium/High priority alerts generate alerts
  26. Incident Response High specificity alerts Low priority alerts don’t generate

    notification e-mails Medium/High priority alerts generate alerts Attackers often generate a lot of noise -- can result in numerous alerts firing!
  27. Responding to an Alert Is this an alert that can

    wait till morning? How many other related alerts went off during this time period?
  28. Responding to an Alert Is this an alert that can

    wait till morning? How many other related alerts went off during this time period? Example: failed logins and bot activity
  29. Responding to an Alert Was there activity our alerts did

    not catch initially? Dashboards, developers, combing through log files
  30. Responding to an Alert Was there activity our alerts did

    not catch initially? Dashboards, developers, combing through log files Incorporate into new alerts, improve sensitivity of old alerts
  31. Alert Maintenance Sometimes certain queries are no longer useful Review

    noisy alerts Add in other useful fields Example: Attacker using an off-the shelf scanner
  32. Multiple 411 Instances Really easy to set up a new

    instance Supports multiple hosts out the box
  33. Multiple 411 Instances Really easy to set up a new

    instance Supports multiple hosts out the box Just need to run a script