Security, Alerting, Monitoring, and More With the Elastic Stack

Dd9d954997353b37b4c2684f478192d3?s=47 Elastic Co
February 18, 2016

Security, Alerting, Monitoring, and More With the Elastic Stack

Elastic's Jay Modi, Alexander Reelsen, Joe Fleming, Tanguy Leroux, and Tim Sullivan present X-Pack — and all the cool features it brings to the Elastic Stack — at Elastic{ON}16 in San Francisco on February 18, 2016.

Dd9d954997353b37b4c2684f478192d3?s=128

Elastic Co

February 18, 2016
Tweet

Transcript

  1. ‹#› Security, Alerting, Monitoring and More with the Elastic Stack

  2. Monitoring Tanguy Leroux Tim Sullivan @tlrx @timsullivanyeah All the Things

  3. ‹#› Logging and monitoring are at the heart of making

    sure your solutions are up and running to your expectations. - David Messina VP, Enterprise Marketing at Docker’s VP of Enterprise Marketing
  4. ‹#› And then Marvel 1.0 Arrived First commercial plugin of

    Elastic!
  5. Cluster Pulse 5

  6. ‹#›

  7. ‹#›

  8. ‹#› Monitoring for Elasticsearch 2.x Yes, we like to rename

    things :)
  9. Multi-Cluster Support 9

  10. Cluster Overview at a Glance 10

  11. Node Listing 11

  12. ‹#›

  13. 19 Kibana 5 New design Issues Cross-Stack Monitoring

  14. ‹#› Spotlight Theater @ 4:40pm Monitoring Elasticsearch

  15. Security Don’t Hack Me Bro! Jay Modi @jaymode2001

  16. 16 Security for the Elastic Stack • Username/password protection Simply

    Secure Elasticsearch Advanced Security When Needed • LDAP, Active Directory, and PKI integration • Role-based access control • Field and document level security • Encrypted communication • Auditing
  17. Adding Users (now) Command Line Utility 17 $ bin/shield/esusers useradd

    jaymode -r admin Enter new password: Retype new password:
  18. Security APIs User and Role management 18 curl –XPUT localhost:9200/_shield/user/jaymode

    –d ‘ { "roles" : ["engineering", "security"], "password" : "changeme" }’ curl –XPUT localhost:9200/_shield/role/security –d ‘ { "cluster": ["all"], "indices": [ { "names": ".shield_audit_log-*",
 "privileges": ["all"] } ] }’
  19. Kibana Sessions and Login Screen 19

  20. User and Roles UI 20

  21. ‹#› Security Configuration API & UI

  22. ‹#› Kibana Security

  23. ‹#› Built In Users

  24. ‹#› Spotlight Theater @ 2:40pm Securing Elasticsearch

  25. Alerting Watch This! Alexander Reelsen @spinscale

  26. ‹#› Notify me on chat, if we have over 1000

    orders per hour - The Startup CEO
  27. ‹#› Trigger an alert, when the same IP accesses all

    services in a certain interval - Your Admin
  28. ‹#› Email me when the product is back in stock!

    - Desperate Online Shopper
  29. ‹#› 5% traffic increase in the last 5 minutes. Ping

    folks on chat! - Your Loadbalancer
  30. ‹#› 5% traffic increase in the last 5 minutes at

    2am. Pager time! - Not your SO
  31. ‹#› Can you predict the additional system resources for the

    next two weeks? - Every system architect ever
  32. 32 A Watch consists of… • Trigger • Input •

    Condition • Actions • Metadata • Transformation
  33. { } 33 PUT /_watcher/watch/cluster_health

  34. { "trigger" : { "schedule" : { "interval" : "10s"

    } } } 34 PUT /_watcher/watch/cluster_health
  35. { "trigger" : { "schedule" : { "interval" : "10s"

    } }, "input" : { "http" : { "request" : { "url" : "http://localhost:9200/_cluster/health" } } } } 35 PUT /_watcher/watch/cluster_health
  36. { "trigger" : { "schedule" : { "interval" : "10s"

    } }, "input" : { "http" : { "request" : { "url" : "http://localhost:9200/_cluster/health" } } }, "condition" : { "compare" : { "ctx.payload.status" : { "eq" : "red" } } } } 36 PUT /_watcher/watch/cluster_health
  37. { "trigger" : { "schedule" : { "interval" : "10s"

    } }, "input" : { "http" : { "request" : { "url" : "http://localhost:9200/_cluster/health" } } }, "condition" : { "compare" : { "ctx.payload.status" : { "eq" : "red" } } }, "actions" : { "send_email" : { "email" : { "to" : "admin@example.org", "subject" : "Cluster Status Warning", "body" : "Cluster status is RED" } } } } 37 PUT /_watcher/watch/cluster_health
  38. ‹#› Recap

  39. 2.0: Hipchat action 39 "actions" : { "notify-hipchat" : {

    "hipchat" : { "account" : "integration-account", "message" : { "body" : "@{{ctx.metadata.userOnDuty}} Encountered {{ctx.payload.hits.total}} errors in the last 5 minutes (facepalm)", "format" : "text", "color" : "red", "notify" : true } } } }
  40. 2.0: Slack action 40 "actions" : { "notify-slack" : {

    "slack" : { "message" : { "from" : "watcher", "to" : [ "#admins", "#errors" ] , "text" : "Monitoring incident", "attachments" : [ { "text" : "@{{ctx.metadata.userOnDuty}} Encountered {{ctx.payload.hits.total}} errors in the last 5 minutes (facepalm)", "title" : "text", "color" : "danger" ] } ...
  41. 2.0: Activate/Deactivate REST API 41 PUT /_watcher/watch/<watch_id>/_activate PUT /_watcher/watch/<watch_id>/_deactivate

  42. 2.0: Array Compare 42 "condition": { "array_compare": { "ctx.payload.aggregations.top_tweeters.buckets" :

    { "path": "doc_count", "gte": { "value": 25, "quantifier": "some" } } } }
  43. 2.1: Chained inputs 43 "input" : { "chain": { "inputs":

    [ { "first": { "simple" : { "path" : "/_search" } } }, { "second": { "http" : { "request" : { … } } } } ] } } ... }
  44. 2.1: Chained inputs 44 {{ctx.payload.second.hits.total}} {{ctx.payload.first.path}} "input" "chain" "inputs" "first"

    "second" } } ... }
  45. 2.3: PagerDuty action 45 "actions" : { "notify-pagerduty" : {

    "pagerduty" : { "message" : { "description" : "Main system down, please check! Happened at {{ctx.execution_time}}", "client" : "/foo/bar/{{ctx.watch_id}}", "attach_payload" : true, "context" : [ { "type" : "response", "href" : "http://www.test.de/foo" } ] ...
  46. 2.3: External email attachments 46 "actions" : { "email_admin" :

    { "email" : { "to" : "ceo@example.org", "attachments" : { "my_id" : { "http" : { "request" : { "url" : "http://example.org/daily-report.pdf" } } } } } ...
  47. ‹#› What's next?

  48. ‹#› Curator

  49. ‹#› Watcher UI

  50. ‹#› Actions

  51. ‹#› Friday, Lunch area @ 11:00am BoF: Alerting & Notifications

    Share Your Watcher Stories
  52. Reporting Kibana For Your Inbox Joe Fleming @w33ble

  53. ‹#› I need this information. Can you send me a

    report? - Every Manager Ever
  54. 54

  55. 55

  56. ‹#› Network’s down. Meeting’s over. - No Manager Ever

  57. None
  58. None
  59. Delivery with Alerting 59 "actions" : { "email_admin" : {

    "email" : { "to" : "ceo@example.org", "attachments" : { "my_id" : { "http" : { "request" : { "url" : "http://example.org/daily-report.pdf" } } } } } ...
  60. ‹#› 5.0 Alpha 1

  61. ‹#› Distributed Rendering

  62. ‹#› Administrative Control

  63. ‹#› Historical Archive

  64. ‹#› Spotlight Theater @ 3:40pm From Dashboard to PDF Generate

    Reports with the Elastic Stack
  65. 65

  66. ‹#› Monitoring Elasticsearch Securing Elasticsearch From Dashboard to PDF BoF:

    Alerting & Notifications Spotlight Theater: 2:40pm Spotlight Theater: 3:40pm Spotlight Theater: 4:40pm Lunch Area: Friday at 11:00am