Security, Alerting, Monitoring, and More With the Elastic Stack

Elastic's Jay Modi, Alexander Reelsen, Joe Fleming, Tanguy Leroux, and Tim Sullivan present X-Pack — and all the cool features it brings to the Elastic Stack — at Elastic{ON}16 in San Francisco on February 18, 2016.

Transcript

1. ‹#› Security, Alerting, Monitoring and More with the Elastic Stack
2. Monitoring Tanguy Leroux Tim Sullivan @tlrx @timsullivanyeah All the Things
3. ‹#› Logging and monitoring are at the heart of making sure your solutions are up and running to your expectations. - David Messina VP, Enterprise Marketing at Docker’s VP of Enterprise Marketing
4. ‹#› And then Marvel 1.0 Arrived First commercial plugin of Elastic!
5. Cluster Pulse 5
6. ‹#›
7. ‹#›
8. ‹#› Monitoring for Elasticsearch 2.x Yes, we like to rename things :)
9. Multi-Cluster Support 9
10. Cluster Overview at a Glance 10
11. Node Listing 11
12. ‹#›
13. 19 Kibana 5 New design Issues Cross-Stack Monitoring
14. ‹#› Spotlight Theater @ 4:40pm Monitoring Elasticsearch
15. Security Don’t Hack Me Bro! Jay Modi @jaymode2001
16. 16 Security for the Elastic Stack • Username/password protection Simply Secure Elasticsearch Advanced Security When Needed • LDAP, Active Directory, and PKI integration • Role-based access control • Field and document level security • Encrypted communication • Auditing
17. Adding Users (now) Command Line Utility 17 $ bin/shield/esusers useradd jaymode -r admin Enter new password: Retype new password:
18. Security APIs User and Role management 18 curl –XPUT localhost:9200/_shield/user/jaymode –d ‘ { "roles" : ["engineering", "security"], "password" : "changeme" }’ curl –XPUT localhost:9200/_shield/role/security –d ‘ { "cluster": ["all"], "indices": [ { "names": ".shield_audit_log-*",
 "privileges": ["all"] } ] }’
19. Kibana Sessions and Login Screen 19
20. User and Roles UI 20
21. ‹#› Security Configuration API & UI
22. ‹#› Kibana Security
23. ‹#› Built In Users
24. ‹#› Spotlight Theater @ 2:40pm Securing Elasticsearch
25. Alerting Watch This! Alexander Reelsen @spinscale
26. ‹#› Notify me on chat, if we have over 1000 orders per hour - The Startup CEO
27. ‹#› Trigger an alert, when the same IP accesses all services in a certain interval - Your Admin
28. ‹#› Email me when the product is back in stock! - Desperate Online Shopper
29. ‹#› 5% traffic increase in the last 5 minutes. Ping folks on chat! - Your Loadbalancer
30. ‹#› 5% traffic increase in the last 5 minutes at 2am. Pager time! - Not your SO
31. ‹#› Can you predict the additional system resources for the next two weeks? - Every system architect ever
32. 32 A Watch consists of… • Trigger • Input • Condition • Actions • Metadata • Transformation
33. { } 33 PUT /_watcher/watch/cluster_health
34. { "trigger" : { "schedule" : { "interval" : "10s" } } } 34 PUT /_watcher/watch/cluster_health
35. { "trigger" : { "schedule" : { "interval" : "10s" } }, "input" : { "http" : { "request" : { "url" : "http://localhost:9200/_cluster/health" } } } } 35 PUT /_watcher/watch/cluster_health
36. { "trigger" : { "schedule" : { "interval" : "10s" } }, "input" : { "http" : { "request" : { "url" : "http://localhost:9200/_cluster/health" } } }, "condition" : { "compare" : { "ctx.payload.status" : { "eq" : "red" } } } } 36 PUT /_watcher/watch/cluster_health
37. { "trigger" : { "schedule" : { "interval" : "10s" } }, "input" : { "http" : { "request" : { "url" : "http://localhost:9200/_cluster/health" } } }, "condition" : { "compare" : { "ctx.payload.status" : { "eq" : "red" } } }, "actions" : { "send_email" : { "email" : { "to" : "admin@example.org", "subject" : "Cluster Status Warning", "body" : "Cluster status is RED" } } } } 37 PUT /_watcher/watch/cluster_health
38. ‹#› Recap
39. 2.0: Hipchat action 39 "actions" : { "notify-hipchat" : { "hipchat" : { "account" : "integration-account", "message" : { "body" : "@{{ctx.metadata.userOnDuty}} Encountered {{ctx.payload.hits.total}} errors in the last 5 minutes (facepalm)", "format" : "text", "color" : "red", "notify" : true } } } }
40. 2.0: Slack action 40 "actions" : { "notify-slack" : { "slack" : { "message" : { "from" : "watcher", "to" : [ "#admins", "#errors" ] , "text" : "Monitoring incident", "attachments" : [ { "text" : "@{{ctx.metadata.userOnDuty}} Encountered {{ctx.payload.hits.total}} errors in the last 5 minutes (facepalm)", "title" : "text", "color" : "danger" ] } ...
41. 2.0: Activate/Deactivate REST API 41 PUT /_watcher/watch/<watch_id>/_activate PUT /_watcher/watch/<watch_id>/_deactivate
42. 2.0: Array Compare 42 "condition": { "array_compare": { "ctx.payload.aggregations.top_tweeters.buckets" : { "path": "doc_count", "gte": { "value": 25, "quantifier": "some" } } } }
43. 2.1: Chained inputs 43 "input" : { "chain": { "inputs": [ { "first": { "simple" : { "path" : "/_search" } } }, { "second": { "http" : { "request" : { … } } } } ] } } ... }
44. 2.1: Chained inputs 44 {{ctx.payload.second.hits.total}} {{ctx.payload.first.path}} "input" "chain" "inputs" "first" "second" } } ... }
45. 2.3: PagerDuty action 45 "actions" : { "notify-pagerduty" : { "pagerduty" : { "message" : { "description" : "Main system down, please check! Happened at {{ctx.execution_time}}", "client" : "/foo/bar/{{ctx.watch_id}}", "attach_payload" : true, "context" : [ { "type" : "response", "href" : "http://www.test.de/foo" } ] ...
46. 2.3: External email attachments 46 "actions" : { "email_admin" : { "email" : { "to" : "ceo@example.org", "attachments" : { "my_id" : { "http" : { "request" : { "url" : "http://example.org/daily-report.pdf" } } } } } ...
47. ‹#› What's next?
48. ‹#› Curator
49. ‹#› Watcher UI
50. ‹#› Actions
51. ‹#› Friday, Lunch area @ 11:00am BoF: Alerting & Notifications Share Your Watcher Stories
52. Reporting Kibana For Your Inbox Joe Fleming @w33ble
53. ‹#› I need this information. Can you send me a report? - Every Manager Ever
56. ‹#› Network’s down. Meeting’s over. - No Manager Ever
59. Delivery with Alerting 59 "actions" : { "email_admin" : { "email" : { "to" : "ceo@example.org", "attachments" : { "my_id" : { "http" : { "request" : { "url" : "http://example.org/daily-report.pdf" } } } } } ...
60. ‹#› 5.0 Alpha 1
61. ‹#› Distributed Rendering
62. ‹#› Administrative Control
63. ‹#› Historical Archive
64. ‹#› Spotlight Theater @ 3:40pm From Dashboard to PDF Generate Reports with the Elastic Stack
66. ‹#› Monitoring Elasticsearch Securing Elasticsearch From Dashboard to PDF BoF: Alerting & Notifications Spotlight Theater: 2:40pm Spotlight Theater: 3:40pm Spotlight Theater: 4:40pm Lunch Area: Friday at 11:00am