Tweet
Tweet

Slide 1

Slide 1 text

What x-citing in x-pack? Monitoring Security Reporting Alerting Graph Chris Earle Jay Modi Brandon Kobel Alexander Reelsen Shaunak Kashyap Mark Harwood @pickypg @jaymode2001 @kobelb @spinscale @shaunak @elasticmark

Slide 2

Slide 2 text

Agenda 2 All the news 1 Management & Monitoring 2 Security 3 Reporting 4 Alerting 5 Graph

Slide 3

Slide 3 text

Management & Monitoring

Slide 4

Slide 4 text

• Foundation (5.0) • Elastic Stack Integration (5.0 for Elasticsearch integration) • It’s not just for Kibana anymore! • User Management (5.0+) • Role Management (5.0+) • Search Profiler (5.1) • Free with Basic license! Thank You, the Management 4

Slide 5

Slide 5 text

• More Management Puns and Buzzwords • Deeper Elastic Stack Integration • Elasticsearch management (e.g., putting a UI on top of complicated APIs) • Logstash management (e.g., shared configurations stored in Elasticsearch) • Beats management • Monitoring integration • Kibana APIs More Synergy to Come 5

Slide 6

Slide 6 text

Did he say Monitoring? That sounds like a good segue

Slide 7

Slide 7 text

• Kibana Monitoring (5.0) • Multiple Series per chart for simplified comparisons (5.0) • Improved HTTP Exporter using Low-level REST Client (5.0) • Advanced Node and Index views (5.1) • Logstash Monitoring (5.2) • Cgroup (Container) metric display for Elasticsearch (5.2) Monitoring: Reloaded 7

Slide 8

Slide 8 text

8

Slide 9

Slide 9 text

9

Slide 10

Slide 10 text

10

Slide 11

Slide 11 text

Wait for Applause to Stop You were applauding, right?

Slide 12

Slide 12 text

• Cluster Alerts • Proactive, automatic notifications of problems via Watcher • Logstash Pipeline Viewer • Find bottlenecks in your Logstash nodes and plugins • Machine Learning integration • Beats integration Monitoring: Revolution(s) 12

Slide 13

Slide 13 text

Last Checked Always Actionable

Slide 14

Slide 14 text

Security

Slide 15

Slide 15 text

Certificate Generation Utility (5.0) 15 Simple CLI tool with a specific purpose $ cat instances.yml instances: - name: "node1" ip: - "192.0.2.1" dns: - "node1.mydomain.com" - name: "node2" ip: - "192.0.2.2" - "198.51.100.1" - name: "node4" dns: - "node4.mydomain.com" - "node4.internal" - name: "CN=node5,OU=IT,DC=mydomain,DC=com" filename: "node5" $ bin/x-pack/certgen -in instances.yml -out certificate-bundle.zip

Slide 16

Slide 16 text

16 Certificate Generation Utility $ unzip certificate-bundle.zip $ tree . ├── ca │ ├── ca.crt │ └── ca.key ├── certificate-bundle.zip ├── node1 │ ├── node1.crt │ └── node1.key ├── node2 │ ├── node2.crt │ └── node2.key ├── node4 │ ├── node4.crt │ └── node4.key └── node5 ├── node5.crt └── node5.key

Slide 17

Slide 17 text

Consistent TLS Configuration 17 xpack.ssl.key: "/home/es/config/x-pack/node01.key" xpack.ssl.certificate: "/home/es/config/x-pack/node01.crt" xpack.ssl.certificate_authorities: [ "/home/es/config/x-pack/ca.crt" ] Setting pattern consistent across the stack

Slide 18

Slide 18 text

Consistent responses (5.1) 18 X-Pack 5.0: $ curl -u elastic localhost:9200/_cat/indices {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","index_uuid":"_na_","index":"_all"}],"type":"index_not_found_exception","reason":"no such index","index_uuid":"_na_","index":"_all"},"status":404} $ Without X-Pack: $ curl -u elastic localhost:9200/_cat/indices $ X-Pack 5.1+: $ curl -u elastic localhost:9200/_cat/indices $

Slide 19

Slide 19 text

19 TLS only for node to node transport

Slide 20

Slide 20 text

20 Password: changeme Goodbye Default Passwords

Slide 21

Slide 21 text

Passwords removed from configuration files 21 xpack: security: ssl: key: '/etc/elasticsearch/config/x-pack/node1.key' key_passphrase: 'my super secret password is changeme!' certificate: '/etc/elasticsearch/config/x-pack/node1.crt' transport: ssl: enabled: true

Slide 22

Slide 22 text

22 Elasticsearch X-pack Master Nodes (3) Ingest Nodes (X) Data Nodes - Hot (X) Data Nodes - Warm (X) Single Sign On Authentication Sources SAML OAuth 2 Kerberos Kibana X-pack Instances (X)

Slide 23

Slide 23 text

Generalized Single Sign On Flow 23 Elasticsearch X-pack Identity Provider 1. Initial Request 2. Requires Authentication 3. Get token/assertion
 4. Request with Authentication 5. Response

Slide 24

Slide 24 text

Reporting

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

Darrell Huff { } When numbers in tabular form are taboo and words will not do the work well, as is often the case, there is one answer left: Draw a picture.

Slide 28

Slide 28 text

Darrell Huff “How to Lie with Statistics” { } When numbers in tabular form are taboo and words will not do the work well, as is often the case, there is one answer left: Draw a picture.

Slide 29

Slide 29 text

No content

Slide 30

Slide 30 text

30 Downloadable PDF

Slide 31

Slide 31 text

31 Utilizes Existing Infrastructure

Slide 32

Slide 32 text

What’s Next?

Slide 33

Slide 33 text

More Layout Options

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

Additional Output Formats

Slide 37

Slide 37 text

EXPORT TO CSV

Slide 38

Slide 38 text

Alerting

Slide 39

Slide 39 text

• Versioned watch history templates • Conditions per action • JIRA action • Email action: Reporting integration (backported to 2.4) • Index action: Specify document id (5.3) Alerting: Past, present & future 39

Slide 40

Slide 40 text

• Watch execution happens on master node Alerting: Past, present & future 40

Slide 41

Slide 41 text

• Watch execution should happen on all nodes Alerting: Past, present & future 41

Slide 42

Slide 42 text

• Watch execution should happen on all nodes Alerting: Past, present & future 42

Slide 43

Slide 43 text

• Move execution to data nodes, where the .watches shards are • No single point of failure • Master node does not do any workload • Add replicas on the fly to scale out execution • Shard Allocation Filtering allows for dedicated watcher nodes • Fully backwards-compatible on API level Alerting: Distributed watch execution 43

Slide 44

Slide 44 text

• Structure of a single watch is too static • The order of execution is simple • input -> condition -> actions • What if you wanted: • input -> condition -> input -> input -> if -> email -> else -> logging • Keep state between watch executions • Making the core execution async Alerting: Past, present & future 44

Slide 45

Slide 45 text

DEMO Alerting: One last thing...

Slide 46

Slide 46 text

Graph

Slide 47

Slide 47 text

• Explore across multiple indices • Simplified field configuration • Saveable/shareable workspaces • Deep linking into Graph • Deep linking out of Graph New Graph UI features 47

Slide 48

Slide 48 text

Demo scenario - risk management 48 Ingest Linking Risk-scoring Investigation Entity resolution, filtering Cleansing, enriching normalisation Graph exploration, anomaly detection, scoring Task lists, case management, visualisation Outcomes

Slide 49

Slide 49 text

Responding to risk alerts 49 Ingest Linking Risk-scoring Investigation See example: http://bit.ly/es_fraud

Slide 50

Slide 50 text

DEMO

Slide 51

Slide 51 text

Graph futures

Slide 52

Slide 52 text

More details behind connections, more perspectives 52 2,386 21 3 21 2 4 adjacency_matrix aggregation 20 20 20 = graphs over time visualizations with nested aggregations…

Slide 53

Slide 53 text

• BoF: Alerting Use-Cases, today, 1:15 • BoF: Effectively Using Monitoring, today, 3:15 • X-Pack Enablement Security Workshop, Thursday, 9:00 • Getting Your Data Graph Ready, Thursday, 12:45 • The Usual Suspects: Automatic Alerts to Monitor your Cluster, Thursday, 1:45 Other Talks You Should See 53

Slide 54

Slide 54 text

54 More Questions? Visit us at the AMA

Slide 55

Slide 55 text

www.elastic.co

Slide 56

Slide 56 text

Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 56 Please attribute Elastic with a link to elastic.co