$30 off During Our Annual Pro Sale. View Details »

What’s X-citing in X-Pack?

Elastic Co
March 08, 2017

What’s X-citing in X-Pack?

Don’t know what X-Pack is?

X-Pack is a single extension lets you add security (formerly Shield), alerting (via Watcher), monitoring (formerly Marvel), reporting, and Graph functionality across the entire Elastic Stack.

Not only have the capabilities of X-Pack expanded in the past year, but so has its usability, from a better getting started experience to the introduction of several UI features. Needless to say, exciting things are afoot, and the engineers behind X-Pack look forward to sharing what they’re working on with you.

Chris Earle l Monitoring Lead l Elastic
Mark Harwood l Software Engineer l Elastic
Shaunak Kashyap l Software Engineer l Elastic
Brandon Kobel l Sr. Javascript Engineer l Elastic
Jay Modi l Security Engineer l Elastic
Alexander Reelsen l Software Engineer l Elastic

Elastic Co

March 08, 2017
Tweet

More Decks by Elastic Co

Other Decks in Technology

Transcript

  1. What x-citing in x-pack?
    Monitoring
    Security
    Reporting
    Alerting
    Graph
    Chris Earle
    Jay Modi
    Brandon Kobel
    Alexander Reelsen
    Shaunak Kashyap
    Mark Harwood
    @pickypg
    @jaymode2001
    @kobelb
    @spinscale
    @shaunak
    @elasticmark

    View Slide

  2. Agenda
    2
    All the news
    1 Management & Monitoring
    2 Security
    3 Reporting
    4 Alerting
    5 Graph

    View Slide

  3. Management & Monitoring

    View Slide

  4. • Foundation (5.0)
    • Elastic Stack Integration (5.0 for Elasticsearch integration)
    • It’s not just for Kibana anymore!
    • User Management (5.0+)
    • Role Management (5.0+)
    • Search Profiler (5.1)
    • Free with Basic license!
    Thank You, the Management
    4

    View Slide

  5. • More Management Puns and Buzzwords
    • Deeper Elastic Stack Integration
    • Elasticsearch management (e.g., putting a UI on top of complicated APIs)
    • Logstash management (e.g., shared configurations stored in Elasticsearch)
    • Beats management
    • Monitoring integration
    • Kibana APIs
    More Synergy to Come
    5

    View Slide

  6. Did he say Monitoring?
    That sounds like a good segue

    View Slide

  7. • Kibana Monitoring (5.0)
    • Multiple Series per chart for simplified comparisons (5.0)
    • Improved HTTP Exporter using Low-level REST Client (5.0)
    • Advanced Node and Index views (5.1)
    • Logstash Monitoring (5.2)
    • Cgroup (Container) metric display for Elasticsearch (5.2)
    Monitoring: Reloaded
    7

    View Slide

  8. 8

    View Slide

  9. 9

    View Slide

  10. 10

    View Slide

  11. Wait for Applause to Stop
    You were applauding, right?

    View Slide

  12. • Cluster Alerts
    • Proactive, automatic notifications of problems via Watcher
    • Logstash Pipeline Viewer
    • Find bottlenecks in your Logstash nodes and plugins
    • Machine Learning integration
    • Beats integration
    Monitoring: Revolution(s)
    12

    View Slide

  13. Last Checked
    Always Actionable

    View Slide

  14. Security

    View Slide

  15. Certificate Generation Utility (5.0)
    15
    Simple CLI tool with a specific purpose
    $ cat instances.yml
    instances:
    - name: "node1"
    ip:
    - "192.0.2.1"
    dns:
    - "node1.mydomain.com"
    - name: "node2"
    ip:
    - "192.0.2.2"
    - "198.51.100.1"
    - name: "node4"
    dns:
    - "node4.mydomain.com"
    - "node4.internal"
    - name: "CN=node5,OU=IT,DC=mydomain,DC=com"
    filename: "node5"
    $ bin/x-pack/certgen -in instances.yml -out certificate-bundle.zip

    View Slide

  16. 16
    Certificate Generation Utility
    $ unzip certificate-bundle.zip
    $ tree
    .
    ├── ca
    │ ├── ca.crt
    │ └── ca.key
    ├── certificate-bundle.zip
    ├── node1
    │ ├── node1.crt
    │ └── node1.key
    ├── node2
    │ ├── node2.crt
    │ └── node2.key
    ├── node4
    │ ├── node4.crt
    │ └── node4.key
    └── node5
    ├── node5.crt
    └── node5.key

    View Slide

  17. Consistent TLS Configuration
    17
    xpack.ssl.key: "/home/es/config/x-pack/node01.key"
    xpack.ssl.certificate: "/home/es/config/x-pack/node01.crt"
    xpack.ssl.certificate_authorities: [ "/home/es/config/x-pack/ca.crt" ]
    Setting pattern consistent across the stack

    View Slide

  18. Consistent responses (5.1)
    18
    X-Pack 5.0:
    $ curl -u elastic localhost:9200/_cat/indices
    {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such
    index","index_uuid":"_na_","index":"_all"}],"type":"index_not_found_exception","reason":"no such
    index","index_uuid":"_na_","index":"_all"},"status":404}
    $
    Without X-Pack:
    $ curl -u elastic localhost:9200/_cat/indices
    $
    X-Pack 5.1+:
    $ curl -u elastic localhost:9200/_cat/indices
    $

    View Slide

  19. 19
    TLS only for node
    to node transport

    View Slide

  20. 20
    Password:
    changeme
    Goodbye Default
    Passwords

    View Slide

  21. Passwords removed from configuration files
    21
    xpack:
    security:
    ssl:
    key: '/etc/elasticsearch/config/x-pack/node1.key'
    key_passphrase: 'my super secret password is changeme!'
    certificate: '/etc/elasticsearch/config/x-pack/node1.crt'
    transport:
    ssl:
    enabled: true

    View Slide

  22. 22
    Elasticsearch
    X-pack
    Master Nodes (3)
    Ingest Nodes (X)
    Data Nodes - Hot (X)
    Data Nodes - Warm (X)
    Single Sign On
    Authentication Sources
    SAML OAuth 2 Kerberos
    Kibana
    X-pack
    Instances (X)

    View Slide

  23. Generalized Single Sign On Flow
    23
    Elasticsearch
    X-pack
    Identity Provider
    1. Initial Request
    2. Requires Authentication
    3. Get token/assertion

    4. Request with Authentication
    5. Response

    View Slide

  24. Reporting

    View Slide

  25. View Slide

  26. View Slide

  27. Darrell Huff
    { }
    When numbers in tabular form are
    taboo and words will not do the work
    well, as is often the case, there is one
    answer left: Draw a picture.

    View Slide

  28. Darrell Huff “How to Lie with Statistics”
    { }
    When numbers in tabular form are
    taboo and words will not do the work
    well, as is often the case, there is one
    answer left: Draw a picture.

    View Slide

  29. View Slide

  30. 30
    Downloadable PDF

    View Slide

  31. 31
    Utilizes Existing
    Infrastructure

    View Slide

  32. What’s Next?

    View Slide

  33. More Layout Options

    View Slide

  34. View Slide

  35. View Slide

  36. Additional Output Formats

    View Slide

  37. EXPORT TO CSV

    View Slide

  38. Alerting

    View Slide

  39. • Versioned watch history templates
    • Conditions per action
    • JIRA action
    • Email action: Reporting integration (backported to 2.4)
    • Index action: Specify document id (5.3)
    Alerting: Past, present & future
    39

    View Slide

  40. • Watch execution happens on master node
    Alerting: Past, present & future
    40

    View Slide

  41. • Watch execution should happen on all nodes
    Alerting: Past, present & future
    41

    View Slide

  42. • Watch execution should happen on all nodes
    Alerting: Past, present & future
    42

    View Slide

  43. • Move execution to data nodes, where the .watches shards are
    • No single point of failure
    • Master node does not do any workload
    • Add replicas on the fly to scale out execution
    • Shard Allocation Filtering allows for dedicated watcher nodes
    • Fully backwards-compatible on API level
    Alerting: Distributed watch execution
    43

    View Slide

  44. • Structure of a single watch is too static
    • The order of execution is simple
    • input -> condition -> actions
    • What if you wanted:
    • input -> condition -> input -> input -> if -> email -> else -> logging
    • Keep state between watch executions
    • Making the core execution async
    Alerting: Past, present & future
    44

    View Slide

  45. DEMO
    Alerting: One last thing...

    View Slide

  46. Graph

    View Slide

  47. • Explore across multiple indices
    • Simplified field configuration
    • Saveable/shareable workspaces
    • Deep linking into Graph
    • Deep linking out of Graph
    New Graph UI features
    47

    View Slide

  48. Demo scenario - risk management
    48
    Ingest Linking Risk-scoring Investigation
    Entity resolution,
    filtering
    Cleansing, enriching
    normalisation
    Graph exploration,
    anomaly detection,
    scoring
    Task lists, case
    management, visualisation
    Outcomes

    View Slide

  49. Responding to risk alerts
    49
    Ingest Linking Risk-scoring Investigation
    See example: http://bit.ly/es_fraud

    View Slide

  50. DEMO

    View Slide

  51. Graph futures

    View Slide

  52. More details behind connections, more perspectives
    52
    2,386 21 3
    21 2
    4
    adjacency_matrix aggregation
    20 20 20
    = graphs over time visualizations
    with nested aggregations…

    View Slide

  53. • BoF: Alerting Use-Cases, today, 1:15
    • BoF: Effectively Using Monitoring, today, 3:15
    • X-Pack Enablement Security Workshop, Thursday, 9:00
    • Getting Your Data Graph Ready, Thursday, 12:45
    • The Usual Suspects: Automatic Alerts to Monitor your Cluster, Thursday, 1:45
    Other Talks You Should See
    53

    View Slide

  54. 54
    More Questions?
    Visit us at the AMA

    View Slide

  55. www.elastic.co

    View Slide

  56. Except where otherwise noted, this work is licensed under
    http://creativecommons.org/licenses/by-nd/4.0/
    Creative Commons and the double C in a circle are
    registered trademarks of Creative Commons in the United States and other countries.
    Third party marks and brands are the property of their respective holders.
    56
    Please attribute Elastic with a link to elastic.co

    View Slide