What’s X-citing in X-Pack?

Don’t know what X-Pack is?

X-Pack is a single extension lets you add security (formerly Shield), alerting (via Watcher), monitoring (formerly Marvel), reporting, and Graph functionality across the entire Elastic Stack.

Not only have the capabilities of X-Pack expanded in the past year, but so has its usability, from a better getting started experience to the introduction of several UI features. Needless to say, exciting things are afoot, and the engineers behind X-Pack look forward to sharing what they’re working on with you.

Chris Earle l Monitoring Lead l Elastic
Mark Harwood l Software Engineer l Elastic
Shaunak Kashyap l Software Engineer l Elastic
Brandon Kobel l Sr. Javascript Engineer l Elastic
Jay Modi l Security Engineer l Elastic
Alexander Reelsen l Software Engineer l Elastic

Transcript

1. What x-citing in x-pack?
   Monitoring
   Security
   Reporting
   Alerting
   Graph
   Chris Earle
   Jay Modi
   Brandon Kobel
   Alexander Reelsen
   Shaunak Kashyap
   Mark Harwood
   @pickypg
   @jaymode2001
   @kobelb
   @spinscale
   @shaunak
   @elasticmark
   

   View Slide

2. Agenda
   2
   All the news
   1 Management & Monitoring
   2 Security
   3 Reporting
   4 Alerting
   5 Graph
   

   View Slide

3. Management & Monitoring
   

   View Slide

4. • Foundation (5.0)
   • Elastic Stack Integration (5.0 for Elasticsearch integration)
   • It’s not just for Kibana anymore!
   • User Management (5.0+)
   • Role Management (5.0+)
   • Search Profiler (5.1)
   • Free with Basic license!
   Thank You, the Management
   4
   

   View Slide

5. • More Management Puns and Buzzwords
   • Deeper Elastic Stack Integration
   • Elasticsearch management (e.g., putting a UI on top of complicated APIs)
   • Logstash management (e.g., shared configurations stored in Elasticsearch)
   • Beats management
   • Monitoring integration
   • Kibana APIs
   More Synergy to Come
   5
   

   View Slide

6. Did he say Monitoring?
   That sounds like a good segue
   

   View Slide

7. • Kibana Monitoring (5.0)
   • Multiple Series per chart for simplified comparisons (5.0)
   • Improved HTTP Exporter using Low-level REST Client (5.0)
   • Advanced Node and Index views (5.1)
   • Logstash Monitoring (5.2)
   • Cgroup (Container) metric display for Elasticsearch (5.2)
   Monitoring: Reloaded
   7
   

   View Slide

8. View Slide

9. View Slide

10. View Slide

11. Wait for Applause to Stop
    You were applauding, right?
    

    View Slide

12. • Cluster Alerts
    • Proactive, automatic notifications of problems via Watcher
    • Logstash Pipeline Viewer
    • Find bottlenecks in your Logstash nodes and plugins
    • Machine Learning integration
    • Beats integration
    Monitoring: Revolution(s)
    12
    

    View Slide

13. Last Checked
    Always Actionable
    

    View Slide

14. Security
    

    View Slide

15. Certificate Generation Utility (5.0)
    15
    Simple CLI tool with a specific purpose
    $ cat instances.yml
    instances:
    - name: "node1"
    ip:
    - "192.0.2.1"
    dns:
    - "node1.mydomain.com"
    - name: "node2"
    ip:
    - "192.0.2.2"
    - "198.51.100.1"
    - name: "node4"
    dns:
    - "node4.mydomain.com"
    - "node4.internal"
    - name: "CN=node5,OU=IT,DC=mydomain,DC=com"
    filename: "node5"
    $ bin/x-pack/certgen -in instances.yml -out certificate-bundle.zip
    

    View Slide

16. 16
    Certificate Generation Utility
    $ unzip certificate-bundle.zip
    $ tree
    .
    ├── ca
    │ ├── ca.crt
    │ └── ca.key
    ├── certificate-bundle.zip
    ├── node1
    │ ├── node1.crt
    │ └── node1.key
    ├── node2
    │ ├── node2.crt
    │ └── node2.key
    ├── node4
    │ ├── node4.crt
    │ └── node4.key
    └── node5
    ├── node5.crt
    └── node5.key
    

    View Slide

17. Consistent TLS Configuration
    17
    xpack.ssl.key: "/home/es/config/x-pack/node01.key"
    xpack.ssl.certificate: "/home/es/config/x-pack/node01.crt"
    xpack.ssl.certificate_authorities: [ "/home/es/config/x-pack/ca.crt" ]
    Setting pattern consistent across the stack
    

    View Slide

18. Consistent responses (5.1)
    18
    X-Pack 5.0:
    $ curl -u elastic localhost:9200/_cat/indices
    {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such
    index","index_uuid":"_na_","index":"_all"}],"type":"index_not_found_exception","reason":"no such
    index","index_uuid":"_na_","index":"_all"},"status":404}
    $
    Without X-Pack:
    $ curl -u elastic localhost:9200/_cat/indices
    $
    X-Pack 5.1+:
    $ curl -u elastic localhost:9200/_cat/indices
    $
    

    View Slide

19. 19
    TLS only for node
    to node transport
    

    View Slide

20. 20
    Password:
    changeme
    Goodbye Default
    Passwords
    

    View Slide

21. Passwords removed from configuration files
    21
    xpack:
    security:
    ssl:
    key: '/etc/elasticsearch/config/x-pack/node1.key'
    key_passphrase: 'my super secret password is changeme!'
    certificate: '/etc/elasticsearch/config/x-pack/node1.crt'
    transport:
    ssl:
    enabled: true
    

    View Slide

22. 22
    Elasticsearch
    X-pack
    Master Nodes (3)
    Ingest Nodes (X)
    Data Nodes - Hot (X)
    Data Nodes - Warm (X)
    Single Sign On
    Authentication Sources
    SAML OAuth 2 Kerberos
    Kibana
    X-pack
    Instances (X)
    

    View Slide

23. Generalized Single Sign On Flow
    23
    Elasticsearch
    X-pack
    Identity Provider
    1. Initial Request
    2. Requires Authentication
    3. Get token/assertion

    4. Request with Authentication
    5. Response
    

    View Slide

24. Reporting
    

    View Slide

25. View Slide

26. View Slide

27. Darrell Huff
    { }
    When numbers in tabular form are
    taboo and words will not do the work
    well, as is often the case, there is one
    answer left: Draw a picture.
    

    View Slide

28. Darrell Huff “How to Lie with Statistics”
    { }
    When numbers in tabular form are
    taboo and words will not do the work
    well, as is often the case, there is one
    answer left: Draw a picture.
    

    View Slide

29. View Slide

30. 30
    Downloadable PDF
    

    View Slide

31. 31
    Utilizes Existing
    Infrastructure
    

    View Slide

32. What’s Next?
    

    View Slide

33. More Layout Options
    

    View Slide

34. View Slide

35. View Slide

36. Additional Output Formats
    

    View Slide

37. EXPORT TO CSV
    

    View Slide

38. Alerting
    

    View Slide

39. • Versioned watch history templates
    • Conditions per action
    • JIRA action
    • Email action: Reporting integration (backported to 2.4)
    • Index action: Specify document id (5.3)
    Alerting: Past, present & future
    39
    

    View Slide

40. • Watch execution happens on master node
    Alerting: Past, present & future
    40
    

    View Slide

41. • Watch execution should happen on all nodes
    Alerting: Past, present & future
    41
    

    View Slide

42. • Watch execution should happen on all nodes
    Alerting: Past, present & future
    42
    

    View Slide

43. • Move execution to data nodes, where the .watches shards are
    • No single point of failure
    • Master node does not do any workload
    • Add replicas on the fly to scale out execution
    • Shard Allocation Filtering allows for dedicated watcher nodes
    • Fully backwards-compatible on API level
    Alerting: Distributed watch execution
    43
    

    View Slide

44. • Structure of a single watch is too static
    • The order of execution is simple
    • input -> condition -> actions
    • What if you wanted:
    • input -> condition -> input -> input -> if -> email -> else -> logging
    • Keep state between watch executions
    • Making the core execution async
    Alerting: Past, present & future
    44
    

    View Slide

45. DEMO
    Alerting: One last thing...
    

    View Slide

46. Graph
    

    View Slide

47. • Explore across multiple indices
    • Simplified field configuration
    • Saveable/shareable workspaces
    • Deep linking into Graph
    • Deep linking out of Graph
    New Graph UI features
    47
    

    View Slide

48. Demo scenario - risk management
    48
    Ingest Linking Risk-scoring Investigation
    Entity resolution,
    filtering
    Cleansing, enriching
    normalisation
    Graph exploration,
    anomaly detection,
    scoring
    Task lists, case
    management, visualisation
    Outcomes
    

    View Slide

49. Responding to risk alerts
    49
    Ingest Linking Risk-scoring Investigation
    See example: http://bit.ly/es_fraud
    

    View Slide

50. DEMO
    

    View Slide

51. Graph futures
    

    View Slide

52. More details behind connections, more perspectives
    52
    2,386 21 3
    21 2
    4
    adjacency_matrix aggregation
    20 20 20
    = graphs over time visualizations
    with nested aggregations…
    

    View Slide

53. • BoF: Alerting Use-Cases, today, 1:15
    • BoF: Effectively Using Monitoring, today, 3:15
    • X-Pack Enablement Security Workshop, Thursday, 9:00
    • Getting Your Data Graph Ready, Thursday, 12:45
    • The Usual Suspects: Automatic Alerts to Monitor your Cluster, Thursday, 1:45
    Other Talks You Should See
    53
    

    View Slide

54. 54
    More Questions?
    Visit us at the AMA
    

    View Slide

55. www.elastic.co
    

    View Slide

56. Except where otherwise noted, this work is licensed under
    http://creativecommons.org/licenses/by-nd/4.0/
    Creative Commons and the double C in a circle are
    registered trademarks of Creative Commons in the United States and other countries.
    Third party marks and brands are the property of their respective holders.
    56
    Please attribute Elastic with a link to elastic.co
    

    View Slide