What's new in version 6 of the Elastic Stack

The presentation that Luca and I gave at the Elastic Meetup in Rome in September 2017.

Transcript

1. 1 Abdon Pijpelink & Luca Cavanna Rome Meetup, September 12th 2017 6.0 is coming
2. 2 Elasticsearch
3. 3 Brand new upgrade experience Upgrades just got much simpler Upgrading to 2.x Upgrading to 5.x Upgrading to 6.x
4. 4 Brand new upgrade experience • Zero downtime upgrades ‒Rolling restarts from 5.6 to 6.x ‒Cross-cluster search across major version Upgrades just got much simpler
5. 5 Space-saving columnar store • Better for storing sparse fields • Save on disk space & file system cache Tapping into Lucene 7 goodness (sparse doc value) user first middle last age phone johns Alex Smith jrice Jill Amy Rice 508.567.1211 mt123 Jeff Twain 56 sadams Sue Adams adoe Amy Doe 31 lp12 Liz Potter
6. 6 Much speedier sorted queries Tapping into Lucene 7 goodness (index sorting) Player 1 Score: 600 5.x Query for top 3 player scores Player 2 Score: 0 Player 3 Score: 200 Player 4 Score: 700 Player 5 Score: 300 Player 1907 Score: 800 ... Query for top 3 player scores ... Player 1907 Score: 800 Player 4 Score: 700 Player 1 Score: 600 Player 5 Score: 300 Player 3 Score: 200 Player 2 Score: 0 6.x Sort at index time vs. query time Optimize on-disk format for some use cases Improve query performance at the cost of index performance
7. 7 Large Improvements to Replication • Limit syncs to only changed documents (instead of file-based recovery) • Fast replica recovery after temporary unavailability (network issues, etc.) • Re-sync on primary failure • Laying foundation for future big league features ‒Cross-datacenter replication ‒Changes API (tbd) New operation-based approach to recovery (sequence numbers) https://www.elastic.co/blog/elasticsearch-sequence-ids-6-0
8. 8 Simpler data models with type removal • Breaking change • Gradual migration path ‒ 6.0 indices can be created with only one type ‒ Existing 5.x indices using _type will continue to function • Introducing new APIs for type-less operations Say goodbye to _type confusion
9. 9 Less disk usage with _all removal • Breaking change • Gradual migration path ‒ 6.0 indices do not allow _all to be enabled ‒ Existing 5.x indices using _all will continue to function • Use all_fields execution mode instead (Kibana) • Create your own "catch all" field using copy_to Save 20-30% disk space
10. 10 Kibana
11. 11 Export saved searches to CSV with a single click Highly requested feature Trigger export via Watcher X-Pack feature (Basic, free)
12. 12 Lock down edits with Dashboard Only mode Share dashboards without worrying about accidental changes X-Pack feature (Gold)
13. 13 Maximize screen space with Full Screen mode Optimized viewing for your NOCs & SOCs
15. 15 6.0 starts Kibana on the accessibility path • High contrast color scheme • Keyboard accessibility • Screen reader support • More improvements on the way Accessibility improvements
16. 16 Kibana now supports multiple query languages • Lucene Query Language (default) • Kuery (off by default, experimental in 6.0) • ... perhaps others in the future We want your feedback! • Enable Kuery from Advanced Settings More ways to query with Kuery Consistent syntax and simple to get started
17. 17 Easily create simple threshold alerts New form based UI for threshold alerts X-Pack feature (Gold)
18. 18 Migration is easier with a UI • UI ships with 5.6 Upgrade Assistant UI X-Pack feature (Basic, free)
19. 19 Logstash
20. 20 • Run multiple, distinct workloads on a single Logstash JVM • Simplify dataflow logic by managing per data source logic independently • Monitor each pipeline separately with the new Pipeline Viewer Multiple Pipelines, One Logstash Logstash JDBC Pipeline Netflow Pipeline Apache Pipeline
21. 21 • Manage multiple pipelines from multiple nodes in a single UI
 • Logstash nodes can poll and dynamically reload pipelines on configuration change
 • Secure access to configuration management with X-Pack Centrally Manage Logstash Pipelines Pipeline Management X-Pack feature (Gold) Elasticsearch Kibana Logstash Apache Logstash Logstash Config Mgmt UI DevOps / Admins Auto-Update Pipelines JDBC Netflow
22. 22 Centrally Manage Logstash Pipelines X-Pack feature (Gold)
23. 23 Convert ingest node to Logstash pipelines with a CLI tool Why Logstash? • More input sources • Multiple outputs • Richer transformations • Buffering, persistent queues Easily Migrate from Ingest Node Pipelines Ingest Node Converter $LS_HOME/bin/ingest-convert.sh --input file:///path/to/ingest_pipeline.json --output file:///path/to/logstash_pipeline.conf
24. 24 Beats
25. 25 More modules for more data sources New Filebeat modules New Metricbeat Modules
26. 26 • Improved dashboards for Metricbeat system module • Filebeat NGINX module ships with Machine Learning jobs ‒ We want your feedback Better Modules
27. 27 • Index pattern versions • Simpler configuration ‒ Module commands and configuration files ‒ module.d directory ‒ ./metricbeat module enable system ‒ Dashboards are easier to load and packaged with the Beat And more awesome all around
28. 28 Become an Elastic Pioneer Download 6.0 preview release Provide feedback via GitHub or Discuss forum Get limited edition Pioneer swag 1 2 3
29. 29 Elastic Pioneer Program Download 6.0 preview release (Beta, RC, etc) Provide feedback via GitHub or Discuss forum Get limited edition Pioneer swag 1 2 3 We want your feedback!
30. 30 Q&A