Elastic Stack 6 is coming

Elasticsearch version 6 is about to be released and offers a lot of new features. On the other hand, Other features will be deprecated. This talk will focus on new developments for Elasticsearch, Kibana, Logstash, Beats and X-Pack, as well as migration strategies for users of older versions of Elastisearch.

Transcript

1. Elastic Stack 6 is coming hendrik.saly@codecentric.de
2. •Elastic Stack 5 recap •Elastic Stack 6 is coming • Elasticsearch • Kibana • Logstash • Beats • Hadoop • X-Pack •What’s gone •Migration •Closing thoughts Index_ 2
3. •Released Oct 2016 •Really a big release • Lucene 6 point values • Ingest node • Timelion („timeline“) • „Painless“ scripting • X-Pack (Graph, ML) • Resiliency • New API’s • Okapi BM25 Elastic Stack 5 recap_ 3
4. •The road so far (until 5.6) • Logstash monitoring + persistent queues • Text/Keyword type instead of „string“ • Cross cluster search (beta for 5.x) • Windows MSI Installer • Kibana Grok debugger (X-Pack basic) • High/Low level Java REST client • Unified Highlighter • Secure settings • Beats+Logstash modules Elastic Stack 5 recap_ 4
5. •Elastic Stack 6 GA expected mid of October (my guess) •Based on Lucene 7 •As of writing 6.0.0-beta2 is the most recent pre-release •No Java 9 support for now Elastic Stack 6 is coming_ 5
6. •Currently released (6.0.0 beta-2) •Elasticsearch • Rolling upgrades • Cross cluster search • Index sorting (on index time) • Sequence numbers (faster recovery) - later more on that • Improved/Sparse doc values (Lucene 7) • Shard pre-filter phase • Better full disk prevention • Strict content type checking Elastic Stack 6 is coming_ 6
7. •Index Sorting • Sorting during indexing • May slow down indexing time up to 50% • Sorted query is very fast • Early termination possible • Index sort settings cannot be changed once index is created • No support for nested fields • Sort first on fields with a low cardinality and are used for filtering Index Sorting_ 7
8. •Cross Cluster Search • Supersedes the tribe node • No „inter cluster joining“ of nodes anymore • But „Search only“ • No new API • Register remote clusters • Search against „remotecluster:index“ • „:“ is ambiguous because „:“ is also allowed in regular index names • Can also span clusters running 5.x and 6.x ES Cross Cluster Search_ 8
9. •Kibana • CSV export from Discover view • Dashboard only mode (+ full screen view) • Migration assistant (X-Pack basic) • UI improvements for alerting •Logstash • Visualize Logstash pipeline in Kibana (X-Pack) • Manage configuration centrally from Kibana (X-Pack) • Migration assistant from ingest node to Logstash • Multiple pipelines in one VM Elastic Stack 6 is coming_ 9
10. •Beats • Kubernetes support • New and more consistent dashboards • New Auditbeat •Hadoop integration • Spark structured streaming Elastic Stack 6 is coming_ 10
11. •X-Pack • Security • TLS mandatory on transport layer • OAuth2 compatible token service • No default passwords anymore • Reporting • Better security Elastic Stack 6 is coming_ 11
12. •What’s gone (until now) • Removed (for 6.x indices) • Types • _all field • _uid field (index#type) • Lenient booleans (only true/false) • Deprecated • Tribe nodes (5.4) in favor of CCS What’s gone_ 12
13. •What’s gone (until now) • Removed completely • Percolator api • Groovy/JS/Python and native scripts • Duplicate keys in xcontent are disallowed • Shadow replicas (segment-based replication) • Content-Type auto detection What’s gone_ 13
14. •What’s gone (until now) • Packaging • (RPM/DEB) custom users are no more • path.conf and $CONF_DIR removed • Default path settings disappeared What’s gone_ 14
15. •How to migrate to 6.0.0 • You can keep 5.x indices • You can not keep 1.x/2.x indices • Check for deprecated or removed settings • Index/Node/Cluster settings • In shell scripts/ansible/puppet etc. • Check for custom plugins if they already support ES 6.0.0 • Reindex indices • Mandatory for pre 5.x indices • Mandatory for several X-Pack indices • Use the X-Pack migration assistant (basic license) Migration_ 15
16. •How to migrate to 6.0.0 • Start removing types from your datastructures/queries/logic • Check usage of _all field (create custom one with copy_to) • Monitor the deprecation logs • Rewrite scripts in „painless“ or implement a ScriptEngine Migration_ 16
17. •What’s (maybe) to be expected beyond 6.0.0 • Deprecation/Removal of TransportClient • Elasticsearch SQL (probably X-Pack feat.) • Sequence numbers possibilities: • "Changes API“ • Cross-datacenter replication • SIEM integration (via opbeat), probably X-Pack feat. • Beats go plugins (at least on linux) • Kibana i18n/Globalization (#6515) • „Thanks to the community, IBM in specific, the groundwork for internationalization is here.“ (5.2) Closing thoughts_ 17
18. •Still missing (IMHO) but no evidence of a plan • (Kibana) Multitenancy support, for now you can use • Search Guard • Own Home • General Data Protection Regulation (GDPR) related features • „Reporting“ which really deserves that name ;-) • Report designer • No cropped PDF’s • Full-fledged central administration from Kibana • Full „deguice“ (started with 5.0 but then stopped suddenly) Closing thoughts_ 18
19. hendrik.saly@codecentric.de @hendrikdev22 Questions? 19