Elastic{ON} Tour 2017 New York : Elasticsearch .Deep Dive

Elastic{ON} Tour New York - November 9, 2017

Cross-cluster search, ingest node, rollover API, shrink API, field collapsing, unified highlighter . . . there's lots to love in Elasticsearch these days. Get up to speed on all things 5.x and see how 6.x will address pain points around scale, upgrading, recovery, and sparse data and disk usage.

Steve Kearns | Senior Director of Product Management | Elastic

Transcript

1. Steve Kearns Sr. Director of Product Management Elasticsearch: Past, Present, & Future
2. Elasticsearch 5.0 October 2016
3. 3 Better at Numbers Safe Simple Things
 Should Be Simple Elasticsearch 5.0
4. Elasticsearch 5.x: Feature-rich 5.6 Released September, 2017
5. 5 Elasticsearch 5.x Still ^ • Keyword normalization • Unified highlighter • Field collapse • Multi-word synonyms+proximity • Cancellable searches • Parallel scroll & reindex
6. 6 Elasticsearch 5.x Still ^ • Numeric & date range fields • Automatic optimizations for range searches • Massive aggregations with partitioning • Faster geo-distance sorting • Faster geo-ip lookups and for logs and for numbers and for geo and ... ^
7. 7 Ever-increasing Scale • More clusters / resource isolation • Easier to manage • Easier to upgrade • Reduce potential outages • Need to query across clusters
8. Cluster UK Master Nodes Data Node Data Node Data Node Tribe Node t1 Node Client Cluster US Master Nodes Data Node Data Node Data Node t2 Node Client 8 Tribe Node Merged Cluster State Kibana
9. 9 Cross-Cluster Search Cluster UK Master Nodes Data Node Data Node Data Node Master/Data Node Cluster US Master Nodes Data Node Data Node Data Node Master/Data Node Kibana Optional dedicated cross-cluster search cluster
10. Elasticsearch 6.0+ Some insights into the future
11. © Marie-Lan Nguyen Wikimedia Commons / CC-BY 2.5 11 What are the major pain points? • Disk usage • Resiliency & recovery • Major version upgrades
12. © Marie-Lan Nguyen Wikimedia Commons / CC-BY 2.5 12 What are the major pain points? • Disk usage • Resiliency & recovery • Major version upgrades
13. 13 Doc Values - Sparse Data (5.x) Segment 1 ID fname lname 1 Shane Connelly 2 Shay Banon 3 Tanya Bragin Segment 2 ID fname lname mi state city 4 Steve Kearns Null Null Boston 5 George Burdell P GA Null 6 Frasier Crane Null WA Seattle Merged Segment 3 Docs fname lname mi state city 1 Shane Connelly Null Null Null 2 Shay Banon Null Null Null 3 Tanya Bragin Null Null Null 4 Steve Kearns Null Null Boston 5 George Burdell P GA Null 6 Frasier Crane Null WA Seattle
14. 14 Sample Metricbeat Dataset Apples-to-apples index size improvements 0 25 50 75 100 5.x 6.0 OOTB _all disabled
15. 15 Disk Space Improvements • Sparse value support coming in 6.0 • _all field off by default with special query handler • More efficient configurations in ingest products • Future: Rollups © Tony Weman / CC-BY 2.5
16. © Marie-Lan Nguyen Wikimedia Commons / CC-BY 2.5 16 What are the major pain points? • Disk usage • Resiliency & recovery • Major version upgrades
17. 17 Recovery (5.x) Segment 1 Segment 1 Segment 2 Segment 2 Segment 3 Segment 3 Primary Replica
18. 18 Recovery (5.x) Segment 1 Segment 1 Segment 2 Segment 2 Segment 3 Segment 3 Offline Primary
19. 19 Recovery (5.x) Segment 1 Segment 1 Segment 2 Segment 4 Segment 3 Primary Segment 1 Segment 2 Segment 3 Offline
20. 20 Recovery (5.x) Segment 1 Segment 1 Segment 2 Segment 4 Segment 3 Primary Segment 1 Segment 2 Segment 3 Repica File copy recovery
21. 21 Recovery (6.x) Primary Repica Transaction-log recovery 1 2 3 4 5 6 7 8 9 1 2 3 4 5
22. © Marie-Lan Nguyen Wikimedia Commons / CC-BY 2.5 22 What are the major pain points? • Disk usage • Resiliency & recovery • Major version upgrades
23. 23 Major version upgrades pain • Full cluster restart == downtime • Client/server version compatibility (Java) • Data retention/lifecycle • Too many hard-to-resolve breaking
 changes • Little/no warnings of deprecations &
 breaking changes © Famartin Wikimedia Commons / CC-BY 2.5
24. 24 Major version upgrades pain • Full cluster restart == downtime • Client/server version compatibility (Java) • Data retention/lifecycle • Too many hard-to-resolve breaking
 changes • Little/no warnings of deprecations &
 breaking changes © Famartin Wikimedia Commons / CC-BY 2.5
25. 25 Rolling upgrades • Upgrade from 5.latest to 6.latest without full cluster restart • Some caveats to be aware of: • There are still breaking changes! However many have options backported. Read & test! • All nodes must be 5.latest before • Security+TLS
26. 26 Major version upgrades pain • Full cluster restart == downtime • Client/server version compatibility (Java) • Data retention/lifecycle • Too many hard-to-resolve breaking
 changes • Little/no warnings of deprecations &
 breaking changes © Famartin Wikimedia Commons / CC-BY 2.5
27. 27 Java high level REST client • Released in 5.6 • Works across major version upgrade • IDE friendly • Similar API to Transport Client • Based on low-level REST client • Supports CRUD & Search
28. 28 Major version upgrades pain • Full cluster restart == downtime • Client/server version compatibility (Java) • Data retention/lifecycle • Too many hard-to-resolve breaking
 changes • Little/no warnings of deprecations &
 breaking changes © Famartin Wikimedia Commons / CC-BY 2.5
29. 29 Data Retention / Lifecycle: What are Your Options? Delete old data Reindex Run v1.2.0 and hope your security team doesn’t notice • Keeps cluster & servers tidy :) • Good where there’s a practical retention period :) • Still a period that you can’t upgrade :( • No in-between / rollups :( • Get the latest & greatest features :) • Can be time consuming depending on data volumes :( • No easy management :( • Path of least resistance? • Eventually, we stop backporting security fixes :( • You never get the latest & greatest :(
30. 30 Cross Major Version Search v5.6.0 Master Nodes Data Node Data Node v6.0.0 Your App Master Nodes Data Node Cross Cluster Client v5.latest
31. 31 DEMO
32. 32 Major version upgrades pain • Full cluster restart == downtime • Client/server version compatibility (Java) • Data retention/lifecycle • Too many hard-to-resolve breaking
 changes • Little/no warnings of deprecations &
 breaking changes © Famartin Wikimedia Commons / CC-BY 2.5
33. Thank You