Elastic{ON} Tour 2018 Munich : Elasticsearch Deep Dive

Elastic{ON} Tour Munich - February 1, 2018

Cross-cluster search, ingest node, rollover API, shrink API, field collapsing, unified highlighter . . . there's lots to love in Elasticsearch these days. Get up to speed on all things 5.x and see how 6.x will address pain points around scale, upgrading, recovery, and sparse data and disk usage.

Simon Willnauer | Co-Founder and Elasticsearch Tech Lead | Elastic

Transcript

1. Simon Willnauer Elastic Founder and Elasticsearch Tech-Lead Elasticsearch: Past, Present, & Future
2. Elasticsearch 5.0 26 October 2016
3. 3 Better at Numbers Safe Simple Things
 Should Be Simple Elasticsearch 5.0
4. Elasticsearch 5.x: Feature-rich 5.6 Released September 11, 2017
5. 5 Elasticsearch 5.x Still ^ • Keyword normalization • Unified highlighter • Field collapse • Multi-word synonyms+proximity • Cancellable searches • Parallel scroll & reindex
6. 6 Elasticsearch 5.x Still ^ • Numeric & date range fields • Automatic optimizations for range searches • Massive aggregations with partitioning • Faster geo-distance sorting • Faster geo-ip lookups and for logs and for numbers and for geo and ... ^
7. 7 Ever-increasing Scale • More clusters / resource isolation • Easier to manage • Easier to upgrade • Reduce potential outages • Need to query across clusters
8. Cluster UK Master Nodes Data Node Data Node Data Node Tribe Node t1 Node Client Cluster US Master Nodes Data Node Data Node Data Node t2 Node Client 8 Tribe Node Merged Cluster State Kibana
9. 9 Cross-Cluster Search Cluster UK Master Nodes Data Node Data Node Data Node Master/Data Node Cluster US Master Nodes Data Node Data Node Data Node Master/Data Node Kibana Optional dedicated cross-cluster search cluster
10. Elasticsearch 6.0+ Some insights into the future
11. © Marie-Lan Nguyen Wikimedia Commons / CC-BY 2.5 11 What are the major pain points? • Disk usage • Resiliency & recovery • Major version upgrades
12. © Marie-Lan Nguyen Wikimedia Commons / CC-BY 2.5 12 What are the major pain points? • Disk usage • Resiliency & Recovery • Major version upgrades
13. 13 Doc Values - Sparse Data (5.x) Segment 1 ID fname lname 1 Simon Willnauer 2 Shay Banon 3 Tanya Bragin Segment 2 ID fname lname mi state city 4 Steve Kearns Null Null Boston 5 George Burdell P GA Null 6 Bill Swerski Null Null Chicago Merged Segment 3 Docs fname lname mi state city 1 Simon Willnauer Null Null Null 2 Shay Banon Null Null Null 3 Tanya Bragin Null Null Null 4 Steve Kearns Null Null Boston 5 George Burdell P GA Null 6 Bill Swerski Null Null Baz
14. 14 Disk Space Improvements • Sparse value support coming in 6.0 • _all field off by default with special query handler • More efficient configurations in ingest products • Future: Rollups © Tony Weman / CC-BY 2.5
15. 15 Sample Metricbeat Dataset Apples-to-apples index size improvements 0 25 50 75 100 5.x 6.0 OOTB _all disabled
16. © Marie-Lan Nguyen Wikimedia Commons / CC-BY 2.5 16 What are the major pain points? • Disk usage • Resiliency & recovery • Major version upgrades
17. 17 Recovery (5.x) Segment 1 Segment 1 Segment 2 Segment 2 Segment 3 Segment 3 Primary Replica
18. 18 Recovery (5.x) Segment 1 Segment 1 Segment 2 Segment 2 Segment 3 Segment 3 Offline Primary
19. 19 Recovery (5.x) Segment 1 Segment 1 Segment 2 Segment 4 Segment 3 Primary Segment 1 Segment 2 Segment 3 Offline
20. 20 Recovery (5.x) Segment 1 Segment 1 Segment 2 Segment 4 Segment 3 Primary Segment 1 Segment 2 Segment 3 Repica File copy recovery
21. 21 Recovery (6.x) Primary Repica Transaction-log recovery 1 2 3 4 5 6 7 8 9 1 2 3 4 5
22. © Marie-Lan Nguyen Wikimedia Commons / CC-BY 2.5 22 What are the major pain points? • Disk usage • Resiliency & recovery • Major version upgrades
23. 23 Major version upgrades pain • Full cluster restart == downtime • Client/server version compatibility (Java) • Data retention/lifecycle • Too many hard-to-resolve breaking
 changes • Little/no warnings of deprecations &
 breaking changes © Famartin Wikimedia Commons / CC-BY 2.5
24. 24 Major version upgrades pain • Full cluster restart == downtime • Client/server version compatibility (Java) • Data retention/lifecycle • Too many hard-to-resolve breaking
 changes • Little/no warnings of deprecations &
 breaking changes © Famartin Wikimedia Commons / CC-BY 2.5
25. 25 Rolling upgrades • Upgrade from 5.latest to 6.x without full cluster restart • Some caveats to be aware of: • There are still breaking changes! However many have options backported. Read & test! • All nodes must be 5.latest before • Security+TLS
26. 26 Major version upgrades pain • Full cluster restart == downtime • Client/server version compatibility (Java) • Data retention/lifecycle • Too many hard-to-resolve breaking
 changes • Little/no warnings of deprecations &
 breaking changes © Famartin Wikimedia Commons / CC-BY 2.5
27. 27 Java client • All other languages use REST interface • Transport client tied to Elasticsearch major version • Complicates firewalls & security
28. 28 Java Low level REST client • Released in 5.0 • JSON strings only • Resilient, but not user friendly • Low level like other language clients • Can sniff nodes and Round-Robin requests
29. 29 Java high level REST client • Released in 5.6 • Works across major version upgrade • IDE friendly • Similar API to Transport Client • Based on low-level REST client • Supports CRUD & Search
30. 30 Major version upgrades pain • Full cluster restart == downtime • Client/server version compatibility (Java) • Data retention/lifecycle • Too many hard-to-resolve breaking
 changes • Little/no warnings of deprecations &
 breaking changes © Famartin Wikimedia Commons / CC-BY 2.5
31. 31 Data Retention / Lifecycle: What are Your Options? Delete old data Reindex Run v1.2.0 and hope your security team doesn’t notice • Keeps cluster & servers tidy :) • Good where there’s a practical retention period :) • Still a period that you can’t upgrade :( • No in-between / rollups :( • Get the latest & greatest features :) • Can be time consuming depending on data volumes :( • No easy management :( • Path of least resistance? • Eventually, we stop backporting security fixes :( • You never get the latest & greatest :(
32. 32 Cross Major Version Search v5.6.0 Master Nodes Data Node Data Node v6.0.0 Your App Master Nodes Data Node Cross Cluster Client v5.latest
33. 33 The latest and greatest in 6.1 and 6.2 • Shard Splitting • Adaptive Replica Selection • Paging Aggregations • Java 9 Support (6.2) • SSO / SAML (6.2) • Plugin Extensibility via SPI (6.2)
34. 34 QUESTIONS?
35. Thank You