What's evolving in Elasticsearch?

Abstract: The 5.0 release of Elasticsearch took advantage of the cleanups in 2.0 to add a slew of new features on top of a rock-solid foundation. Ryan will give an overview of the changes already released in the 5.x series, and a taste of the big new features coming in 6.0.

Ryan Ernst is an Apache Lucene/Solr committer and PMC member. He is an Elasticsearch developer and enjoys working on anything with bits. Prior to Elastic, he worked on Amazon's Product Search and AWS CloudSearch.

https://www.meetup.com/Silicon-Valley-Elastic-Fantastics/events/241914389/

Transcript

1. 1 Ryan Ernst @ryanjernst 6.0 is coming
2. 2 Agenda • Preparing for upgrade • Upgrading (across major versions) • Things to Watch Out For • Pioneer Program
3. 3 Preparing to Upgrade
4. 4 Read the docs! • Breaking changes docs • https://www.elastic.co/guide/en/elasticsearch/reference/6.0/breaking- changes-6.0.html • Blog posts • https://www.elastic.co/blog/elasticsearch-6-0-0-alpha1-released • https://www.elastic.co/blog/elasticsearch-6-0-0-alpha2-released • https://www.elastic.co/blog/elasticsearch-6-0-0-beta1-released
5. 5 Deprecation logs • Since 2.x • Verbose (duplicates) • But covers many things!
6. 6 Upgrade assistant • UI ships with 5.6 X-Pack feature (Basic, free)
7. 7 Upgrading
8. 8 Upgrading experience Upgrading to 2.x Upgrading to 5.x Upgrading to 6.x
9. 9 Rolling upgrades across major versions • Compatibility from 5.6 to any 6.x node! • What about… ➡ The java client ➡ Tribe nodes ➡ Indexing while upgrading
10. 10 Java Client: Transport to REST • “Low level” client Introduced in 5.x • “High level” rest client added with 6.0 ➡Request/response objects like with transport client • Compatible across major versions • Caveat: Still depends on Elasticsearch core jar and dependencies
11. 11 Tribe Node Reimagined: Cross cluster search • Limited scope: only supports search operations • Uses low level rest client to talk across major versions • Tribe will be removed in a future release (likely 7.0)
12. 12 How Cross-Cluster search works Cluster Sales Master Nodes Data Node Data Node Data Node Cluster R&D Master Nodes Data Node Data Node Data Node
13. 13 How Cross-Cluster search works Cluster Sales Master Nodes Data Node Data Node Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Any node can perform cross-cluster search
14. 14 How Cross-Cluster search works Cluster Sales Master Nodes Data Node Data Node Data Node Optional dedicated cross-cluster search cluster Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node
15. 15 How Cross-Cluster search works Cluster Sales Master Nodes Data Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node PUT _cluster/settings { "transient": { "search.remote": { "sales.seeds": "10.0.0.1:9300", “r_and_d.seeds”: "10.1.0.1:9300" } } } Dynamic settings Optional dedicated cross-cluster search cluster
16. 16 How Cross-Cluster search works Cluster Sales Master Nodes Data Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node No cluster state updates Optional dedicated cross-cluster search cluster
17. 17 How Cross-Cluster search works Cluster Sales Master Nodes Data Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node Kibana Optional dedicated cross-cluster search cluster
18. 18 How Cross-Cluster search works Cluster Sales Master Nodes Data Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node Kibana Can create indices Optional dedicated cross-cluster search cluster
19. 19 How Cross-Cluster search works Cluster Sales Master Nodes Data Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node Kibana Optional dedicated cross-cluster search cluster
20. 20 How Cross-Cluster search works Cluster Sales Master Nodes Data Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node Kibana Few lightweight connections Optional dedicated cross-cluster search cluster
21. 21 How Cross-Cluster search works Cluster Sales Master Nodes Data Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node Kibana Index namespacing GET sales:*,r_and_d:logs*/_search { "query": { … } } Optional dedicated cross-cluster search cluster
22. 22 How Cross-Cluster search works Cluster Sales Master Nodes Data Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node Kibana With many shards Batched Reduce Phase Optional dedicated cross-cluster search cluster
23. 23 Indexing while upgrading: Replication improvements • Limit syncs to only changed documents (instead of file-based recovery) • Fast replica recovery after temporary unavailability (network issues, etc.) • Re-sync on primary failure • Laying foundation for future big league features ‒Cross-datacenter replication ‒Changes API (tbd)
24. 24 Elasticsearch: Recovery Segment-based Segment 1 Segment 2 Segment 3 Segment 2 Segment 3 Primary Replica Request Request Segment 1
25. 25 Elasticsearch: Recovery Segment 1 Segment 2 Segment 3 Segment 3 Primary Primary Segment 2 Segment 1 Background merge Segment-based
26. 26 Elasticsearch: Recovery Segment 1 Segment 2 Segment 3 Segment 2 Primary Primary Segment 1 Segment-based
27. 27 Elasticsearch: Recovery Replica Primary Segment 1 Segment 2 Segment 3 Segment 2 Segment 1 Segment-based
28. 28 Elasticsearch: Recovery Replica Primary Segment 2 Segment 1 Segment 2 Segment 1 File copy Segment-based
29. 29 Elasticsearch: Recovery Operation-based (sequence IDs) Segment 1 Segment 2 Segment 3 Segment 2 Segment 3 Primary Replica Request Segment 1 1 2 3 4 1 2 3
30. 30 Elasticsearch: Recovery Operation-based (sequence IDs) Segment 1 Segment 2 Segment 3 Segment 2 Segment 3 Primary Replica Request Request Segment 1 1 2 3 4 1 2 3 4
31. 31 Elasticsearch: Recovery Operation-based (sequence IDs) Segment 1 Segment 2 Segment 3 Segment 2 Segment 3 Primary Replica Segment 1 1 2 3 4 1 2 3 4 ACK
32. 32 Elasticsearch: Recovery Operation-based (sequence IDs) Segment 1 Segment 2 Segment 3 Segment 2 Segment 3 Replica Primary Segment 1 1 2 3 4 1 2 3 4 Background merge
33. 33 Elasticsearch: Recovery Operation-based (sequence IDs) Segment 1 Segment 2 Segment 3 Segment 4 Replica Primary Segment 1 1 2 3 4 1 2 3 4
34. 34 Elasticsearch: Recovery Operation-based (sequence IDs) Segment 1 Segment 2 Segment 3 Segment 4 Replica Primary Segment 1 1 2 3 4 1 2 3 4
35. 35 Elasticsearch: Recovery Operation-based (sequence IDs) Segment 1 Segment 2 Segment 3 Segment 4 Replica Primary Segment 1 1 2 3 4 1 2 3 4
36. 36 Large Improvements to Replication Operation-based (sequence IDs)
37. 37 Things to Watch Out For
38. 38 _type removal • Cannot create indexes with more than one `_type` value • Will be removed in 7.0/8.0 • Still have to specify type in indexing operations…
39. 39 _all removal • Use query_string and simple_query_string • Use copy_to to have a catch-all field • Backcompat supported for indexes created before 6.0
40. 40 Other changes • Indices created in 2.x and prior
 • on/off/1/0/yes/no no longer accepted as substitutes for true/false
 • 32-bit JVMs
 • Content-type detection
 • Scripting: Groovy, JavaScript, Python, “native” scripts, lang removed; security/context changes
41. 41 Become an Elastic Pioneer Download 6.0 preview release Provide feedback via GitHub or Discuss forum Get limited edition Pioneer swag 1 2 3
42. 42 Elastic Pioneer Program Download 6.0 preview release (alpha, beta, etc) Provide feedback via GitHub or Discuss forum Get limited edition Pioneer swag 1 2 3 We want your feedback!
43. 43 Questions?