Save 37% off PRO during our Black Friday Sale! »

What's evolving in Elasticsearch?

Dd9d954997353b37b4c2684f478192d3?s=47 Elastic Co
August 16, 2017

What's evolving in Elasticsearch?

Abstract: The 5.0 release of Elasticsearch took advantage of the cleanups in 2.0 to add a slew of new features on top of a rock-solid foundation. Ryan will give an overview of the changes already released in the 5.x series, and a taste of the big new features coming in 6.0.

Ryan Ernst is an Apache Lucene/Solr committer and PMC member. He is an Elasticsearch developer and enjoys working on anything with bits. Prior to Elastic, he worked on Amazon's Product Search and AWS CloudSearch.

https://www.meetup.com/Silicon-Valley-Elastic-Fantastics/events/241914389/

Dd9d954997353b37b4c2684f478192d3?s=128

Elastic Co

August 16, 2017
Tweet

Transcript

  1. 1 Ryan Ernst @ryanjernst 6.0 is coming

  2. 2 Agenda • Preparing for upgrade • Upgrading (across major

    versions) • Things to Watch Out For • Pioneer Program
  3. 3 Preparing to Upgrade

  4. 4 Read the docs! • Breaking changes docs • https://www.elastic.co/guide/en/elasticsearch/reference/6.0/breaking-

    changes-6.0.html • Blog posts • https://www.elastic.co/blog/elasticsearch-6-0-0-alpha1-released • https://www.elastic.co/blog/elasticsearch-6-0-0-alpha2-released • https://www.elastic.co/blog/elasticsearch-6-0-0-beta1-released
  5. 5 Deprecation logs • Since 2.x • Verbose (duplicates) •

    But covers many things!
  6. 6 Upgrade assistant • UI ships with 5.6 X-Pack feature

    (Basic, free)
  7. 7 Upgrading

  8. 8 Upgrading experience Upgrading to 2.x Upgrading to 5.x Upgrading

    to 6.x
  9. 9 Rolling upgrades across major versions • Compatibility from 5.6

    to any 6.x node! • What about… ➡ The java client ➡ Tribe nodes ➡ Indexing while upgrading
  10. 10 Java Client: Transport to REST • “Low level” client

    Introduced in 5.x • “High level” rest client added with 6.0 ➡Request/response objects like with transport client • Compatible across major versions • Caveat: Still depends on Elasticsearch core jar and dependencies
  11. 11 Tribe Node Reimagined: Cross cluster search • Limited scope:

    only supports search operations • Uses low level rest client to talk across major versions • Tribe will be removed in a future release (likely 7.0)
  12. 12 How Cross-Cluster search works Cluster Sales Master Nodes Data

    Node Data Node Data Node Cluster R&D Master Nodes Data Node Data Node Data Node
  13. 13 How Cross-Cluster search works Cluster Sales Master Nodes Data

    Node Data Node Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Any node can perform cross-cluster search
  14. 14 How Cross-Cluster search works Cluster Sales Master Nodes Data

    Node Data Node Data Node Optional dedicated cross-cluster search cluster Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node
  15. 15 How Cross-Cluster search works Cluster Sales Master Nodes Data

    Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node PUT _cluster/settings { "transient": { "search.remote": { "sales.seeds": "10.0.0.1:9300", “r_and_d.seeds”: "10.1.0.1:9300" } } } Dynamic settings Optional dedicated cross-cluster search cluster
  16. 16 How Cross-Cluster search works Cluster Sales Master Nodes Data

    Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node No cluster state updates Optional dedicated cross-cluster search cluster
  17. 17 How Cross-Cluster search works Cluster Sales Master Nodes Data

    Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node Kibana Optional dedicated cross-cluster search cluster
  18. 18 How Cross-Cluster search works Cluster Sales Master Nodes Data

    Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node Kibana Can create indices Optional dedicated cross-cluster search cluster
  19. 19 How Cross-Cluster search works Cluster Sales Master Nodes Data

    Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node Kibana Optional dedicated cross-cluster search cluster
  20. 20 How Cross-Cluster search works Cluster Sales Master Nodes Data

    Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node Kibana Few lightweight connections Optional dedicated cross-cluster search cluster
  21. 21 How Cross-Cluster search works Cluster Sales Master Nodes Data

    Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node Kibana Index namespacing GET sales:*,r_and_d:logs*/_search { "query": { … } } Optional dedicated cross-cluster search cluster
  22. 22 How Cross-Cluster search works Cluster Sales Master Nodes Data

    Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node Kibana With many shards Batched Reduce Phase Optional dedicated cross-cluster search cluster
  23. 23 Indexing while upgrading: Replication improvements • Limit syncs to

    only changed documents (instead of file-based recovery) • Fast replica recovery after temporary unavailability (network issues, etc.) • Re-sync on primary failure • Laying foundation for future big league features ‒Cross-datacenter replication ‒Changes API (tbd)
  24. 24 Elasticsearch: Recovery Segment-based Segment 1 Segment 2 Segment 3

    Segment 2 Segment 3 Primary Replica Request Request Segment 1
  25. 25 Elasticsearch: Recovery Segment 1 Segment 2 Segment 3 Segment

    3 Primary Primary Segment 2 Segment 1 Background merge Segment-based
  26. 26 Elasticsearch: Recovery Segment 1 Segment 2 Segment 3 Segment

    2 Primary Primary Segment 1 Segment-based
  27. 27 Elasticsearch: Recovery Replica Primary Segment 1 Segment 2 Segment

    3 Segment 2 Segment 1 Segment-based
  28. 28 Elasticsearch: Recovery Replica Primary Segment 2 Segment 1 Segment

    2 Segment 1 File copy Segment-based
  29. 29 Elasticsearch: Recovery Operation-based (sequence IDs) Segment 1 Segment 2

    Segment 3 Segment 2 Segment 3 Primary Replica Request Segment 1 1 2 3 4 1 2 3
  30. 30 Elasticsearch: Recovery Operation-based (sequence IDs) Segment 1 Segment 2

    Segment 3 Segment 2 Segment 3 Primary Replica Request Request Segment 1 1 2 3 4 1 2 3 4
  31. 31 Elasticsearch: Recovery Operation-based (sequence IDs) Segment 1 Segment 2

    Segment 3 Segment 2 Segment 3 Primary Replica Segment 1 1 2 3 4 1 2 3 4 ACK
  32. 32 Elasticsearch: Recovery Operation-based (sequence IDs) Segment 1 Segment 2

    Segment 3 Segment 2 Segment 3 Replica Primary Segment 1 1 2 3 4 1 2 3 4 Background merge
  33. 33 Elasticsearch: Recovery Operation-based (sequence IDs) Segment 1 Segment 2

    Segment 3 Segment 4 Replica Primary Segment 1 1 2 3 4 1 2 3 4
  34. 34 Elasticsearch: Recovery Operation-based (sequence IDs) Segment 1 Segment 2

    Segment 3 Segment 4 Replica Primary Segment 1 1 2 3 4 1 2 3 4
  35. 35 Elasticsearch: Recovery Operation-based (sequence IDs) Segment 1 Segment 2

    Segment 3 Segment 4 Replica Primary Segment 1 1 2 3 4 1 2 3 4
  36. 36 Large Improvements to Replication Operation-based (sequence IDs)

  37. 37 Things to Watch Out For

  38. 38 _type removal • Cannot create indexes with more than

    one `_type` value • Will be removed in 7.0/8.0 • Still have to specify type in indexing operations…
  39. 39 _all removal • Use query_string and simple_query_string • Use

    copy_to to have a catch-all field • Backcompat supported for indexes created before 6.0
  40. 40 Other changes • Indices created in 2.x and prior


    • on/off/1/0/yes/no no longer accepted as substitutes for true/false
 • 32-bit JVMs
 • Content-type detection
 • Scripting: Groovy, JavaScript, Python, “native” scripts, lang removed; security/context changes
  41. 41 Become an Elastic Pioneer Download 6.0 preview release Provide

    feedback via GitHub or Discuss forum Get limited edition Pioneer swag 1 2 3
  42. 42 Elastic Pioneer Program Download 6.0 preview release (alpha, beta,

    etc) Provide feedback via GitHub or Discuss forum Get limited edition Pioneer swag 1 2 3 We want your feedback!
  43. 43 Questions?