Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Elasticsearch (R)Evolution — You Know, for Search...

Elasticsearch (R)Evolution — You Know, for Search...

Elasticsearch is a distributed, RESTful search and analytics engine built on top of Apache Lucene. After the initial release in 2010 it has become the most widely used full-text search engine, but it is not stopping there.

The revolution happened and now it is time for evolution. We dive into the following questions:
* How did leniency help the initial adoption, but why and how do we lean more on strictness today?
* How can upgrades be improved to avoid any downtime even when changing major versions?
* How can new resiliency features improve recovery scenarios and add totally new features?
* Why are types finally disappearing and how are we are trying to avoid the upgrade pain as much as possible?
* What are examples for some clever performance improvements?
* How can you shrink and (finally) split shards in a highly efficient way?

Attendees learn both about new and upcoming features as well as the motivation and engineering challenges behind them.

Ce4685da897c912aa41a815435b40a5a?s=128

Philipp Krenn

January 08, 2019
Tweet

Transcript

  1. (R)Evolution Philipp Krenn̴̴̴̴̴̴̴̴@xeraa

  2. Developer

  3. Revolution

  4. https://db-engines.com/en/ranking

  5. None
  6. Who uses Elasticsearch?

  7. None
  8. Terms Cluster, Node, Index, Shard, Document, ID

  9. $ curl http://localhost:9200 { "name": "elasticsearch1", "cluster_name": "docker-cluster", "cluster_uuid": "NflLphonSp2QPKx2llxkgQ",

    "version": { "number": "5.6.14", "build_hash": "f310fe9", "build_date": "2018-12-05T21:20:16.416Z", "build_snapshot": false, "lucene_version": "6.6.1" }, "tagline": "You Know, for Search" }
  10. $ curl http://localhost:9200 { "name": "elasticsearch1", "cluster_name": "docker-cluster", "cluster_uuid": "NflLphonSp2QPKx2llxkgQ",

    "version": { "number": "5.6.14", "build_hash": "f310fe9", "build_date": "2018-12-05T21:20:16.416Z", "build_snapshot": false, "lucene_version": "6.6.1" }, "tagline": "You Know, for Search" }
  11. None
  12. None
  13. None
  14. None
  15. None
  16. None
  17. None
  18. Evolution

  19. Rolling Upgrades* 6.0 * Demo

  20. None
  21. PS: Kibana 6.6 elasticsearch.url elasticsearch.hosts

  22. Floodstage Watermark* 6.0 * Demo

  23. None
  24. Low 85% High 90% Floodstage 95%

  25. Sequence Numbers* 6.0 * Demo

  26. None
  27. None
  28. 63 bits ought to be enough for anyone.

  29. Tradeoff index.translog.retention.size: 512MB index.translog.retention.age: 12h

  30. Cross Datacenter Replication 6.5

  31. Types* 5.6 to 8.0 * Demo

  32. None
  33. Why Data types Sparsity Scoring

  34. How 5.6 opt-in for single type 6.x single type for

    new indices 7.x types optional fixed API PUT {index}/_doc/{id} include_type_name: true for GET|PUT _mapping 8.x no more types https://www.elastic.co/guide/en/elasticsearch/reference/current/ removal-of-types.html#_schedule_for_removal_of_mapping_types
  35. Shrink & Split* ̴5.0̴̴̴6.1 * Demo

  36. None
  37. Shrink Combine shards by a factor

  38. None
  39. Split Split into a factor of number_of_routing_shards Not required in

    7.0+
  40. Shards 7.0

  41. None
  42. Default: 1 Shard per Index Oversharding

  43. Simon Says Use a single shard until it blows up

  44. Conclusion

  45. Don't be too Shay to improve

  46. Rolling Upgrades Floodstage Watermark Sequence Numbers

  47. Types Shrink & Split Default Shard Number

  48. Questions? Philipp Krenn̴̴̴̴̴̴̴@xeraa