Elasticsearch (R)Evolution — You Know, for Search...

Elasticsearch is a distributed, RESTful search and analytics engine built on top of Apache Lucene. After the initial release in 2010 it has become the most widely used full-text search engine, but it is not stopping there.

The revolution happened and now it is time for evolution. We dive into the following questions:
* How did leniency help the initial adoption, but why and how do we lean more on strictness today?
* How can upgrades be improved to avoid any downtime even when changing major versions?
* How can new resiliency features improve recovery scenarios and add totally new features?
* Why are types finally disappearing and how are we are trying to avoid the upgrade pain as much as possible?
* What are examples for some clever performance improvements?
* How can you shrink and (finally) split shards in a highly efficient way?

Attendees learn both about new and upcoming features as well as the motivation and engineering challenges behind them.

Transcript

1. (R)Evolution Philipp Krenn̴̴̴̴̴̴̴̴@xeraa
2. Developer
3. Revolution
4. https://db-engines.com/en/ranking
6. Who uses Elasticsearch?
8. Terms Cluster, Node, Index, Shard, Document, ID
9. $ curl http://localhost:9200 { "name": "elasticsearch1", "cluster_name": "docker-cluster", "cluster_uuid": "NflLphonSp2QPKx2llxkgQ", "version": { "number": "5.6.14", "build_hash": "f310fe9", "build_date": "2018-12-05T21:20:16.416Z", "build_snapshot": false, "lucene_version": "6.6.1" }, "tagline": "You Know, for Search" }
10. $ curl http://localhost:9200 { "name": "elasticsearch1", "cluster_name": "docker-cluster", "cluster_uuid": "NflLphonSp2QPKx2llxkgQ", "version": { "number": "5.6.14", "build_hash": "f310fe9", "build_date": "2018-12-05T21:20:16.416Z", "build_snapshot": false, "lucene_version": "6.6.1" }, "tagline": "You Know, for Search" }
18. Evolution
19. Rolling Upgrades* 6.0 * Demo
21. PS: Kibana 6.6 elasticsearch.url elasticsearch.hosts
22. Floodstage Watermark* 6.0 * Demo
24. Low 85% High 90% Floodstage 95%
25. Sequence Numbers* 6.0 * Demo
28. 63 bits ought to be enough for anyone.
29. Tradeoff index.translog.retention.size: 512MB index.translog.retention.age: 12h
30. Cross Datacenter Replication 6.5
31. Types* 5.6 to 8.0 * Demo
33. Why Data types Sparsity Scoring
34. How 5.6 opt-in for single type 6.x single type for new indices 7.x types optional fixed API PUT {index}/_doc/{id} include_type_name: true for GET|PUT _mapping 8.x no more types https://www.elastic.co/guide/en/elasticsearch/reference/current/ removal-of-types.html#_schedule_for_removal_of_mapping_types
35. Shrink & Split* ̴5.0̴̴̴6.1 * Demo
37. Shrink Combine shards by a factor
39. Split Split into a factor of number_of_routing_shards Not required in 7.0+
40. Shards 7.0
42. Default: 1 Shard per Index Oversharding
43. Simon Says Use a single shard until it blows up
44. Conclusion
45. Don't be too Shay to improve
46. Rolling Upgrades Floodstage Watermark Sequence Numbers
47. Types Shrink & Split Default Shard Number
48. Questions? Philipp Krenn̴̴̴̴̴̴̴@xeraa