Elastic{ON} 2018 - Elasticsearch Consensus - The Past, The Present and The Future

Transcript

1. Elasticsearch Consensus The Past, the Present, and the Future Boaz Leskes @bleskes Yannick Welsch @ywelsch Jason Tedor @jasontedor David Turner @davecturner
2. The Perfect Cluster PUT beats-2018-02-28 POST _bulk { “_index”: “product” … } GET _search { “query”: { … } … } POST twitter/_settings { “..allocation.include”: “” }
3. A Real Cluster means Compromises
4. 2014 - 2018
5. Data & Metadata Data - indexing - searching Metadata - mapping - shard allocation - node membership Data - big - high throughput - few copies Metadata - less frequent - small - required on all nodes
6. , the early days...
7. 7 (In-)Consistencies
8. 8 Split brain
9. 9 Avoiding split brains minimum_master_nodes = 2
10. 10 Diffs & Batching
11. 11 Overloaded master
12. Data layer Hardened and formalized
13. 13 Tracking in-sync shard copies
14. 14 Do not lose acknowledged writes
15. 15 Primary terms
16. Back to the metadata
17. 17 Primary terms
18. 18 Consensus
19. 19 Formal methods
20. 20 This is a sample image Architecture
21. • Less need to handle inconsistency • New features • discovery.zen.minimum_master_nodes Benefits 21 Cluster state always consistent
22. 22 This is a sample image Master migrations
23. 23 Challenges • More design validation • Implementation • Growing team
24. 24 This is a sample image Timeline
25. 25 More Questions? Visit us at the AMA
26. www.elastic.c o
27. Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 27 Please attribute Elastic with a link to elastic.co