Every Document Deserves a Home - Resiliency in Elasticsearch

How does Elasticsearch work in a resilient and performant way?

This talk is a deep dive into its internals:
* What are the different node types?
* How is the master node selected?
* How are indexes spread over shards and how are they allocated?
* What is the replication protocol?
* How is data actually written and queried?
* How are node failures and added nodes handled?

While we will focus on the current implementation, we will also dedicate some time to past and future developments. What has gone wrong and how did we fix it.

Transcript

1. Distributed Queries with Philipp Krenn̴̴̴̴@xeraa
5. Infrastructure | Developer Advocate
6. ViennaDB Papers We Love Vienna
7. Who uses Elasticsearch
13. Exhibit A: A JSON Document { "name": "Elasticsearch", "author": "Shay Banon", "stable_version": "5.6.1", "preview_version": "6.0.0-beta2" }
14. Exhibit B: A cURL Command $ curl -XPOST -i localhost:9200/databases/nosql -d ' { "name": "Elasticsearch", "author": "Shay Banon", "stable_version": "5.6.1", "preview_version": "6.0.0-beta2" }'
15. Exhibit B: A cURL Command HTTP/1.1 201 Created Location: /databases/nosql/AVfD8XQaeuK3k1LGtT8- content-type: application/json; charset=UTF-8 content-length: 162 { "_index":"databases", "_type":"nosql", "_id":"AVfD8XQaeuK3k1LGtT8-", "_version":1, "result":"created", "_shards": { "total":2, "successful":1, "failed":0 }, "created":true }
16. Exhibit C: A Console Command POST /databases/nosql { "name": "Elasticsearch", "author": "Shay Banon", "stable_version": "5.6.1", "preview_version": "6.0.0-beta2" }
17. Exhibit C: A Console Command { "_index": "databases", "_type": "nosql", "_id": "AVfD6ukyeuK3k1LGtSwT", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 1, "failed": 0 }, "created": true }
21. Single node $ cd elasticsearch-<version> $ ./bin/elasticsearch
23. Cluster
25. Node Types
26. Master-eligible node Default by default
27. Data node Default by default
28. Coordinating node Smart load balancer
29. More Ingest, Machine Learning, Tribe / Cross Cluster Search
30. Discovery
31. Zen discovery No more broadcasting
32. Peer to peer network Unicast̴Who to contact Ping̴Actively contact Sniff̴Get all nodes
33. One master node Elected or joined to
34. Three dedicated master nodes for production
35. discovery.zen.minimum_master_nodes
36. Healthcheck Master pings all nodes Every node pings master
37. discovery.zen.no_master_block write | all
38. Split brain !
39. GET /_cluster/health green̴̴yellow̴̴red
40. Alternative discovery Azure, EC2, GCE
41. Indexing a Document
43. Elasticsearch index Lucene index To index
45. Document Unique combination:̴_index̴_type̴_id PS: Types will be removed
46. POST /databases/nosql vs PUT /databases/nosql/ elasticsearch
47. Autogenerated ID 20 characters, URL-safe, Base64-encoded, GUID strings AVfD6ukyeuK3k1LGtSwT
48. Consistent hashing Before 2.0: djb2
49. unsigned long hash(unsigned char *str) { unsigned long hash = 5381; int c; while (c = *str++) hash = ((hash << 5) + hash) + c; /* hash * 33 + c */ return hash; }
50. Consistent hashing Current default: murmur3 https://github.com/elastic/elasticsearch/blob/5.6/core/src/ main/java/org/elasticsearch/common/hash/MurmurHash3.java
51. Consistent hashing Better distribution 100,000 incremental IDs https://github.com/elastic/elasticsearch/pull/7954
52. Consistent hashing 3 shards murmur3 [33185, 33347, 33468] djb2 [30100, 30000, 39900]
53. Consistent hashing 5 shards murmur3 [19933, 19964, 19940, 20030, 20133] djb2 [20000, 20000, 20000, 20000, 20000]
54. Consistent hashing 33 shards murmur3 [2999, 3096, 2930, 2986, 3070, 3093, 3023, 3052, 3112, 2940, 3036, 2985, 3031, 3048, 3127, 2961, 2901, 3105, 3041, 3130, 3013, 3035, 3031, 3019, 3008, 3022, 3111, 3086, 3016, 2996, 3075, 2945, 2977] djb2 [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 900, 900, 900, 900, 1000, 1000, 10000, 10000, 10000, 10000, 9100, 9100, 9100, 9100, 9000, 9000, 0, 0, 0, 0, 0, 0]
55. Shard decision shard = hash(doc_id) % (num_of_primary_shards)
56. Write
60. Acknowledge
64. Optimistic concurrency control _version
65. Flaky nodes index.unassigned.node_left.delayed_timeout: 1m
66. Lucene Segment
67. Lucene at work
77. Sequency numbers Quick recovery in 6.0
78. Translog defaults in 6.0 index.translog.retention.size: 512mb index.translog.retention.age: 12h
79. Segments are immutable
80. Merge Combine (and clean up) segments
81. Visualize merges http://blog.mikemccandless.com/2011/02/visualizing-lucenes- segment-merges.html
83. Searching
84. Coordinating node Scatter & gather
85. Query phase
89. Fetch phase
93. Demo
94. Common Questions
95. When will version 6.0 be released?
96. Can I use Elasticsearch as my primary datastore?
97. Should I use Elasticsearch as my primary datastore?
98. What about cross data center support?
99. Conclusion
100. The meaning of index
101. Store document
102. Search a document
103. Questions? Philipp Krenn̴̴̴̴̴@xeraa PS: Stickers
104. Image Credit → Schnitzel https://flic.kr/p/9m27wm → Architecture https://flic.kr/p/6dwCAe → Conchita https://flic.kr/p/nBqSHT