Elasticsearch and Resiliency

Resiliency is a primary development focus in Elasticsearch. A lot of improvements were released in ES 2.0, and even more is coming in future releases. From faster recovery times to more durable writes, attend this session to learn everything you ever wanted to know about resiliency in Elasticsearch.

Transcript

1. ‹#› Jason Tedor @jasontedor Resiliency in Elasticsearch Boaz Leskes @bleskes
2. ‹#› noun 1. the power or ability to return to the original form, position, etc., after being bent, compressed, or stretched; elasticity. 2. ability to recover readily from illness, depression, adversity, or the like; buoyancy re·sil·ience
3. https://github.com/elastic/elasticsearch/issues?q=label%3Aresiliency 3 181 ISSUES AND PULL REQUESTS
4. https://www.elastic.co/guide/en/elasticsearch/resiliency/current/index.html 4
5. 5 git log --after="2015-3-9" docs/resiliency/index.asciidoc
6. ‹#› Memory Pressure 6
7. Disk based data structures • Doc values (Lucene’s columnar store) • on by default from 2.0 (#10209) • Norms stay on disk in Lucene 5.3.0 • LUCENE-6504 • released in ES 2.1.0 7
8. Smarter algorithms • Automatic query caching • #10897 (ES 2.0) • LUCENE-6077 • Breadth first aggregation trees • Added in #6128 (ES 1.3) • Working on automatic application #9825 8
9. Prevent abuse #11511 • Limit total number of hit count #9311 (2.1.0) • Circuit breakers have been around since 1.0 • Recent additions #16011: ‒ Limit the size of single request ‒ Limit total the size of inflight concurrent request • ThreadPools hard upper bounds #15582 • Settings validation #15278 9
10. ‹#› Long Garbage Collection, Network hiccups, Node restarts 10
11. Long Garbage Collection Impacts Cluster Stability 11 master node node 1s fault detection pings GC re-join
12. Long Garbage Collection, Network Partition & Restarts 12 master node node there is no way to tell them apart
13. Temporary Node Leave 13 node4 node1 a0 b1 b2 node3 b1 a0 node2 a1 b0 b0 a1 b2
14. Temporary Node Leave 14 node4 node1 a0 b1 b2 node3 b1 a0 node2 a1 b0 b0 a1 b2
15. node4 Temporary Node Leave - Promote Primary & Add Replicas 15 b2 node1 a0 b1 b2 node3 b1 a0 node2 a1 b0 b0 a1 b1 a1 needed but potentially expensive
16. node4 Temporary Node Leave - A Grace Period #11712 (1.7) 16 b2 node1 a0 b1 b2 node3 b1 a0 node2 a1 b0 b0 a1 index.unassigned.node_left.delayed_timeout: 1m
17. node4 Still Requires A Recovery When Node Re-Join 17 b2 node1 a0 b1 b2 node3 b1 a0 node2 a1 b0 b0 a1
18. Resync Files With Primary 18 segment 1 + 2 segment 3 segment 4 + 5 Primary segment 2 + 3 segment 1 Replica segment 4 segment 5 reuse existing segments?
19. Re-Sync Files With Primary & Synced Flush #10032 (1.6) 19 segment 1 + 2 segment 3 segment 4 + 5 Primary segment 2 + 3 segment 1 Replica segment 4 segment 5 automatically use inactivity periods to add a
 sync id marker, guaranteeing doc level equality & instant recovery sync_id: 0XYB321 sync_id: 0XYB321
20. node4 Cancel Ongoing Recoveries If A Perfect Match Found #12421 (2.0) 20 b2 node1 a0 b1 b2 node3 b1 a0 node2 a1 b0 b0 a1 b1 a1
21. ‹#› Data Durability 21
22. Durability 22 index a doc time lucene flush buffer index a doc buffer index a doc buffer buffer segment
23. Durability 23 index a doc time lucene flush buffer segment trans_log buffer trans_log buffer trans_log elasticsearch flush doc op lucene commit segment segment
24. Durability - fsync translog every 5s (1.X) 24 index a doc buffer trans_log doc op index a doc buffer trans_log doc op Primary Replica redundancy doesn’t help if all nodes lose power
25. Durability - Translog fsync on every request #11011 (2.0) 25 • For low volume indexing, fsync matters less • For high volume indexing, we can amortize the costs and fsync on every bulk • Concurrent requests can share an fsync
26. Translog fsync concurrent requests #11011 (2.0) 26 bulk 1 bulk 2 single fsync
27. Durability - Recovery after Translog Crash #11143 (2.0) 27 fsync • fsync every 5s means some ops can be partially written • 1.x had to ignore EOF exceptions, making it non-resilient to translog trim • fsync every request still suffers from this • Write a check point file every fsync #12341 2.0 { offset: 3034, ops: 1302 }.ckp
28. ‹#› Dynamic Mapping Updates 28
29. Dynamic Mappings 29 curl -XPUT localhost:9200/twitter/tweet -d ' { "id": "467495741683150848", "text": "the problem with distributed systems jokes is that you're never sure if everyone gets it" "created_on": "Sat May 17 02:44:05 +0000 2014" }' { "id": { "type" : "string" } "text": { "type" : "string" } "created_on": { "type" : "date" } }
30. Dynamic Mappings Optimize for Speed 1.x Concurrent indexing requests, new field 30 master node node {"i": 1} {"i": 10}
31. Dynamic Mappings Optimize for Speed 1.x Assumption: there is a "true" schema, we just need to learn it 31 master node node {"i": 1} {"i": 10} "i": int "i": int
32. Dynamic Mappings Optimize for Speed 1.x If you make assumptions, you're going to have a bad time 32 master node node {"f": 1} {"f": "text"} "f": int "f": string ?
33. Dynamic Mappings 2.x PR #10634 Validate dynamic mapping updates on the master node 33 master node node {"f": 1} {"f": "text"} "f": int "f": string
34. Dynamic Mappings 2.x PR #10634 Validate dynamic mapping updates on the master node 34 master node node {"f": 1} {"f": "text"} "f": int "f": string ✔ ✔
35. ‹#›
36. ‹#› Stale shards 36
37. Do not promote stale shards issue #14671 Index to primary 37 node1 a0 node2 a0
38. Do not promote stale shards issue #14671 Primary fails, replica is promoted 38 node1 a0 node2 a0
39. Do not promote stale shards issue #14671 Old primary is now stale 39 node1 a0 node2 a0
40. Do not promote stale shards issue #14671 Restart cluster 40 node1 a0 node2 a0
41. Do not promote stale shards issue #14671 Stale shard promoted 41 node1 a0 node2 a0
42. ‹#›
43. Allocate primary shards using allocation IDs #14739 Persist allocation IDs as metadata 43 node1 a0 node2 a0 9154ff 5aa72e master a0: 9154ff, 5aa72e
44. Allocate primary shards using allocation IDs #14739 Index document 44 node1 a0 node2 a0 6f91cc master a0: 6f91cc 9154ff
45. ‹#›
46. ‹#› Isolation during indexing 46
47. Shard failures Index document 47 node1 a0 node2 a0 master
48. Shard failures Replicate 48 node1 a0 node2 a0 master
49. Shard failures Replication fails 49 node1 a0 node2 a0 master
50. Shard failures Primary notifies the master and acknowledges the write 50 node1 a0 node2 a0 master
51. Shard failures Master fails the shard 51 node1 a0 node2 master
52. Isolation during indexing issue #7572 Index document during network partition 52 node1 a0 node2 a0 master
53. Isolation during indexing issue #7572 Primary fails to notify master and acknowledges the write 53 node1 a0 node2 a0 master
54. Isolation during indexing issue #7572 The bad replica is promoted to primary and acknowledged writes are lost 54 node1 a0 node2 a0 master
55. ‹#›
56. Waiting while failing a shard issue #14252 56 Wait for failure publication Master left No longer the primary • Master submits cluster state update task • And waits for the update to be processed • Master responds with success or failure • Enter retry loop waiting for new cluster state with a master changed event • The same retry loop is used for any master channel failure • The old primary is failed • Retry the request on the new primary • The new primary now manages the indexing request The request is no longer acknowledged if the shard is not failed
57. ‹#›
58. ‹#› Join on election 58
59. Master Election 1.x Pings 59 master1 master2 master3
60. Master Election 1.x Master elected 60 master1 master2 master3
61. Master Election 1.x Nodes join the master 61 master1 master2 master3 Join Join
62. Master Election 1.x New cluster state published 62 master1 master2 master3 CS CS
63. Master Election 1.x (again) Pings 63 master1 master2 master3
64. Master Election 1.x (again) Master elected 64 master1 master2 master3
65. Master Election 1.x (again) Network partition 65 master1 master2 master3
66. Master Election 1.x (again) Pings round two 66 master1 master2 master3
67. Master Election 1.x (again) Dueling masters 67 master1 master2 master3
68. Master Election 2.x PR #12161 Pings 68 master1 master2 master3
69. Master Election 2.x PR #12161 Master elected…but waits 69 master1 master2 master3 ❓
70. Master Election 2.x PR #12161 Elected master waits for joins 70 master1 master2 master3 Join Join ❓
71. Master Election 2.x PR #12161 Enough nodes join, master is elected 71 master1 master2 master3 CS CS
72. ‹#›
73. ‹#› Resiliency testing 73
74. ‹#›
75. DiscoveryWithServiceDisruptionsIT testFailWithMinimumMasterNodesConfigured testNodesFDAfterMasterReelection testVerifyApiBlocksDuringPartition testIsolateMasterAndVerifyClusterStateConsensus testMasterNodeGCs testStaleMasterNotHijackingMajority testRejoinDocumentExistsInAllShardCopies testUnicastSinglePingResponseContainsMaster testIsolatedUnicastNodes testClusterJoinDespiteOfPublishingIssues testSendingShardFailure testClusterFormingWithASlowNode testNodeNotReachableFromMaster testSearchWithRelocationAndSlowClusterStateProcessing testIndexImportedFromDataOnlyNodesIfMasterLostDataFolder testIndicesDeleted Resiliency tests Service disruptions tests 75
76. CorruptedFileIT testCorruptFileAndRecover testCorruptPrimaryNoReplica testCorruptionOnNetworkLayerFinalizingRecovery testCorruptionOnNetworkLayer testCorruptFileThenSnapshotAndRestore testReplicaCorruption Resiliency tests Corruption tests 76
77. ServiceDisruptionScheme NoOpDisruptionScheme SingleNodeDisruption SlowClusterStateProcessing BlockClusterStateProcessing LongGCDisruption IntermittentLongGCDisruption NetworkDisruption NetworkDisconnectPartition NetworkDelaysPartition NetworkUnresponsivePartition Resiliency tests Service disruption schemes 77
78. ‹#› User input is very important 78
79. Resiliency updates Where to go 79 Reporting: https://github.com/elastic/elasticsearch/issues 1 Tracking: https://github.com/elastic/elasticsearch/issues?q=label%3Aresiliency 3 Status: https://www.elastic.co/guide/en/elasticsearch/resiliency/current/index.html 4 2015: https://www.elastic.co/elasticon/2015/sf/resiliency-in-elasticsearch-and-lucene 5 Discourse: https://discuss.elastic.co/c/elasticsearch 2