Amsterdam Elasticsearch meetup: Cross Cluster Search with Elasticsearch

Transcript

1. ‹#› @lucacavanna Cross Cluster Search with Elasticsearch
2. Agenda 2 Why Cross Cluster Search? How does it work? Follow-up search improvements 1 2 3 4 How does it compare to Tribe node?
3. ‹#› Why Cross Cluster Search?
6. ‹#› How does Cross Cluster Search work?
7. ‹#› Search detour
8. cluster Search api 8 Client node1 logs 2P posts 1P node2 logs 3P users 1P node3 logs 1P posts 2P GET /posts/_search posts index: 2 primaries logs index: 3 primaries users index: 1 primary
9. cluster Search request parsing 9 Client node1 logs 2P posts 1P node2 logs 3P users 1P node3 logs 1P posts 2P The coordinating node parses the request GET /posts/_search
10. cluster Query phase 10 Client node1 logs 2P posts 1P node2 logs 3P users 1P node3 logs 1P posts 2P GET /posts/_search The query gets executed on the relevant shards The query gets executed on the relevant shards
11. cluster Reduce phase 11 Client node1 logs 2P posts 1P node2 logs 3P users 1P node3 logs 1P posts 2P GET /posts/_search The coordinating node receives "size" hits per shard and performs reduction
12. cluster Fetch phase 12 Client node1 logs 2P posts 1P node2 logs 3P users 1P node3 logs 1P posts 2P GET /posts/_search The coordinating node fetches the top hits from the relevant shards The coordinating node fetches the top hits from the relevant shards
13. cluster Search response 13 Client node1 logs 2P posts 1P node2 logs 3P users 1P node3 logs 1P posts 2P The coordinating node returns the top hits back to the client
14. ‹#› Cross Cluster Search
15. 15 europe eu1 posts 1P users 1P eu2 posts 3P posts 2P usa us1 posts 2P posts 1P australia au1 posts 1P au2 logs 1P
16. Register remote clusters PUT /_cluster/settings { "persistent" : { "search.remote" : { "australia" : { "seeds": ["host_au1:9300", "host_au2:9300"] }, "usa" : { "seeds": "host_us1:9300" } } } } 16
17. 17 Client europe eu1 posts 1P users 1P eu2 posts 3P posts 2P usa us1 posts 2P posts 1P australia au1 posts 1P au2 logs 1P GET /posts,usa:posts,australia:posts/_search The coordinating node parses the request
18. 18 Client europe eu1 posts 1P users 1P eu2 posts 3P posts 2P usa us1 posts 2P posts 1P australia The coordinating node fetches info about remote indices and their shards au2 logs 1P au1 posts 1P GET /posts,usa:posts,australia:posts/_search
19. 19 Client europe eu1 posts 1P users 1P eu2 posts 3P posts 2P usa us1 posts 2P posts 1P australia posts 1P The query gets executed on the relevant local shards au2 logs 1P au1 posts 1P GET /posts,usa:posts,australia:posts/_search
20. 20 Client europe eu1 posts 1P users 1P eu2 posts 3P posts 2P usa us1 posts 2P posts 1P australia The query gets also executed on the relevant remote shards au2 logs 1P au1 posts 1P GET /posts,usa:posts,australia:posts/_search
21. 21 Client europe eu1 posts 1P users 1P eu2 posts 3P posts 2P usa us1 posts 2P posts 1P australia au2 logs 1P au1 posts 1P The coordinating node receives "size" hits per shard and performs reduction GET /posts,usa:posts,australia:posts/_search
22. 22 Client europe eu1 posts 1P users 1P eu2 posts 3P posts 2P usa us1 posts 2P posts 1P australia The coordinating node fetches the top hits from the relevant shards au2 logs 1P au1 posts 1P GET /posts,usa:posts,australia:posts/_search
23. 23 Client europe eu1 posts 1P users 1P eu2 posts 3P posts 2P usa us1 posts 2P posts 1P australia The coordinating node returns the top hits back to the client au2 logs 1P au1 posts 1P GET /posts,usa:posts,australia:posts/_search
24. Search response "hits" : [ { "_index" : "australia:posts", ... }, { "_index" : "posts", ... }, { "_index" : "usa:posts", ... } ] 24
25. Which nodes connect to which remote nodes? 25
26. 26 europe eu1 posts 1P PUT /_cluster/settings { "persistent" : { "search.remote" : { "australia" : { "seeds": ["host_au1", "host_au3"] } } } } australia au1 posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P eu2 posts 2P
27. europe 27 eu1 posts 1P australia au1 posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P eu2 posts 2P Fetch the cluster state and select remote nodes PUT /_cluster/settings { "persistent" : { "search.remote" : { "australia" : { "seeds": ["host_au1", "host_au3"] } } } }
28. Disable cross cluster calls from specific nodes cat config/elasticsearch.yml # ---------------- Cluster ---------------- # # Use a descriptive name for your cluster: # cluster.name: europe # # ---------------- Node -------------------- # # Use a descriptive name for the node: # node.name: eu2 # Disable connections to remote clusters from this node search.remote.connect: false 28
29. europe 29 eu1 posts 1P eu2 posts 2P Fetch the cluster state and select remote nodes PUT /_cluster/settings { "persistent" : { "search.remote" : { "australia" : { "seeds": ["host_au1", "host_au3"] } } } } australia au1 posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P
30. europe 30 eu1 posts 1P eu2 posts 2P Fetch the cluster state and select remote nodes PUT /_cluster/settings { "persistent" : { "search.remote" : { "australia" : { "seeds": ["host_au1", "host_au3"] } } } } australia au1 posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P
31. europe 31 eu1 posts 1P eu2 posts 2P Fetch the cluster state and select remote nodes PUT /_cluster/settings { "persistent" : { "search.remote" : { "australia" : { "seeds": ["host_au1", "host_au3"] } } } } australia au1 posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P
32. Maximum number of remote nodes to connect to cat config/elasticsearch.yml # ---------------- Cluster ---------------- # # Use a descriptive name for your cluster: # cluster.name: europe # # ---------------- Node -------------------- # # Use a descriptive name for the node: # node.name: eu1 # Max number of remote nodes to connect to search.remote.connections_per_cluster: 3 32
33. europe 33 eu1 posts 1P eu2 posts 2P australia au1 posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P Client GET /australia:posts/_search
34. europe 34 eu1 posts 1P eu2 posts 2P australia au1 posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P Client GET /australia:posts/_search The coordinating node fetches info about remote indices and their shards
35. europe 35 eu1 posts 1P eu2 posts 2P australia au1 posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P Client GET /australia:posts/_search The query gets executed on the relevant remote shards
36. europe 36 eu1 posts 1P eu2 posts 2P australia au1 posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P Client GET /australia:posts/_search The coordinating node fetches the top hits from the relevant shards
37. Pick node attribute to tag remote nodes with cat config/elasticsearch.yml # ---------------- Cluster ---------------- # # Use a descriptive name for your cluster: # cluster.name: europe # # ---------------- Node -------------------- # # Use a descriptive name for the node: # node.name: eu1 # Attribute that enables/disables remote connections search.remote.node.attr: gateway 37
38. cat config/elasticsearch.yml # ---------------- Cluster ---------------- # # Use a descriptive name for your cluster: # cluster.name: australia # # ---------------- Node -------------------- # # Use a descriptive name for the node: # node.name: au5 # # Add custom attributes to the node: # node.attr.gateway: true 38 Tag selected nodes as remote gateways (2)
39. europe 39 eu1 posts 1P eu2 posts 2P australia au1 posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P gateway PUT /_cluster/settings { "persistent" : { "search.remote" : { "australia" : { "seeds": ["host_au1", "host_au3"] } } } } Fetch the cluster state and select remote nodes
40. europe 40 eu1 posts 1P eu2 posts 2P australia au1 posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P Client GET /australia:posts/_search gateway
41. europe 41 eu1 posts 1P eu2 posts 2P australia au1 posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P Client GET /australia:posts/_search gateway The coordinating node fetches info about remote indices and their shards
42. europe 42 eu1 posts 1P eu2 posts 2P australia au1 posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P Client GET /australia:posts/_search The query gets executed on the relevant remote shards gateway
43. europe 43 eu1 posts 1P eu2 posts 2P australia au1 posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P Client GET /australia:posts/_search gateway The coordinating node fetches the top hits from the relevant shards
44. How about version compatibility? 44
45. europe 45 eu1 posts 1P eu2 posts 2P Remote nodes selection: - is node available? - is node gateway? - is version compatible? PUT /_cluster/settings { "persistent" : { "search.remote" : { "australia" : { "seeds": ["host_au1", "host_au3"] } } } } australia au1 posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P
46. ‹#› Upgrading to 6.0 with Rolling Restarts The hardest part about upgrading to a new major version has been the fact that you have to do a full cluster restart to get there. No more! You will be able to upgrade from the latest 5.x version to 6.0 using rolling restarts, without any cluster downtime. https://www.elastic.co/blog/elasticsearch-6-0-0-alpha1-released
47. europe 47 eu1 posts 1P eu2 posts 2P 6.0 won't be able to talk directly to 5.4, only to 5.latest PUT /_cluster/settings { "persistent" : { "search.remote" : { "australia" : { "seeds": ["host_au1", "host_au3"] } } } } australia au1 posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P 5.4 5.4 5.4 5.4 5.4 5.4 6.0 6.0
48. europe 48 eu1 posts 1P eu2 posts 2P PUT /_cluster/settings { "persistent" : { "search.remote" : { "australia" : { "seeds": "host_au7" } } } } australia au1 posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P 5.4 5.4 5.4 5.4 5.4 5.4 6.0 au7 5.latest Fetch the cluster state and select remote nodes 6.0
49. europe 49 eu1 posts 1P eu2 posts 2P PUT /_cluster/settings { "persistent" : { "search.remote" : { "australia" : { "seeds": "host_au7" } } } } australia au1 posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P 5.4 5.4 5.4 5.4 5.4 5.4 au7 5.latest Fetch the cluster state and select remote nodes 6.0 6.0
50. europe 50 eu1 posts 1P eu2 posts 2P australia au1 posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P 5.4 5.4 5.4 5.4 5.4 5.4 au7 5.latest 6.0 6.0 Client GET /australia:posts/_search
51. europe 51 eu1 posts 1P eu2 posts 2P australia au1 posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P 5.4 5.4 5.4 5.4 5.4 5.4 au7 5.latest 6.0 6.0 Client GET /australia:posts/_search The coordinating node fetches info about remote indices and their shards
52. europe 52 eu1 posts 1P eu2 posts 2P australia au1 posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P 5.4 5.4 5.4 5.4 5.4 5.4 au7 5.latest 6.0 6.0 Client GET /australia:posts/_search The query gets executed on the relevant remote shards
53. europe 53 eu1 posts 1P eu2 posts 2P australia au1 posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P 5.4 5.4 5.4 5.4 5.4 5.4 au7 5.latest 6.0 6.0 Client GET /australia:posts/_search The coordinating node fetches the top hits from the relevant shards
54. ‹#› Follow-up search improvements
55. What happens if I search against many shards? 55
56. europe 56 eu1 posts 1P eu2 posts 2P australia au1 posts 1P au2 posts 4P au3 posts 2P au4 posts 5P au6 posts 6P au5 posts 3P Client GET /australia:posts/_search gateway The coordinating node receives "size" hits per shard and performs reduction
57. Shard count limit (5.3) PUT /_cluster/settings { "persistent" : { "action.search.shard_count.limit" : 1000 } } 57
58. Multiple reduction phases (5.4) GET /posts,*:posts/_search?batched_reduce_size=512 { "query" : { … }, "aggs" : { … } } 58
59. Multiple reduction phases (5.4) { "took" : 43, "timed_out" : false, "num_reduce_phases": 6, "_shards" : { "total" : 3000, "successful" : 3000, "failed" : 0 }, "hits" : { "total" : 3987245, "max_score" : 1.0, "hits" : […] } } 59
60. Shard count limit (5.4) PUT /_cluster/settings { "persistent" : { "action.search.shard_count.limit" : Long.MAX_VALUE } } 60
61. ‹#› How does it compare to Tribe node?
62. • Adding remote clusters requires node restart 62 Tribe node Cross Cluster Search • Remote clusters can be dynamically registered
63. • Adding remote clusters requires node restart • Receives all cluster state updates from remote clusters 63 Tribe node Cross Cluster Search • Remote clusters can be dynamically registered • Retrieves info on demand from remote clusters
64. • Adding remote clusters requires node restart • Receives all cluster state updates from remote clusters • Doesn't support indices with same names on different clusters 64 Tribe node Cross Cluster Search • Remote clusters can be dynamically registered • Retrieves info on demand from remote clusters • No limitations on indices naming
65. • Adding remote clusters requires node restart • Receives all cluster state updates from remote clusters • Doesn't support indices with same names on different clusters • Requires an additional node (tribe) to join all the remote clusters 65 Tribe node Cross Cluster Search • Remote clusters can be dynamically registered • Retrieves info on demand from remote clusters • No limitations on indices naming • No additional nodes required
66. • Bi-directional connections to every remote node 66 Tribe node Cross Cluster Search • Uni-directional connections to selected gateway nodes
67. • Bi-directional connections to every remote node • Specific type of node with its own handling and edge cases, hard to test and maintain 67 Tribe node Cross Cluster Search • Uni-directional connections to selected gateway nodes • Isolated component integrated into a specific API: easier to test and maintain
68. • Bi-directional connections to every remote node • Specific type of node with its own handling and edge cases, hard to test and maintain • Works with almost every api 68 Tribe node Cross Cluster Search • Uni-directional connections to selected gateway nodes • Isolated component integrated into a specific API: easier to test and maintain • Specific to _search api
69. ‹#› https://www.elastic.co/downloads/elasticsearch https://www.elastic.co/guide/en/elasticsearch/reference/5.x/ modules-cross-cluster-search.html First released with Elasticsearch 5.3.0
70. ‹#› Thank you