Node discovery in cloud environment

Talk given for Elasticsearch FR meetup #12
At Microsoft France
http://www.meetup.com/elasticsearchfr/events/219274450/

Dans « Elasticsearch », il y a « Elastic ». Démarrer des nouveaux noeuds pour augmenter la capacité du cluster est très simple avec le mode multicast par défaut.Les plates-formes cloud interdisent évidemment le multicast. Aussi, comment garder cette élasticité simple en environnement cloud ?Venez découvrir ce qui se cache derrière les plugins cloud pour Amazon, Google et bien sûr Azure !

Transcript

1. node discovery in cloud environment @dadoonet
2. 2 Building a 3 nodes cluster The slides and video will be made available to you after the webinar. Bare Metal node 1 IP1 Bare Metal node 2 IP2 Bare Metal node 3 IP3
3. discovery
 multicast
4. 4 Discovery with multicast The slides and video will be made available to you after the webinar. Bare Metal node 1 IP1 NETWORK UDP Group: 224.2.2.4
 Port: 54328
5. 5 Discovery with multicast The slides and video will be made available to you after the webinar. Bare Metal node 1 IP1 NETWORK Bare Metal node 2 IP2 UDP Group: 224.2.2.4
 Port: 54328
6. 6 Discovery with multicast The slides and video will be made available to you after the webinar. Bare Metal node 1 IP1 NETWORK Bare Metal node 2 IP2 UDP Group: 224.2.2.4
 Port: 54328
7. 7 Discovery with multicast The slides and video will be made available to you after the webinar. NETWORK Bare Metal node 1 IP1 Bare Metal node 2 IP2
8. 8 Discovery with multicast The slides and video will be made available to you after the webinar. NETWORK Bare Metal node 1 IP1 Bare Metal node 2 IP2 Bare Metal node 3 IP3 UDP Group: 224.2.2.4
 Port: 54328
9. 9 Discovery with multicast The slides and video will be made available to you after the webinar. NETWORK Bare Metal node 1 IP1 Bare Metal node 2 IP2 Bare Metal node 3 IP3 UDP Group: 224.2.2.4
 Port: 54328
10. 10 Discovery with multicast The slides and video will be made available to you after the webinar. NETWORK Bare Metal node 1 IP1 Bare Metal node 2 IP2 Bare Metal node 3 IP3
11. discovery
 unicast
12. 12 Discovery with unicast The slides and video will be made available to you after the webinar. Bare Metal node 1 IP1 NETWORK TCP ip1
 Port: 9300 config/elaticsearch.yml discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["ip1"]
13. 13 Discovery with unicast The slides and video will be made available to you after the webinar. Bare Metal node 1 IP1 NETWORK Bare Metal node 2 IP2 config/elaticsearch.yml discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["ip1","ip2"] TCP ip1, ip2
 Port: 9300
14. 14 Discovery with unicast The slides and video will be made available to you after the webinar. Bare Metal node 1 IP1 NETWORK Bare Metal node 2 IP2 config/elaticsearch.yml discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["ip1","ip2"] TCP ip1, ip2
 Port: 9300
15. 15 Discovery with unicast The slides and video will be made available to you after the webinar. NETWORK Bare Metal node 1 IP1 Bare Metal node 2 IP2 config/elaticsearch.yml discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["ip1","ip2"]
16. 16 Discovery with unicast The slides and video will be made available to you after the webinar. NETWORK Bare Metal node 1 IP1 Bare Metal node 2 IP2 Bare Metal node 3 IP3 TCP ip1, ip2, ip3
 Port: 9300 config/elaticsearch.yml discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["ip1","ip2","ip3"]
17. 17 Discovery with unicast The slides and video will be made available to you after the webinar. NETWORK Bare Metal node 1 IP1 Bare Metal node 2 IP2 Bare Metal node 3 IP3 TCP ip1, ip2, ip3
 Port: 9300 config/elaticsearch.yml discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["ip1","ip2","ip3"]
18. 18 Discovery with unicast The slides and video will be made available to you after the webinar. NETWORK Bare Metal node 1 IP1 Bare Metal node 2 IP2 Bare Metal node 3 IP3
19. discovery
 cloud environment
20. 20 Cloud constraints The slides and video will be made available to you after the webinar. • IP can change • Number of nodes should scale easily • Multicast is forbidden Problems config/elaticsearch.yml discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["ip1", "ip2", "ip3", "ip4", "ip5", "ip6", "ip7", "ip8", "ip9", "ip10", "ip11", "ip12", "ip13", "ip14", "ip15", "ip16", "ip17", "ip18"]
21. 21 Discovery with cloud plugin The slides and video will be made available to you after the webinar. VM node 1 IP1 NETWORK CLOUD PROVIDER API
22. 22 Discovery with cloud plugin The slides and video will be made available to you after the webinar. VM node 1 IP1 NETWORK CLOUD PROVIDER API discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["ip1"]
23. 23 Discovery with cloud plugin The slides and video will be made available to you after the webinar. VM node 1 IP1 NETWORK TCP ip1
 Port: 9300 CLOUD PROVIDER API discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["ip1"]
24. 24 Discovery with cloud plugin The slides and video will be made available to you after the webinar. VM node 1 IP1 NETWORK VM node 2 IP2 CLOUD PROVIDER API
25. 25 Discovery with cloud plugin The slides and video will be made available to you after the webinar. VM node 1 IP1 NETWORK VM node 2 IP2 discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["ip1","ip2"] CLOUD PROVIDER API
26. 26 Discovery with cloud plugin The slides and video will be made available to you after the webinar. VM node 1 IP1 NETWORK VM node 2 IP2 TCP ip1, ip2
 Port: 9300 discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["ip1","ip2"] CLOUD PROVIDER API
27. 27 Discovery with cloud plugin The slides and video will be made available to you after the webinar. NETWORK VM node 1 IP1 VM node 2 IP2 discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["ip1","ip2"] CLOUD PROVIDER API
28. 28 Discovery with cloud plugin The slides and video will be made available to you after the webinar. NETWORK VM node 1 IP1 VM node 2 IP2 VM node 3 IP3 CLOUD PROVIDER API
29. 29 Discovery with cloud plugin The slides and video will be made available to you after the webinar. NETWORK VM node 1 IP1 VM node 2 IP2 VM node 3 IP3 TCP ip1, ip2, ip3
 Port: 9300 discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["ip1","ip2","ip3"] CLOUD PROVIDER API
30. 30 Discovery with cloud plugin The slides and video will be made available to you after the webinar. NETWORK VM node 1 IP1 VM node 2 IP2 VM node 3 IP3 TCP ip1, ip2, ip3
 Port: 9300 discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["ip1","ip2","ip3"] CLOUD PROVIDER API
31. 31 Discovery with cloud plugin The slides and video will be made available to you after the webinar. NETWORK VM node 1 IP1 VM node 2 IP2 VM node 3 IP3 CLOUD PROVIDER API
32. 32 Discovery on Google Compute Engine The slides and video will be made available to you after the webinar. • Install gce plugin • Update config/elasticsearch.yml • Start elasticsearch https://github.com/elasticsearch/elasticsearch-cloud-gce cloud: gce: project_id: es-cloud zone: europe-west1-a discovery: type: gce
33. 33 Discovery on EC2 The slides and video will be made available to you after the webinar. • Install aws plugin • Update config/elasticsearch.yml • Start elasticsearch https://github.com/elasticsearch/elasticsearch-cloud-aws cloud: aws: access_key: AKVAIQBF2RECL7FJWGJQ secret_key: vExyMThREXeRMm/b/LRzEB8jWwvzQeXgjqMX+6br discovery: type: ec2
34. 34 Discovery on Azure VM The slides and video will be made available to you after the webinar. • Install azure plugin • Update config/elasticsearch.yml • Start elasticsearch https://github.com/elasticsearch/elasticsearch-cloud-azure cloud: azure: keystore: /path/to/keystore password: your_password_for_keystore subscription_id: your_azure_subscription_id service_name: your_azure_cloud_service_name discovery: type: azure
36. 36 Q&A Time Join the Community 
 www.elasticsearch.org/community Twitter: (@elasticsearch) IRC: #elasticsearch on irc.freenode.net Shield: It’s Coming Be among the first to know when Shield, security for Elasticsearch, is available. www.elasticsearch.com/ products/shield Elasticsearch Subscriptions Elasticsearch Subscriptions cover you in development and production with architecture guidance and best practices. www.elasticsearch.com/support The first Elasticsearch user conference. www.elasticon.com