Make sense of your (BIG) data on Azure!

Presented by David Pilato at Microsoft Tech Days Paris. These slides are en francais.

Sous licence Apache2, elasticsearch est un moteur de recherche puissant, distribué et scalable. Il fournit également des agrégations en temps réel en fonction de vos besoins. Couplé à Kibana, dashboard générique et hautement personnalisable, il vous permet de donner immédiatement du sens à vos données. En forte progression au niveau de son adhésion par les entreprises et les sites publics, découvrez ce que sont elasticsearch et Kibana et à quel point il est simple de les déployer facilement sur la plate-forme Windows Azure. Thomas et David illustreront à l'aide de cas clients les bénéfices obtenus à travers ces solutions.

Transcript

2. StartUp Elasticsearch sur Azure
 Make sense of your (BIG) data ! David Pilato / Thomas Conté! Développeurs - Evangélistes Elasticsearch / Microsoft @dadoonet / @tomconte
3. #mstechdays #elasticsearch StartUp Depuis votre smartphone sur : http://notes.mstechdays.fr ! De nombreux lots à gagner toute les heures !!! Claviers, souris et jeux Microsoft… ! Merci de nous aider à améliorer les Techdays ! Donnez votre avis !
4. #mstechdays #elasticsearch StartUp WINDOWS AZURE Rappels
5. #mstechdays #elasticsearch StartUp Back-ends Android, iOS & Node.js via Azure Mobile Services Windows Azure Plateforme Ouverte Languages Apps Data Operating Systems Devices SDKs pour Java, Python, PHP, Ruby, Node.js WebSites pour PHP, Node.js, Python & App Gallery MySQL via ClearDB, MongoDB via MongoLab, Hadoop via HDInsight Linux VMs via Image Gallery & VMDepot
6. #mstechdays #elasticsearch StartUp Web Sites Identity Cloud Services Blob Storage 3rd Party
 Services Virtual Machines Service Bus Caching Integration Big Data CDN Media Services Table Storage SQL 
 Azure Mobile Services Analytics “Plateforme Ouverte” Windows Azure : rappels
7. #mstechdays #elasticsearch StartUp ELASTICSEARCH Overview
8. #mstechdays #elasticsearch StartUp search = like % ?
9. #mstechdays #elasticsearch StartUp search = like % ? SELECT doc.*, pays.* FROM doc, pays WHERE doc.pays_code = pays.code AND doc.date_doc > to_date('2011-12', 'yyyy-mm') AND doc.date_doc < to_date('2012-01', 'yyyy-mm') AND lower(pays.libelle) = 'france' AND lower(doc.commentaire) LIKE ‘%produit%' AND lower(doc.commentaire) LIKE ‘%david%';
10. #mstechdays #elasticsearch StartUp Moteur de recherche ?
11. #mstechdays #elasticsearch StartUp Moteur de recherche ?
12. #mstechdays #elasticsearch StartUp elasticsearch ?
13. #mstechdays #elasticsearch StartUp elasticsearch ? Lucene
14. #mstechdays #elasticsearch StartUp elasticsearch ? REST/JSON Lucene
15. #mstechdays #elasticsearch StartUp elasticsearch ? REST/JSON scalable Lucene
16. #mstechdays #elasticsearch StartUp elasticsearch ? plug & play REST/JSON scalable Lucene
17. #mstechdays #elasticsearch StartUp plug & play REST/JSON scalable Apache 2 license Lucene elasticsearch
18. #mstechdays #elasticsearch StartUp Start… $ wget https://download.elasticsearch.org/elasticsearch/ elasticsearch/elasticsearch-1.0.0.tar.gz! $ tar -xf elasticsearch-1.0.0.tar.gz! $ ./elasticsearch-1.0.0/bin/elasticsearch! [INFO ][node ][Ghost Maker] {1.0.0}[5645]: initializing
19. #mstechdays #elasticsearch StartUp … and play! $ curl -XPUT localhost:9200/sessions/session/1 -d '{! "title" : "Elasticsearch sur Azure",! "subtitle" : "Make sense of your (BIG) data !",! "date" : "2014-02-13T16:30:00",! "tags" : [ "elasticsearch", "azure", "cloud" ],! "speaker" : [ {! "first_name" : "Thomas", ! "last_name" : "Conté" ! }, {! "first_name" : "David", ! "last_name" : "Pilato" ! }]! }'
20. #mstechdays #elasticsearch StartUp Search! $ curl -XPOST http://localhost:9200/sessions/session/_search -d' { "query": { "multi_match": { "query": "elasticsearch azure thomas", "fields": [ "title^2", "speaker.first_name" ] } }, "post_filter": { "range": { "date": { "from": "2014-02-13", "to": "2014-02-14" } } } }'
21. #mstechdays #elasticsearch StartUp Compute?
22. #mstechdays #elasticsearch StartUp $ curl -XPOST http://localhost:9200/sessions/session/_search -d' { "query": { ... }, "aggs": { "by_date": { "date_histogram": { "field": "date", "interval": "day", "format" : "dd/MM/yyyy" } } } }' Compute!
23. #mstechdays #elasticsearch StartUp $ curl -XPOST http://localhost:9200/sessions/session/_search -d' { "query": { ... }, "aggs": { "by_date": { "date_histogram": { "field": "date", "interval": "day", "format" : "dd/MM/yyyy" } } } }' "by_date": [ { "key_as_string": "11/02/2014", "doc_count": 1 }, { "key_as_string": "12/02/2014", "doc_count": 2 }, { "key_as_string": "13/02/2014", "doc_count": 3 } ] Compute!
24. #mstechdays #elasticsearch StartUp $ curl -XPOST http://localhost:9200/sessions/session/_search -d' { "query": { ... }, "aggs": { "by_date": { "date_histogram": { "field": "date", "interval": "day", "format" : "dd/MM/yyyy" } } } }' "by_date": [ { "key_as_string": "11/02/2014", "doc_count": 1 }, { "key_as_string": "12/02/2014", "doc_count": 2 }, { "key_as_string": "13/02/2014", "doc_count": 3 } ] Compute!
25. démo #mstechdays #elasticsearch StartUp MAKE SENSE OF YOUR (BIG) DATA! let’s inject some marketing documents…
26. #mstechdays #elasticsearch StartUp ELASTICSEARCH Elastique ? Distribué ?
27. #mstechdays #elasticsearch StartUp Distributed indices node 1
28. #mstechdays #elasticsearch StartUp Distributed indices node 1 $ curl -XPUT localhost:9200/orders -d '{! "settings.index.number_of_shards" : 4,! "settings.index.number_of_replicas" : 1! }'
29. #mstechdays #elasticsearch StartUp Distributed indices node 1 orders $ curl -XPUT localhost:9200/orders -d '{! "settings.index.number_of_shards" : 4,! "settings.index.number_of_replicas" : 1! }'
30. #mstechdays #elasticsearch StartUp Distributed indices node 1 orders 1 2 3 4 $ curl -XPUT localhost:9200/orders -d '{! "settings.index.number_of_shards" : 4,! "settings.index.number_of_replicas" : 1! }'
31. #mstechdays #elasticsearch StartUp Distributed indices node 1 orders 1 2 3 4 $ curl -XPUT localhost:9200/orders -d '{! "settings.index.number_of_shards" : 4,! "settings.index.number_of_replicas" : 1! }' $ curl -XPUT localhost:9200/products -d '{! "settings.index.number_of_shards" : 2,! "settings.index.number_of_replicas" : 0! }'
32. #mstechdays #elasticsearch StartUp Distributed indices node 1 orders products 1 2 3 4 1 2 $ curl -XPUT localhost:9200/orders -d '{! "settings.index.number_of_shards" : 4,! "settings.index.number_of_replicas" : 1! }' $ curl -XPUT localhost:9200/products -d '{! "settings.index.number_of_shards" : 2,! "settings.index.number_of_replicas" : 0! }'
33. #mstechdays #elasticsearch StartUp Start new nodes $ ./elasticsearch-1.0.0/bin/elasticsearch! [INFO ][node ][Armageddon] {1.0.0}[5645]: initializing
 [INFO ][cluster.service][Armageddon] detected_master [Ghost Maker]
34. #mstechdays #elasticsearch StartUp Start new nodes $ ./elasticsearch-1.0.0/bin/elasticsearch! [INFO ][node ][Armageddon] {1.0.0}[5645]: initializing
 [INFO ][cluster.service][Armageddon] detected_master [Ghost Maker] $ ./elasticsearch-1.0.0/bin/elasticsearch! [INFO ][node ][Cyclops] {1.0.0}[5645]: initializing
 [INFO ][cluster.service][Cyclops] detected_master [Ghost Maker]
35. #mstechdays #elasticsearch StartUp Start new nodes $ ./elasticsearch-1.0.0/bin/elasticsearch! [INFO ][node ][Armageddon] {1.0.0}[5645]: initializing
 [INFO ][cluster.service][Armageddon] detected_master [Ghost Maker] $ ./elasticsearch-1.0.0/bin/elasticsearch! [INFO ][node ][Cyclops] {1.0.0}[5645]: initializing
 [INFO ][cluster.service][Cyclops] detected_master [Ghost Maker] $ ./elasticsearch-1.0.0/bin/elasticsearch! [INFO ][node ][Karnak] {1.0.0}[5645]: initializing
 [INFO ][cluster.service][Karnak] detected_master [Ghost Maker]
36. #mstechdays #elasticsearch StartUp Distributed indices node 1 orders products 1 2 3 4 1 2
37. #mstechdays #elasticsearch StartUp Distributed indices node 1 orders products 1 2 3 4 1 2 $ ./elasticsearch-1.0.0/bin/elasticsearch! [INFO ][node ][Armageddon] {1.0.0}[5645]: initializing
 [INFO ][cluster.service][Armageddon] detected_master [Ghost Maker]
38. #mstechdays #elasticsearch StartUp Distributed indices node 1 orders products 1 2 3 4 1 2 node 2 $ ./elasticsearch-1.0.0/bin/elasticsearch! [INFO ][node ][Armageddon] {1.0.0}[5645]: initializing
 [INFO ][cluster.service][Armageddon] detected_master [Ghost Maker]
39. #mstechdays #elasticsearch StartUp Distributed indices node 1 orders products 1 4 1 node 2 orders products 2 3 2 3 2
40. #mstechdays #elasticsearch StartUp Distributed indices node 1 orders products 1 4 1 node 2 orders products 2 3 2 3 1 4 2
41. #mstechdays #elasticsearch StartUp Distributed indices node 1 orders products 1 4 1 node 2 orders products 2 3 2 2 3 1 4
42. #mstechdays #elasticsearch StartUp Distributed indices node 1 orders products 1 4 1 node 2 orders products 2 3 2 2 3 1 4 $ ./elasticsearch-1.0.0/bin/elasticsearch! [INFO ][node ][Karnak] {1.0.0}[5645]: initializing
 [INFO ][cluster.service][Karnak] detected_master [Ghost Maker]
43. #mstechdays #elasticsearch StartUp node 3 Distributed indices node 1 orders products 1 4 1 node 2 orders products 2 3 2 2 3 1 4 $ ./elasticsearch-1.0.0/bin/elasticsearch! [INFO ][node ][Karnak] {1.0.0}[5645]: initializing
 [INFO ][cluster.service][Karnak] detected_master [Ghost Maker]
44. #mstechdays #elasticsearch StartUp node 3 products orders Distributed indices node 1 orders products 1 4 1 node 2 orders products 2 3 3 2 2 3 1 4
45. #mstechdays #elasticsearch StartUp node 3 products orders Distributed indices node 1 orders products 1 4 1 node 2 orders products 2 3 2 2 3 1 4
46. #mstechdays #elasticsearch StartUp ELASTICSEARCH SUR AZURE Discovery ?
47. #mstechdays #elasticsearch StartUp Deploying on Azure (unicast) VM node 1 private IP1 VM node 2 private IP2 discovery.zen.ping.multicast.enabled: false! discovery.zen.ping.unicast.hosts: ["ip1", "ip2"] Cloud service: your-es.cloudapp.net
48. #mstechdays #elasticsearch StartUp VM node 3 private IP3 Deploying on Azure (unicast) VM node 1 private IP1 VM node 2 private IP2 Cloud service: your-es.cloudapp.net discovery.zen.ping.multicast.enabled: false! discovery.zen.ping.unicast.hosts: ["ip1", "ip2", "ip3"]
49. #mstechdays #elasticsearch StartUp Deploying on Azure (cloud plugin) Azure REST API VM node 3 private IP3 VM node 1 private IP1 VM node 2 private IP2 Cloud service: your-es.cloudapp.net
50. #mstechdays #elasticsearch StartUp Deploying on Azure (cloud plugin) Azure REST API VM node 3 private IP3 VM node 1 private IP1 VM node 2 private IP2 Cloud service: your-es.cloudapp.net cloud:! azure:! keystore: /path/to/keystore! password: your_password_for_keystore! subscription_id: your_azure_subscription_id! service_name: your-es! discovery:! type: azure
51. #mstechdays #elasticsearch StartUp https://github.com/elasticsearch/elasticsearch-cloud-azure
52. démo #mstechdays #elasticsearch StartUp ELASTICSEARCH ON AZURE Scale out!
53. © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation . Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION. Digital is business @dadoonet / @tomconte QUESTIONS ?
54. © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation . Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION. Digital is business @dadoonet / @tomconte QUESTIONS ?