Make sense of your (BIG) data!

Presented at Esprit JUG 2014, Tunisia

Elasticsearch is a search server based on Lucene. It provides a distributed, multitenant-capable full-text search engine with a RESTful web interface and schema-free JSON documents. You can use it as well to compute information on live data. Kibana is a full web application built with AngularJS. It's a generic and powerful visualisation tool for your data. Both projects are under Apache 2 License. In this presentation, you will discover how Elasticsearch actually works. We will inject marketing data into Elasticsearch and build live a dashboard using Kibana. In minutes, you will know how to build YOUR own dashboard and make sense of YOUR data.

Transcript

1. #elasticsearch MAKE SENSE OF YOUR (BIG) DATA! David Pilato Technical advocate! ! elasticsearch. @dadoonet
3. #elasticsearch data ?
4. #elasticsearch
5. #elasticsearch
6. #elasticsearch
7. #elasticsearch
8. #elasticsearch
9. #elasticsearch BIG data ?
10. #elasticsearch BIG data ?
11. #elasticsearch Source: http://www.csc.com/insights/flxwd/78931-big_data_just_beginning_to_explode 35.000.000.000.000.000 mb
12. #elasticsearch Source: http://www.domo.com/learn/data-never-sleeps-2
14. #elasticsearch search = like % ? SELECT ! doc.*, country.* ! FROM ! doc, country! WHERE ! doc.country_code = country.code AND! doc.date_doc > to_date('2011-12', 'yyyy-mm') AND ! doc.date_doc < to_date('2012-01', 'yyyy-mm') AND ! lower(country.name) = 'france' AND ! lower(doc.comment) LIKE ‘%product%' AND lower(doc.comment) LIKE ‘%david%';
15. #elasticsearch Search engine ?
16. #elasticsearch plug & play REST/JSON scalable Apache 2 license Lucene elasticsearch
17. #elasticsearch Start… $ wget https://download.elasticsearch.org/elasticsearch/ elasticsearch/elasticsearch-1.1.1.tar.gz! $ tar -xf elasticsearch-1.1.1.tar.gz! $ ./elasticsearch-1.1.0/bin/elasticsearch! [INFO ][node ][Ghost Maker] {1.1.1}[5645]: initializing
18. #elasticsearch … and play! $ curl -XPUT localhost:9200/sessions/session/1 -d '{! "title" : "Elasticsearch",! "subtitle" : "Make sense of your (BIG) data !",! "date" : "2014-05-07T15:00:00",! "tags" : [ "elasticsearch", "espritjug", "bigdata" ],! "speakers" : [{! "first_name" : "David", ! "last_name" : "Pilato" ! }]! }'
19. #elasticsearch Search! $ curl http://localhost:9200/sessions/session/_search -d' { "query": { "multi_match": { "query": "elasticsearch espritjug david", "fields": [ "title^3", "tags^2", "speakers.first_name" ] } }, "post_filter": { "range": { "date": { "from": "2014-05-07", "to": "2014-05-09" } } } }'
20. #elasticsearch Compute?
21. #elasticsearch $ curl http://localhost:9200/sessions/session/_search -d' { "query": { ... }, "aggs": { "by_date": { "date_histogram": { "field": "date", "interval": "day", "format" : "dd/MM/yyyy" } } } }' "by_date": [ { "key_as_string": "07/05/2014", "doc_count": 1 }, { "key_as_string": "08/05/2014", "doc_count": 2 }, { "key_as_string": "09/05/2014", "doc_count": 3 } ] Compute!
23. #mstechdays #elasticsearch StartUp #elasticsearch • logs! • twitter! • github! • marketing data! • ...! • your data! • your big data Let’s make sense of …
24. #mstechdays #elasticsearch StartUp #elasticsearch • logs! • twitter! • github! • marketing data! • ...! • your data! • your big data Let’s make sense of … { "name":"Pilato David", "dateOfBirth":"1971-12-26", "gender":"male", "children":3, "marketing":{ "fashion":334, "music":3363, "hifi":2351 }, "address":{ "country":"France", "city":"Paris", "location": [2.332395, 48.861871] } }
25. démo #mstechdays #elasticsearch StartUp #elasticsearch MAKE SENSE OF YOUR (BIG) DATA! let’s inject some marketing documents…
26. #elasticsearch ELASTICSEARCH
27. #elasticsearch Distributed indices node 1 orders 1 2 3 4 $ curl -XPUT localhost:9200/orders -d '{! "settings.index.number_of_shards" : 4,! "settings.index.number_of_replicas" : 1! }'
28. #elasticsearch Distributed indices node 1 orders products 1 2 3 4 1 2 $ curl -XPUT localhost:9200/orders -d '{! "settings.index.number_of_shards" : 4,! "settings.index.number_of_replicas" : 1! }' $ curl -XPUT localhost:9200/products -d '{! "settings.index.number_of_shards" : 2,! "settings.index.number_of_replicas" : 0! }'
29. #elasticsearch Distributed indices node 1 orders products 1 2 3 4 1 2 node 2 $ bin/elasticsearch! [INFO ][cluster.service][Armageddon] detected_master [Ghost Maker]
30. #elasticsearch Distributed indices node 1 orders products 1 4 1 node 2 orders products 2 3 2 2 3 1 4
31. #elasticsearch node 3 Distributed indices node 1 orders products 1 4 1 node 2 orders products 2 3 2 2 3 1 4 $ bin/elasticsearch! [INFO ][cluster.service][Karnak] detected_master [Ghost Maker]
32. #elasticsearch node 3 products orders Distributed indices node 1 orders products 1 4 1 node 2 orders products 2 3 2 2 3 1 4
33. elasticsearch. elasticsearch kibana logstash Marvel
34. elasticsearch. Training (public and on-site)
35. elasticsearch. Training (public and on-site) Development support Production support
36. elasticsearch. Training (public and on-site) Development support Production support Marvel
38. @dadoonet questions ? we are hiring! jobs@elasticsearch.com
39. @dadoonet اركش