JMaghreb 2014: Make sense of your (BIG) data!

Talk given at Casablanca, Morocco

Elasticsearch: you know, for search! But you can use it as well to compute information on live data.

During this session, you will discover how elasticsearch actually works behind the scene and why this great open source software is really more than "search".

We will inject marketing data into elasticsearch and build live a dashboard using Kibana, generic and powerful visualisation tool under Apache2 license.

In minutes, you will know how to build YOUR own dashboard and make sense of YOUR data!

Transcript

1. #elasticsearch MAKE SENSE OF YOUR (BIG) DATA! David Pilato Technical advocate elasticsearch. @dadoonet
3. StartUp #elasticsearch #elasticsearch data ?
4. StartUp #elasticsearch #elasticsearch
5. StartUp #elasticsearch #elasticsearch
6. StartUp #elasticsearch #elasticsearch
7. StartUp #elasticsearch #elasticsearch
8. StartUp #elasticsearch #elasticsearch
9. StartUp #elasticsearch #elasticsearch
10. StartUp #elasticsearch #elasticsearch
11. StartUp #elasticsearch #elasticsearch
12. StartUp #elasticsearch #elasticsearch
14. StartUp #elasticsearch #elasticsearch BIG data ?
15. StartUp #elasticsearch #elasticsearch Source: http://www.csc.com/insights/flxwd/78931-big_data_just_beginning_to_explode
16. StartUp #elasticsearch #elasticsearch Source: http://www.csc.com/insights/flxwd/78931-big_data_just_beginning_to_explode 35.000.000.000.000.000 mb
17. StartUp #elasticsearch #elasticsearch Source: http://www.domo.com/learn/data-never-sleeps-2
19. StartUp #elasticsearch #elasticsearch search = like % ?
20. StartUp #elasticsearch #elasticsearch search = like % ? SELECT doc.*, country.* FROM doc, country WHERE doc.country_code = country.code AND doc.date_doc > to_date('2011-12', 'yyyy-mm') AND doc.date_doc < to_date('2012-01', 'yyyy-mm') AND lower(country.name) = 'france' AND lower(doc.comment) LIKE '%product%' AND lower(doc.comment) LIKE '%david%';
21. StartUp #elasticsearch #elasticsearch Search engine ?
22. StartUp #elasticsearch #elasticsearch Search engine ?
23. StartUp #elasticsearch #elasticsearch elasticsearch ?
24. StartUp #elasticsearch #elasticsearch elasticsearch ? Lucene
25. StartUp #elasticsearch #elasticsearch elasticsearch ? REST/JSON Lucene
26. StartUp #elasticsearch #elasticsearch elasticsearch ? REST/JSON scalable Lucene
27. StartUp #elasticsearch #elasticsearch elasticsearch ? plug & play REST/JSON scalable Lucene
28. StartUp #elasticsearch #elasticsearch plug & play REST/JSON scalable Apache 2 license Lucene elasticsearch
29. #elasticsearch #elasticsearch Start… $ wget https://download.elasticsearch.org/elasticsearch/ elasticsearch/elasticsearch-1.4.0.tar.gz $ tar -xf elasticsearch-1.4.0.tar.gz $ ./elasticsearch-1.4.0/bin/elasticsearch [INFO ][node ][Ghost Maker] {1.4.0}[5645]: initializing
30. #elasticsearch #elasticsearch … and play! $ curl -XPUT localhost:9200/sessions/session/1 -d '{ "title" : "Elasticsearch", "subtitle" : "Make sense of your (BIG) data !", "date" : "2014-11-06T10:30:00", "tags" : [ "elasticsearch", "jmaghreb", "bigdata" ], "speakers" : [{ "first_name" : "David", "last_name" : "Pilato" }] }'
31. #elasticsearch #elasticsearch Search! $ curl http://localhost:9200/sessions/session/_search -d' { "query": { "multi_match": { "query": "elasticsearch jmaghreb david", "fields": [ "title^3", "tags^2", "speakers.first_name" ] } }, "post_filter": { "range": { "date": { "from": "2014-11-01", "to": "2014-12-01" } } } }'
32. StartUp #elasticsearch #elasticsearch Compute?
33. #elasticsearch #elasticsearch $ curl http://localhost:9200/sessions/session/_search -d' { "query": { ... }, "aggs": { "by_date": { "date_histogram": { "field": "date", "interval": "day", "format" : "dd/MM/yyyy" } } } }' Compute!
34. #elasticsearch #elasticsearch $ curl http://localhost:9200/sessions/session/_search -d' { "query": { ... }, "aggs": { "by_date": { "date_histogram": { "field": "date", "interval": "day", "format" : "dd/MM/yyyy" } } } }' "by_date": [ { "key_as_string": "03/04/2014", "doc_count": 1 }, { "key_as_string": "12/04/2014", "doc_count": 2 }, { "key_as_string": "16/04/2014", "doc_count": 3 } ] Compute!
35. #elasticsearch #elasticsearch $ curl http://localhost:9200/sessions/session/_search -d' { "query": { ... }, "aggs": { "by_date": { "date_histogram": { "field": "date", "interval": "day", "format" : "dd/MM/yyyy" } } } }' "by_date": [ { "key_as_string": "03/04/2014", "doc_count": 1 }, { "key_as_string": "12/04/2014", "doc_count": 2 }, { "key_as_string": "16/04/2014", "doc_count": 3 } ] Compute!
37. #mstechdays #elasticsearch StartUp #elasticsearch #elasticsearch • logs • twitter • github • marketing data • ... • your data • your big data Let’s make sense of …
38. #mstechdays #elasticsearch StartUp #elasticsearch #elasticsearch • logs • twitter • github • marketing data • ... • your data • your big data Let’s make sense of … { "name":"Pilato David", "dateOfBirth":"1971-12-26", "gender":"male", "children":3, "marketing":{ "fashion":334, "music":3363, "hifi":2351 }, "address":{ "country":"France", "city":"Paris", "location": [2.332395, 48.861871] } }
39. démo #mstechdays #elasticsearch StartUp #elasticsearch #elasticsearch MAKE SENSE OF YOUR (BIG) DATA! let’s inject some marketing documents…
40. #elasticsearch ELASTICSEARCH
41. elasticsearch.
42. elasticsearch. elasticsearch kibana logstash Marvel
43. elasticsearch.
44. elasticsearch. Training (public and on-site)
45. elasticsearch. Training (public and on-site) Development support Production support
46. elasticsearch. Training (public and on-site) Development support Production support Marvel
49. @dadoonet اركش we are hiring!