Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Make sense of your (BIG) data!

Make sense of your (BIG) data!

Presented at Esprit JUG 2014, Tunisia

Elasticsearch is a search server based on Lucene. It provides a distributed, multitenant-capable full-text search engine with a RESTful web interface and schema-free JSON documents. You can use it as well to compute information on live data. Kibana is a full web application built with AngularJS. It's a generic and powerful visualisation tool for your data. Both projects are under Apache 2 License. In this presentation, you will discover how Elasticsearch actually works. We will inject marketing data into Elasticsearch and build live a dashboard using Kibana. In minutes, you will know how to build YOUR own dashboard and make sense of YOUR data.

Elasticsearch Inc

May 07, 2014
Tweet

More Decks by Elasticsearch Inc

Other Decks in Technology

Transcript

  1. #elasticsearch MAKE SENSE OF YOUR (BIG) DATA! David Pilato Technical

    advocate! ! elasticsearch. @dadoonet
  2. None
  3. #elasticsearch data ?

  4. #elasticsearch

  5. #elasticsearch

  6. #elasticsearch

  7. #elasticsearch

  8. #elasticsearch

  9. #elasticsearch BIG data ?

  10. #elasticsearch BIG data ?

  11. #elasticsearch Source: http://www.csc.com/insights/flxwd/78931-big_data_just_beginning_to_explode 35.000.000.000.000.000 mb

  12. #elasticsearch Source: http://www.domo.com/learn/data-never-sleeps-2

  13. None
  14. #elasticsearch search = like % ? SELECT ! doc.*, country.*

    ! FROM ! doc, country! WHERE ! doc.country_code = country.code AND! doc.date_doc > to_date('2011-12', 'yyyy-mm') AND ! doc.date_doc < to_date('2012-01', 'yyyy-mm') AND ! lower(country.name) = 'france' AND ! lower(doc.comment) LIKE ‘%product%' AND lower(doc.comment) LIKE ‘%david%';
  15. #elasticsearch Search engine ?

  16. #elasticsearch plug & play REST/JSON scalable Apache 2 license Lucene

    elasticsearch
  17. #elasticsearch Start… $ wget https://download.elasticsearch.org/elasticsearch/ elasticsearch/elasticsearch-1.1.1.tar.gz! $ tar -xf elasticsearch-1.1.1.tar.gz!

    $ ./elasticsearch-1.1.0/bin/elasticsearch! [INFO ][node ][Ghost Maker] {1.1.1}[5645]: initializing
  18. #elasticsearch … and play! $ curl -XPUT localhost:9200/sessions/session/1 -d '{!

    "title" : "Elasticsearch",! "subtitle" : "Make sense of your (BIG) data !",! "date" : "2014-05-07T15:00:00",! "tags" : [ "elasticsearch", "espritjug", "bigdata" ],! "speakers" : [{! "first_name" : "David", ! "last_name" : "Pilato" ! }]! }'
  19. #elasticsearch Search! $ curl http://localhost:9200/sessions/session/_search -d' { "query": { "multi_match":

    { "query": "elasticsearch espritjug david", "fields": [ "title^3", "tags^2", "speakers.first_name" ] } }, "post_filter": { "range": { "date": { "from": "2014-05-07", "to": "2014-05-09" } } } }'
  20. #elasticsearch Compute?

  21. #elasticsearch $ curl http://localhost:9200/sessions/session/_search -d' { "query": { ... },

    "aggs": { "by_date": { "date_histogram": { "field": "date", "interval": "day", "format" : "dd/MM/yyyy" } } } }' "by_date": [ { "key_as_string": "07/05/2014", "doc_count": 1 }, { "key_as_string": "08/05/2014", "doc_count": 2 }, { "key_as_string": "09/05/2014", "doc_count": 3 } ] Compute!
  22. None
  23. #mstechdays #elasticsearch StartUp #elasticsearch • logs! • twitter! • github!

    • marketing data! • ...! • your data! • your big data Let’s make sense of …
  24. #mstechdays #elasticsearch StartUp #elasticsearch • logs! • twitter! • github!

    • marketing data! • ...! • your data! • your big data Let’s make sense of … { "name":"Pilato David", "dateOfBirth":"1971-12-26", "gender":"male", "children":3, "marketing":{ "fashion":334, "music":3363, "hifi":2351 }, "address":{ "country":"France", "city":"Paris", "location": [2.332395, 48.861871] } }
  25. démo #mstechdays #elasticsearch StartUp #elasticsearch MAKE SENSE OF YOUR (BIG)

    DATA! let’s inject some marketing documents…
  26. #elasticsearch ELASTICSEARCH

  27. #elasticsearch Distributed indices node 1 orders 1 2 3 4

    $ curl -XPUT localhost:9200/orders -d '{! "settings.index.number_of_shards" : 4,! "settings.index.number_of_replicas" : 1! }'
  28. #elasticsearch Distributed indices node 1 orders products 1 2 3

    4 1 2 $ curl -XPUT localhost:9200/orders -d '{! "settings.index.number_of_shards" : 4,! "settings.index.number_of_replicas" : 1! }' $ curl -XPUT localhost:9200/products -d '{! "settings.index.number_of_shards" : 2,! "settings.index.number_of_replicas" : 0! }'
  29. #elasticsearch Distributed indices node 1 orders products 1 2 3

    4 1 2 node 2 $ bin/elasticsearch! [INFO ][cluster.service][Armageddon] detected_master [Ghost Maker]
  30. #elasticsearch Distributed indices node 1 orders products 1 4 1

    node 2 orders products 2 3 2 2 3 1 4
  31. #elasticsearch node 3 Distributed indices node 1 orders products 1

    4 1 node 2 orders products 2 3 2 2 3 1 4 $ bin/elasticsearch! [INFO ][cluster.service][Karnak] detected_master [Ghost Maker]
  32. #elasticsearch node 3 products orders Distributed indices node 1 orders

    products 1 4 1 node 2 orders products 2 3 2 2 3 1 4
  33. elasticsearch. elasticsearch kibana logstash Marvel

  34. elasticsearch. Training (public and on-site)

  35. elasticsearch. Training (public and on-site) Development support Production support

  36. elasticsearch. Training (public and on-site) Development support Production support Marvel

  37. None
  38. @dadoonet questions ? we are hiring! [email protected]

  39. @dadoonet اركش