Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Make sense of your (BIG) data!

Make sense of your (BIG) data!

Presented by David Pilato at the CloudConf.IT 2014 conference.

Elasticsearch is a search server based on Lucene. It provides a distributed, multitenant-capable full-text search engine with a RESTful web interface and schema-free JSON documents. You can use it as well to compute information on live data. Kibana is a full web application built with AngularJS. It's a generic and powerful visualisation tool for your data. Both projects are under Apache 2 License. In this presentation, you will discover how Elasticsearch actually works. We will inject marketing data into Elasticsearch and build live a dashboard using Kibana. In minutes, you will know how to build YOUR own dashboard and make sense of YOUR data.

Elasticsearch Inc

April 03, 2014
Tweet

More Decks by Elasticsearch Inc

Other Decks in Technology

Transcript

  1. #elasticsearch MAKE SENSE OF YOUR (BIG) DATA! David Pilato Technical

    advocate ! elasticsearch. @dadoonet
  2. None
  3. StartUp #elasticsearch data ?

  4. StartUp #elasticsearch

  5. StartUp #elasticsearch

  6. StartUp #elasticsearch

  7. StartUp #elasticsearch

  8. StartUp #elasticsearch

  9. #elasticsearch BIG data ?

  10. #elasticsearch Source: http://www.csc.com/insights/flxwd/78931-big_data_just_beginning_to_explode 35.000.000.000.000.000 mb

  11. #elasticsearch Source: http://www.thebigdatainsightgroup.com/site/article/big-data-infographic

  12. None
  13. StartUp #elasticsearch search = like % ? SELECT doc.*, country.*

    FROM doc, country WHERE doc.country_code = country.code AND doc.date_doc > to_date('2011-12', 'yyyy-mm') AND doc.date_doc < to_date('2012-01', 'yyyy-mm') AND lower(country.name) = 'france' AND lower(doc.comment) LIKE ‘%product%' AND lower(doc.comment) LIKE ‘%david%';
  14. StartUp #elasticsearch Search engine ?

  15. #elasticsearch elasticsearch ? plug & play REST/JSON scalable Apache 2

    license Lucene elasticsearch
  16. #elasticsearch Start… $ wget https://download.elasticsearch.org/elasticsearch/ elasticsearch/elasticsearch-1.1.0.tar.gz! $ tar -xf elasticsearch-1.1.0.tar.gz!

    $ ./elasticsearch-1.1.0/bin/elasticsearch! [INFO ][node ][Ghost Maker] {1.1.0}[5645]: initializing
  17. #elasticsearch … and play! $ curl -XPUT localhost:9200/sessions/session/1 -d '{!

    "title" : "Elasticsearch",! "subtitle" : "Make sense of your (BIG) data !",! "date" : "2014-04-03T10:30:00",! "tags" : [ "elasticsearch", "cloudconf", "bigdata" ],! "speaker" : [{! "first_name" : "David", ! "last_name" : "Pilato" ! }]! }'
  18. #elasticsearch Search! $ curl http://localhost:9200/sessions/session/_search -d' { "query": { "multi_match":

    { "query": "elasticsearch cloudconf david", "fields": [ "title^3", "tags^2", "speaker.first_name" ] } }, "post_filter": { "range": { "date": { "from": "2014-04", "to": "2014-05" } } } }'
  19. StartUp #elasticsearch Compute?

  20. #elasticsearch $ curl http://localhost:9200/sessions/session/_search -d' { "query": { ... },

    "aggs": { "by_date": { "date_histogram": { "field": "date", "interval": "day", "format" : "dd/MM/yyyy" } } } }' "by_date": [ { "key_as_string": "03/04/2014", "doc_count": 1 }, { "key_as_string": "12/04/2014", "doc_count": 2 }, { "key_as_string": "16/04/2014", "doc_count": 3 } ] Compute!
  21. #elasticsearch

  22. #elasticsearch Let’s make sense of … • logs • twitter

    • github • marketing data • ... • your data • your big data
  23. #elasticsearch { "name":"Pilato David", "dateOfBirth":"1971-12-26", "gender":"male", "children":3, "marketing":{ "fashion":334, "music":3363,

    "hifi":2351 }, "address":{ "country":"France", "city":"Paris", "location": [2.332395, 48.861871] } } Let’s make sense of … • logs • twitter • github • marketing data • ... • your data • your big data
  24. démo #mstechdays #elasticsearch StartUp #elasticsearch MAKE SENSE OF YOUR (BIG)

    DATA! let’s inject some marketing documents…
  25. #elasticsearch ELASTICSEARCH elastic? distributed?

  26. StartUp #elasticsearch Distributed indices node 1 orders products 1 2

    3 4 1 2 $ curl -XPUT localhost:9200/orders -d '{! "settings.index.number_of_shards" : 4,! "settings.index.number_of_replicas" : 1! }' $ curl -XPUT localhost:9200/products -d '{! "settings.index.number_of_shards" : 2,! "settings.index.number_of_replicas" : 0! }'
  27. StartUp #elasticsearch Distributed indices node 1 orders products 1 2

    3 4 1 2 node 2 $ bin/elasticsearch! [INFO ][cluster.service][Armageddon] detected_master [Ghost Maker]
  28. StartUp #elasticsearch Distributed indices node 1 orders products 1 4

    1 node 2 orders products 2 3 2 2 3 1 4 2 3 2
  29. StartUp #elasticsearch node 3 Distributed indices node 1 orders products

    1 4 1 node 2 orders products 2 3 2 2 3 1 4 $ bin/elasticsearch! [INFO ][cluster.service][Karnak] detected_master [Ghost Maker]
  30. StartUp #elasticsearch node 3 products orders Distributed indices node 1

    orders products 1 4 1 node 2 orders products 2 3 3 2 2 3 1 4 3 1 4
  31. elasticsearch. elasticsearch kibana logstash Marvel

  32. elasticsearch. Training (public and on-site) Development support Production support Marvel

  33. None
  34. @dadoonet questions ? we are hiring! [email protected]