Make sense of your (BIG) data!

Presented by David Pilato at the CloudConf.IT 2014 conference.

Elasticsearch is a search server based on Lucene. It provides a distributed, multitenant-capable full-text search engine with a RESTful web interface and schema-free JSON documents. You can use it as well to compute information on live data. Kibana is a full web application built with AngularJS. It's a generic and powerful visualisation tool for your data. Both projects are under Apache 2 License. In this presentation, you will discover how Elasticsearch actually works. We will inject marketing data into Elasticsearch and build live a dashboard using Kibana. In minutes, you will know how to build YOUR own dashboard and make sense of YOUR data.

Transcript

1. #elasticsearch
   MAKE SENSE OF YOUR (BIG) DATA!
   David Pilato
   Technical advocate
   
   !
   elasticsearch.
   @dadoonet
   

   View Slide

2. View Slide

3. StartUp #elasticsearch
   data ?
   

   View Slide

4. StartUp #elasticsearch
   

   View Slide

5. StartUp #elasticsearch
   

   View Slide

6. StartUp #elasticsearch
   

   View Slide

7. StartUp #elasticsearch
   

   View Slide

8. StartUp #elasticsearch
   

   View Slide

9. #elasticsearch
   BIG data ?
   

   View Slide

10. #elasticsearch
    Source: http://www.csc.com/insights/flxwd/78931-big_data_just_beginning_to_explode
    35.000.000.000.000.000 mb
    

    View Slide

11. #elasticsearch
    Source: http://www.thebigdatainsightgroup.com/site/article/big-data-infographic
    

    View Slide

12. View Slide

13. StartUp #elasticsearch
    search = like % ?
    SELECT
    
    doc.*, country.*
    
    FROM
    
    doc, country
    
    WHERE
    
    doc.country_code = country.code AND
    
    doc.date_doc > to_date('2011-12', 'yyyy-mm') AND
    
    doc.date_doc < to_date('2012-01', 'yyyy-mm') AND
    
    lower(country.name) = 'france' AND
    
    lower(doc.comment) LIKE ‘%product%' AND
    lower(doc.comment) LIKE ‘%david%';
    

    View Slide

14. StartUp #elasticsearch
    Search engine ?
    

    View Slide

15. #elasticsearch
    elasticsearch ?
    plug & play
    REST/JSON
    scalable
    Apache 2 license
    Lucene
    elasticsearch
    

    View Slide

16. #elasticsearch
    Start…
    $ wget https://download.elasticsearch.org/elasticsearch/
    elasticsearch/elasticsearch-1.1.0.tar.gz!
    $ tar -xf elasticsearch-1.1.0.tar.gz!
    $ ./elasticsearch-1.1.0/bin/elasticsearch!
    [INFO ][node ][Ghost Maker] {1.1.0}[5645]: initializing
    

    View Slide

17. #elasticsearch
    … and play!
    $ curl -XPUT localhost:9200/sessions/session/1 -d '{!
    "title" : "Elasticsearch",!
    "subtitle" : "Make sense of your (BIG) data !",!
    "date" : "2014-04-03T10:30:00",!
    "tags" : [ "elasticsearch", "cloudconf", "bigdata" ],!
    "speaker" : [{!
    "first_name" : "David", !
    "last_name" : "Pilato" !
    }]!
    }'
    

    View Slide

18. #elasticsearch
    Search!
    $ curl http://localhost:9200/sessions/session/_search -d'
    {
    "query": {
    "multi_match": {
    "query": "elasticsearch cloudconf david",
    "fields": [ "title^3", "tags^2", "speaker.first_name" ]
    }
    },
    "post_filter": {
    "range": {
    "date": {
    "from": "2014-04",
    "to": "2014-05"
    }
    }
    }
    }'
    

    View Slide

19. StartUp #elasticsearch
    Compute?
    

    View Slide

20. #elasticsearch
    $ curl http://localhost:9200/sessions/session/_search -d'
    {
    "query": { ... },
    "aggs": {
    "by_date": {
    "date_histogram": {
    "field": "date",
    "interval": "day",
    "format" : "dd/MM/yyyy"
    }
    }
    }
    }'
    "by_date": [
    { "key_as_string": "03/04/2014", "doc_count": 1 },
    { "key_as_string": "12/04/2014", "doc_count": 2 },
    { "key_as_string": "16/04/2014", "doc_count": 3 }
    ]
    Compute!
    

    View Slide

21. #elasticsearch
    

    View Slide

22. #elasticsearch
    Let’s make sense of …
    • logs
    
    • twitter
    
    • github
    
    • marketing data
    
    • ...
    
    • your data
    
    • your big data
    

    View Slide

23. #elasticsearch
    {
    "name":"Pilato David",
    "dateOfBirth":"1971-12-26",
    "gender":"male",
    "children":3,
    "marketing":{
    "fashion":334,
    "music":3363,
    "hifi":2351
    },
    "address":{
    "country":"France",
    "city":"Paris",
    "location": [2.332395, 48.861871]
    }
    }
    Let’s make sense of …
    • logs
    
    • twitter
    
    • github
    
    • marketing data
    
    • ...
    
    • your data
    
    • your big data
    

    View Slide

24. démo
    #mstechdays #elasticsearch StartUp #elasticsearch
    MAKE SENSE OF YOUR (BIG) DATA!
    let’s inject some marketing documents…
    

    View Slide

25. #elasticsearch
    ELASTICSEARCH
    elastic? distributed?
    

    View Slide

26. StartUp #elasticsearch
    Distributed indices
    node 1
    orders
    products
    1 2
    3 4
    1 2
    $ curl -XPUT localhost:9200/orders -d '{!
    "settings.index.number_of_shards" : 4,!
    "settings.index.number_of_replicas" : 1!
    }'
    $ curl -XPUT localhost:9200/products -d '{!
    "settings.index.number_of_shards" : 2,!
    "settings.index.number_of_replicas" : 0!
    }'
    

    View Slide

27. StartUp #elasticsearch
    Distributed indices
    node 1
    orders
    products
    1 2
    3 4
    1 2
    node 2
    $ bin/elasticsearch!
    [INFO ][cluster.service][Armageddon] detected_master [Ghost Maker]
    

    View Slide

28. StartUp #elasticsearch
    Distributed indices
    node 1
    orders
    products
    1
    4
    1
    node 2
    orders
    products
    2
    3
    2
    2
    3
    1
    4
    2
    3
    2
    

    View Slide

29. StartUp #elasticsearch
    node 3
    Distributed indices
    node 1
    orders
    products
    1
    4
    1
    node 2
    orders
    products
    2
    3
    2
    2
    3
    1
    4
    $ bin/elasticsearch!
    [INFO ][cluster.service][Karnak] detected_master [Ghost Maker]
    

    View Slide

30. StartUp #elasticsearch
    node 3
    products
    orders
    Distributed indices
    node 1
    orders
    products
    1
    4
    1
    node 2
    orders
    products
    2
    3
    3
    2
    2
    3
    1
    4 3
    1
    4
    

    View Slide

31. elasticsearch.
    elasticsearch
    kibana
    logstash
    Marvel
    

    View Slide

32. elasticsearch.
    Training (public and on-site)
    Development support
    Production support
    Marvel
    

    View Slide

33. View Slide

34. @dadoonet
    questions ?
    we are hiring! jobs@elasticsearch.com
    

    View Slide