$30 off During Our Annual Pro Sale. View Details »

Make sense of your (BIG) data!

Make sense of your (BIG) data!

Presented by David Pilato at the CloudConf.IT 2014 conference.

Elasticsearch is a search server based on Lucene. It provides a distributed, multitenant-capable full-text search engine with a RESTful web interface and schema-free JSON documents. You can use it as well to compute information on live data. Kibana is a full web application built with AngularJS. It's a generic and powerful visualisation tool for your data. Both projects are under Apache 2 License. In this presentation, you will discover how Elasticsearch actually works. We will inject marketing data into Elasticsearch and build live a dashboard using Kibana. In minutes, you will know how to build YOUR own dashboard and make sense of YOUR data.

Elasticsearch Inc

April 03, 2014
Tweet

More Decks by Elasticsearch Inc

Other Decks in Technology

Transcript

  1. #elasticsearch
    MAKE SENSE OF YOUR (BIG) DATA!
    David Pilato
    Technical advocate

    !
    elasticsearch.
    @dadoonet

    View Slide

  2. View Slide

  3. StartUp #elasticsearch
    data ?

    View Slide

  4. StartUp #elasticsearch

    View Slide

  5. StartUp #elasticsearch

    View Slide

  6. StartUp #elasticsearch

    View Slide

  7. StartUp #elasticsearch

    View Slide

  8. StartUp #elasticsearch

    View Slide

  9. #elasticsearch
    BIG data ?

    View Slide

  10. #elasticsearch
    Source: http://www.csc.com/insights/flxwd/78931-big_data_just_beginning_to_explode
    35.000.000.000.000.000 mb

    View Slide

  11. #elasticsearch
    Source: http://www.thebigdatainsightgroup.com/site/article/big-data-infographic

    View Slide

  12. View Slide

  13. StartUp #elasticsearch
    search = like % ?
    SELECT

    doc.*, country.*

    FROM

    doc, country

    WHERE

    doc.country_code = country.code AND

    doc.date_doc > to_date('2011-12', 'yyyy-mm') AND

    doc.date_doc < to_date('2012-01', 'yyyy-mm') AND

    lower(country.name) = 'france' AND

    lower(doc.comment) LIKE ‘%product%' AND
    lower(doc.comment) LIKE ‘%david%';

    View Slide

  14. StartUp #elasticsearch
    Search engine ?

    View Slide

  15. #elasticsearch
    elasticsearch ?
    plug & play
    REST/JSON
    scalable
    Apache 2 license
    Lucene
    elasticsearch

    View Slide

  16. #elasticsearch
    Start…
    $ wget https://download.elasticsearch.org/elasticsearch/
    elasticsearch/elasticsearch-1.1.0.tar.gz!
    $ tar -xf elasticsearch-1.1.0.tar.gz!
    $ ./elasticsearch-1.1.0/bin/elasticsearch!
    [INFO ][node ][Ghost Maker] {1.1.0}[5645]: initializing

    View Slide

  17. #elasticsearch
    … and play!
    $ curl -XPUT localhost:9200/sessions/session/1 -d '{!
    "title" : "Elasticsearch",!
    "subtitle" : "Make sense of your (BIG) data !",!
    "date" : "2014-04-03T10:30:00",!
    "tags" : [ "elasticsearch", "cloudconf", "bigdata" ],!
    "speaker" : [{!
    "first_name" : "David", !
    "last_name" : "Pilato" !
    }]!
    }'

    View Slide

  18. #elasticsearch
    Search!
    $ curl http://localhost:9200/sessions/session/_search -d'
    {
    "query": {
    "multi_match": {
    "query": "elasticsearch cloudconf david",
    "fields": [ "title^3", "tags^2", "speaker.first_name" ]
    }
    },
    "post_filter": {
    "range": {
    "date": {
    "from": "2014-04",
    "to": "2014-05"
    }
    }
    }
    }'

    View Slide

  19. StartUp #elasticsearch
    Compute?

    View Slide

  20. #elasticsearch
    $ curl http://localhost:9200/sessions/session/_search -d'
    {
    "query": { ... },
    "aggs": {
    "by_date": {
    "date_histogram": {
    "field": "date",
    "interval": "day",
    "format" : "dd/MM/yyyy"
    }
    }
    }
    }'
    "by_date": [
    { "key_as_string": "03/04/2014", "doc_count": 1 },
    { "key_as_string": "12/04/2014", "doc_count": 2 },
    { "key_as_string": "16/04/2014", "doc_count": 3 }
    ]
    Compute!

    View Slide

  21. #elasticsearch

    View Slide

  22. #elasticsearch
    Let’s make sense of …
    • logs

    • twitter

    • github

    • marketing data

    • ...

    • your data

    • your big data

    View Slide

  23. #elasticsearch
    {
    "name":"Pilato David",
    "dateOfBirth":"1971-12-26",
    "gender":"male",
    "children":3,
    "marketing":{
    "fashion":334,
    "music":3363,
    "hifi":2351
    },
    "address":{
    "country":"France",
    "city":"Paris",
    "location": [2.332395, 48.861871]
    }
    }
    Let’s make sense of …
    • logs

    • twitter

    • github

    • marketing data

    • ...

    • your data

    • your big data

    View Slide

  24. démo
    #mstechdays #elasticsearch StartUp #elasticsearch
    MAKE SENSE OF YOUR (BIG) DATA!
    let’s inject some marketing documents…

    View Slide

  25. #elasticsearch
    ELASTICSEARCH
    elastic? distributed?

    View Slide

  26. StartUp #elasticsearch
    Distributed indices
    node 1
    orders
    products
    1 2
    3 4
    1 2
    $ curl -XPUT localhost:9200/orders -d '{!
    "settings.index.number_of_shards" : 4,!
    "settings.index.number_of_replicas" : 1!
    }'
    $ curl -XPUT localhost:9200/products -d '{!
    "settings.index.number_of_shards" : 2,!
    "settings.index.number_of_replicas" : 0!
    }'

    View Slide

  27. StartUp #elasticsearch
    Distributed indices
    node 1
    orders
    products
    1 2
    3 4
    1 2
    node 2
    $ bin/elasticsearch!
    [INFO ][cluster.service][Armageddon] detected_master [Ghost Maker]

    View Slide

  28. StartUp #elasticsearch
    Distributed indices
    node 1
    orders
    products
    1
    4
    1
    node 2
    orders
    products
    2
    3
    2
    2
    3
    1
    4
    2
    3
    2

    View Slide

  29. StartUp #elasticsearch
    node 3
    Distributed indices
    node 1
    orders
    products
    1
    4
    1
    node 2
    orders
    products
    2
    3
    2
    2
    3
    1
    4
    $ bin/elasticsearch!
    [INFO ][cluster.service][Karnak] detected_master [Ghost Maker]

    View Slide

  30. StartUp #elasticsearch
    node 3
    products
    orders
    Distributed indices
    node 1
    orders
    products
    1
    4
    1
    node 2
    orders
    products
    2
    3
    3
    2
    2
    3
    1
    4 3
    1
    4

    View Slide

  31. elasticsearch.
    elasticsearch
    kibana
    logstash
    Marvel

    View Slide

  32. elasticsearch.
    Training (public and on-site)
    Development support
    Production support
    Marvel

    View Slide

  33. View Slide

  34. @dadoonet
    questions ?
    we are hiring! [email protected]

    View Slide