Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Elasticsearch and Kibana 4 workshop

Elasticsearch and Kibana 4 workshop

Update for elasticsearch 1.4.0 and Kibana 4

Elasticsearch Inc

November 19, 2014
Tweet

More Decks by Elasticsearch Inc

Other Decks in Technology

Transcript

  1. Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited
    workshop
    elasticsearch

    View full-size slide

  2. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    elasticsearch - the company
    • Founded in 2012

    by the people behind elasticsearch project

    • Professional services

    Training (public and on site)

    Development support

    Production support subscription

    • Commercial product

    Marvel (included with support)

    View full-size slide

  3. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Agenda
    • Elasticsearch overview

    • Workshop 0: getting started

    • Workshop 1: let’s index some documents

    • Workshop 2: let’s search them

    • Workshop 3: let’s pull some analytics

    • Workshop 4: let’s add a powerful live UI on top

    • Workshop 5: snapshot and restore

    View full-size slide

  4. Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited
    fundamentals

    View full-size slide

  5. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    a search engine
    • Create indices from documents

    • Search in indices

    View full-size slide

  6. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    elasticsearch
    • Cloud based search engine

    • Based on Lucene

    • Hide Lucene complexity by exposing all services

    HTTP / REST / JSON

    • Works with all technologies

    • Horizontal scaling, replication, fail over, load balancing

    • Blazing fast!

    • It’s a search engine! Not a search tool in a box!

    View full-size slide

  7. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    think document!
    • Change your mindset:

    Forget SQL!

    Index what you want to find

    • A document

    A JSON object

    Core field types (string, numbers, booleans)

    Complex field types (arrays, objects)

    Additional field types (geo points, geo shapes)

    View full-size slide

  8. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    organize your documents!
    • Documents coordinates:

    index (hold setup)

    type (holds mapping)

    id (can be auto-generated)
    {
    "name" : "elasticsearch",
    "website" : "http://www.elasticsearch.com",
    "category" : "software",
    "founded_year" : 2012,
    "overview" : "The company behind the elasticsearch open source project",
    "tags" : ["search", “datastore", "analytics", "realtime", "scalability"],
    "location" : {
    "city" : "Amsterdam",
    "country_code" : "NL",
    "geo" : {
    "lat" : 52.370176,
    "lon" : 4.895008
    }
    }
    }

    View full-size slide

  9. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    glossary
    • Node

    a running elasticsearch instance (JVM process)

    • Cluster

    a group of nodes

    • Shard

    a part of an index

    a Lucene index under the hood

    primary: unique in the cluster

    replica: one or more copy of the primary

    View full-size slide

  10. Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited
    setup
    workshop 0

    View full-size slide

  11. Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited
    setup
    bin/plugin -install marvel -url file:../binaries/marvel-latest.zip
    • get elasticsearch x.x.x

    • edit config/elasticsearch.yml

    • install marvel plugin
    unzip binaries/elasticsearch-x.x.x.zip
    cd elasticsearch-x.x.x
    cluster.name: workshop
    discovery.zen.ping.multicast.enabled: false
    discovery.zen.ping.unicast.hosts: ["127.0.0.1"]

    View full-size slide

  12. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    play with nodes
    • start an elasticsearch node

    • open marvel
    bin/elasticsearch
    open http://localhost:9200/_plugin/marvel/

    View full-size slide

  13. Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited
    we index persons
    workshop 1

    View full-size slide

  14. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    indexing a document
    PUT /person/person/1
    {
    "name":"Anaelle Alessio",
    "dateOfBirth":"2009-09-05",
    "gender":"female",
    "marketing":{
    "shoes":1000,
    "fashion":1200,
    "music":800
    },
    "address":{
    "country":"England",
    "zipcode":"5226",
    "city":"Plymouth",
    "countrycode":"GB"
    }
    }
    {
    "_index": "person",
    "_type": "person",
    "_id": "1",
    "_version": 1,
    "created": true
    }

    View full-size slide

  15. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    getting a document
    GET /person/person/1
    {
    "_index": "person",
    "_type": "person",
    "_id": "1",
    "_version": 1,
    "found": true,
    "_source": {
    "name": "Anaelle Alessio",
    "dateOfBirth": "2009-09-05",
    "gender": "female",
    "marketing": {
    "shoes": 1000,
    "fashion": 1200,
    "music": 800
    },
    "address": {
    "country": "England",
    "zipcode": "5226",
    "city": "Plymouth",
    "countrycode": "GB"
    }
    }
    }

    View full-size slide

  16. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    updating a document
    PUT /person/person/1
    {
    "name":"Anaelle Alessio",
    "dateOfBirth":"2009-09-05",
    "gender":"female",
    "marketing":{
    "shoes":1001,
    "fashion":1200,
    "music":800
    },
    "address":{
    "country":"England",
    "zipcode":"5226",
    "city":"Plymouth",
    "countrycode":"GB"
    }
    }
    {
    "_index": "person",
    "_type": "person",
    "_id": "1",
    "_version": 2,
    "created": false
    }

    View full-size slide

  17. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    deleting a document
    DELETE /person/person/1
    {
    "found": true,
    "_index": "person",
    "_type": "person",
    "_id": "1",
    "_version": 3
    }

    View full-size slide

  18. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    workshop 1: index some persons
    PUT /person/person/1
    {
    "name":"Anaelle Alessio"
    }
    PUT /person/person/1
    {
    "name":"Anaelle Alessio",
    "dateOfBirth":"2009-09-05"
    }
    PUT /person/person/2
    {
    "name":"Joe Smith"
    }
    PUT /person/person/2
    {
    "name":"Joe Smith",

    "gender":"male"
    }

    View full-size slide

  19. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    workshop 1: 500 000 persons
    • use injector script

    • see effect in marvel

    • start more nodes
    java -jar injector-x.x.x.jar 500000 10000 workshop
    open http://localhost:9200/_plugin/marvel/
    bin/elasticsearch
    bin/elasticsearch
    bin/elasticsearch
    ...

    View full-size slide

  20. Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited
    we search

    for persons
    workshop 2

    View full-size slide

  21. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    searching persons in Germany
    GET /person/person/_search
    {
    "query": {
    "term": {
    "address.country": {
    "value": "Germany"
    }
    }
    }
    }
    {
    "took" : 3,
    "hits" : {
    "total" : 0,
    "max_score" : null,
    "hits" : [ ]
    }
    }

    View full-size slide

  22. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    searching persons in germany
    GET /person/person/_search
    {
    "query": {
    "term": {
    "address.country": {
    "value": "germany"
    }
    }
    }
    }
    {
    "took" : 4,
    "hits" : {
    "total" : 30004,
    "max_score" : 2.100946,
    "hits" : [ {
    "_index" : "person",
    "_type" : "person",
    "_id" : "SUy7Py3zSvqhjQroJPVFCw",
    "_score" : 2.100946,
    "_source" : {"name":"Fadi Norah", "address":{"country":"Germany"}}
    }, { ... }
    ]
    }
    }

    View full-size slide

  23. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    searching persons in Germany
    GET /person/person/_search
    {
    "query": {
    "match": {
    "address.country": "Germany"
    }
    }
    }
    {
    "took" : 4,
    "hits" : {
    "total" : 30004,
    "max_score" : 2.100946,
    "hits" : [ {
    "_index" : "person",
    "_type" : "person",
    "_id" : "SUy7Py3zSvqhjQroJPVFCw",
    "_score" : 2.100946,
    "_source" : {"name":"Fadi Norah", "address":{"country":"Germany"}}
    }, { ... }
    ]
    }
    }

    View full-size slide

  24. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    searching for persons
    GET /person/person/_search
    {
    "query": {
    "bool": {
    "must": [ {
    "match": {
    "address.country": "Germany"
    }
    }, {
    "range": {
    "dateOfBirth": {
    "from": "1970",
    "to": "1971"
    }
    }
    }
    ]
    }
    }
    }

    View full-size slide

  25. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    workshop 2: reinject with mapping
    • delete old data

    • use injector script

    • get mapping
    java -jar injector-x.x.x.jar 1000000 10000 workshop
    DELETE /person
    GET /person/person/_mapping

    View full-size slide

  26. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    workshop 2: search again
    GET /person/person/_search
    {
    "query": {
    "term": {
    "address.country": "Germany"
    }
    }
    }
    {
    "took" : 4,
    "hits" : {
    "total" : 30004,
    "max_score" : 2.100946,
    "hits" : [ {
    "_index" : "person",
    "_type" : "person",
    "_id" : "SUy7Py3zSvqhjQroJPVFCw",
    "_score" : 2.100946,
    "_source" : {"name":"Fadi Norah", "address":{"country":"Germany"}}
    }, { ... }
    ]
    }
    }

    View full-size slide

  27. Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited
    make sense of
    your data: aggs!
    workshop 3

    View full-size slide

  28. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    break by country
    GET /person/person/_search

    {
    "aggs": {
    "by_country": {
    "terms": {
    "field": "address.country"
    }
    }
    }
    } { ..., "aggregations" : {
    "by_country" : {
    "buckets" : [ {
    "key" : "England",
    "doc_count" : 30051
    }, {
    "key" : "Germany",
    "doc_count" : 30004
    }, {
    "key" : "France",
    "doc_count" : 15034
    }, {
    "key" : "Spain",
    "doc_count" : 14912
    } ]}}}
    17 %
    17 %
    33 %
    33 %
    England
    Germany
    France
    Spain

    View full-size slide

  29. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    date of birth histogram
    GET /person/person/_search

    {
    "aggs": {
    "by_date": {
    "date_histogram": {
    "field": "dateOfBirth",
    "interval": "year",
    "format": "yyyy"
    }
    }
    }
    }
    { ..., "aggregations": {
    "by_date": {
    "buckets": [
    {
    "key_as_string": "1960",
    "key": -946080000000,
    "doc_count": 39
    },
    {
    "key_as_string": "1961",
    "key": -630720000000,
    "doc_count": 12677
    },
    {
    "key_as_string": "1962",
    "key": -315360000000,
    "doc_count": 12936
    }, ...
    ]
    }
    }}
    0
    7500
    15000
    22500
    30000
    1940 1960 1980 2000

    View full-size slide

  30. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    searching for persons with aggs
    GET /person/person/_search
    {
    "query": { "bool": { "must": [
    { "match": { "address.country": "Germany"} },
    { "range": { "dateOfBirth": { "from": "1970", "to": "1971" }}}
    ]}},
    "aggs": {
    "by_date": {
    "date_histogram": {
    "field": "dateOfBirth", "interval": "month", "format": "yyyy-MM"
    },
    "aggs": {
    "by_gender": {
    "terms": {
    "field": "gender"
    },
    "aggs": {
    "children": {
    "stats": {
    "field": "children"
    }
    }
    }
    }
    }
    }
    }
    }

    View full-size slide

  31. Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited
    click & play!
    workshop 4

    View full-size slide

  32. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    setup
    • get kibana

    • launch kibana
    unzip binaries/kibana-x.x.x.zip
    cd kibana-x.x.x
    bin/kibana
    open http://0.0.0.0:5601/

    View full-size slide

  33. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    build

    View full-size slide

  34. Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited
    snapshot and
    restore
    workshop 5

    View full-size slide

  35. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    • create repository

    • backup

    • show all backups
    backup
    PUT /_snapshot/main_backup
    {
    "type" : "fs",
    "settings" : {
    "location" : "/tmp/es-backup"
    }
    }
    PUT /_snapshot/main_backup/snap1?wait_for_completion=true
    GET /_snapshot/main_backup/_all

    View full-size slide

  36. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    • create repository (if needed)

    • restore
    restore
    PUT /_snapshot/main_backup
    {
    "type" : "fs",
    "settings" : {
    "location" : "/tmp/es-backup"
    }
    }
    POST /_snapshot/main_backup/snap1/_restore?wait_for_completion=true
    {
    "indices":"+person"
    }
    POST /_snapshot/main_backup/snap1/_restore
    {
    "indices":"+person",
    "rename_pattern": "person",
    "rename_replacement": "new_person"
    }

    View full-size slide

  37. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    thank you!
    http://elasticsearch.com/support
    @elasticsearch

    View full-size slide