Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Web2Day Nantes 2015 - workshop

Web2Day Nantes 2015 - workshop

Elasticsearch and Kibana workshop
Talk given in Nantes, France: http://web2day.co/en/speakers/david-pilato-2/
Let’s start to discover elasticsearch and Kibana.

For this session, you will need:

* a JVM
* a browser

We will install elasticsearch, Kibana and Marvel and will use that tools to:

* index/update/get/delete documents
* search
* compute
* build dashboards to make sense of marketing data
* snapshot and restore our data

Avatar for Elastic Co

Elastic Co

June 05, 2015
Tweet

More Decks by Elastic Co

Other Decks in Technology

Transcript

  1. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 3 Agenda • Elasticsearch overview • Workshop 0: getting started • Workshop 1: let’s index some documents • Workshop 2: let’s search them • Workshop 3: let’s pull some analytics • Workshop 4: let’s add a powerful live UI on top • Workshop 5: snapshot and restore
  2. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 4 fundamentals elasticsearch
  3. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 5 a search engine • Create indices from documents • Search in indices
  4. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 6 elasticsearch • Cloud based search engine • Based on Lucene • Hide Lucene complexity by exposing all services – HTTP / REST / JSON • Works with all technologies • Horizontal scaling, replication, fail over, load balancing • Blazing fast! • It’s a search engine! Not a search tool in a box!
  5. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 7 think document! • Change your mindset: – Forget SQL! – Index what you want to find • A document – A JSON object – Core field types (string, numbers, booleans) – Complex field types (arrays, objects) – Additional field types (geo points, geo shapes)
  6. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 8 organize your documents! • Documents coordinates: – index (hold setup) – type (holds mapping) – id (can be auto-generated) { "name" : "elasticsearch", "website" : "http://www.elasticsearch.com", "category" : "software", "founded_year" : 2012, "overview" : "The company behind the elasticsearch open source project", "tags" : ["search", “datastore", "analytics", "realtime", "scalability"], "location" : { "city" : "Amsterdam", "country_code" : "NL", "geo" : { "lat" : 52.370176, "lon" : 4.895008 } } }
  7. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 9 glossary • Node – a running elasticsearch instance (JVM process) • Cluster – a group of nodes • Shard – a part of an index – a Lucene index under the hood – primary: unique in the cluster – replica: one or more copy of the primary
  8. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 11 setup • get elasticsearch x.x.x • edit config/elasticsearch.yml • install marvel plugin bin/plugin -install marvel -url file:../binaries/marvel-latest.zip unzip binaries/elasticsearch-x.x.x.zip cd elasticsearch-x.x.x cluster.name: workshop discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["127.0.0.1"]
  9. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 12 play with nodes • start an elasticsearch node • open marvel bin/elasticsearch open http://localhost:9200/_plugin/marvel/
  10. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 13 we index persons workshop 1
  11. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 14 indexing a document PUT /person/person/1 { "name":"Anaelle Alessio", "dateOfBirth":"2009-09-05", "gender":"female", "marketing":{ "shoes":1000, "fashion":1200, "music":800 }, "address":{ "country":"England", "zipcode":"5226", "city":"Plymouth", "countrycode":"GB" } } { "_index": "person", "_type": "person", "_id": "1", "_version": 1, "created": true }
  12. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 15 getting a document GET /person/person/1 { "_index": "person", "_type": "person", "_id": "1", "_version": 1, "found": true, "_source": { "name": "Anaelle Alessio", "dateOfBirth": "2009-09-05", "gender": "female", "marketing": { "shoes": 1000, "fashion": 1200, "music": 800 }, "address": { "country": "England", "zipcode": "5226", "city": "Plymouth", "countrycode": "GB" } } }
  13. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 16 updating a document PUT /person/person/1 { "name":"Anaelle Alessio", "dateOfBirth":"2009-09-05", "gender":"female", "marketing":{ "shoes":1001, "fashion":1200, "music":800 }, "address":{ "country":"England", "zipcode":"5226", "city":"Plymouth", "countrycode":"GB" } } { "_index": "person", "_type": "person", "_id": "1", "_version": 2, "created": false }
  14. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 17 deleting a document DELETE /person/person/1 { "found": true, "_index": "person", "_type": "person", "_id": "1", "_version": 3 }
  15. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 18 workshop 1: index some persons PUT /person/person/1 { "name":"Anaelle Alessio" } PUT /person/person/1 { "name":"Anaelle Alessio", "dateOfBirth":"2009-09-05" } PUT /person/person/2 { "name":"Joe Smith" } PUT /person/person/2 { "name":"Joe Smith",
 "gender":"male" }
  16. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 19 workshop 1: 500 000 persons • use injector script • see effect in marvel • start more nodes java -jar injector-x.x.x.jar 500000 10000 workshop open http://localhost:9200/_plugin/marvel/ bin/elasticsearch bin/elasticsearch bin/elasticsearch ...
  17. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 20 we search for persons workshop 2
  18. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 21 searching persons in Germany GET /person/person/_search { "query": { "term": { "address.country": { "value": "Germany" } } } } { "took" : 3, "hits" : { "total" : 0, "max_score" : null, "hits" : [ ] } }
  19. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 22 searching persons in germany GET /person/person/_search { "query": { "term": { "address.country": { "value": "germany" } } } } { "took" : 4, "hits" : { "total" : 30004, "max_score" : 2.100946, "hits" : [ { "_index" : "person", "_type" : "person", "_id" : "SUy7Py3zSvqhjQroJPVFCw", "_score" : 2.100946, "_source" : {"name":"Fadi Norah", "address":{"country":"Germany"}} }, { ... } ] } }
  20. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 23 searching persons in Germany GET /person/person/_search { "query": { "match": { "address.country": "Germany" } } } { "took" : 4, "hits" : { "total" : 30004, "max_score" : 2.100946, "hits" : [ { "_index" : "person", "_type" : "person", "_id" : "SUy7Py3zSvqhjQroJPVFCw", "_score" : 2.100946, "_source" : {"name":"Fadi Norah", "address":{"country":"Germany"}} }, { ... } ] } }
  21. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 24 searching for persons GET /person/person/_search { "query": { "bool": { "must": [ { "match": { "address.country": "Germany" } }, { "range": { "dateOfBirth": { "from": "1970", "to": "1971" } } } ] } } }
  22. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 25 workshop 2: reinject with mapping • delete old data • use injector script • get mapping java -jar injector-x.x.x.jar 1000000 10000 workshop DELETE /person GET /person/person/_mapping
  23. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 26 workshop 2: search again GET /person/person/_search { "query": { "term": { "address.country": "Germany" } } } { "took" : 4, "hits" : { "total" : 30004, "max_score" : 2.100946, "hits" : [ { "_index" : "person", "_type" : "person", "_id" : "SUy7Py3zSvqhjQroJPVFCw", "_score" : 2.100946, "_source" : {"name":"Fadi Norah", "address":{"country":"Germany"}} }, { ... } ] } }
  24. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 27 make sense of 
 your data: aggs! workshop 3
  25. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 28 break by country GET /person/person/_search
 { "aggs": { "by_country": { "terms": { "field": "address.country" } } } } { ..., "aggregations" : { "by_country" : { "buckets" : [ { "key" : "England", "doc_count" : 30051 }, { "key" : "Germany", "doc_count" : 30004 }, { "key" : "France", "doc_count" : 15034 }, { "key" : "Spain", "doc_count" : 14912 } ]}}} 17 % 17 % 33 % 33 % England Germany France Spain
  26. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 29 date of birth histogram GET /person/person/_search
 { "aggs": { "by_date": { "date_histogram": { "field": "dateOfBirth", "interval": "year", "format": "yyyy" } } } } { ..., "aggregations": { "by_date": { "buckets": [ { "key_as_string": "1960", "key": -946080000000, "doc_count": 39 }, { "key_as_string": "1961", "key": -630720000000, "doc_count": 12677 }, { "key_as_string": "1962", "key": -315360000000, "doc_count": 12936 }, ... ] } }} 0 7500 15000 22500 30000 1940 1960 1980 2000
  27. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 30 searching for persons with aggs GET /person/person/_search { "query": { "bool": { "must": [ { "match": { "address.country": "Germany"} }, { "range": { "dateOfBirth": { "from": "1970", "to": "1971" }}} ]}}, "aggs": { "by_date": { "date_histogram": { "field": "dateOfBirth", "interval": "month", "format": "yyyy-MM" }, "aggs": { "by_gender": { "terms": { "field": "gender" }, "aggs": { "children": { "stats": { "field": "children" } } } } } } } }
  28. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 31 click and play! workshop 4
  29. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 32 setup • get kibana • launch kibana unzip binaries/kibana-x.x.x.zip cd kibana-x.x.x bin/kibana open http://0.0.0.0:5601/
  30. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 33 build • get kibana • launch kibana
  31. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 34 snapshot and restore workshop 5
  32. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 35 backup • create repository • backup • show all backups PUT /_snapshot/main_backup { "type" : "fs", "settings" : { "location" : "/tmp/es-backup" } } PUT /_snapshot/main_backup/snap1?wait_for_completion=true GET /_snapshot/main_backup/_all
  33. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 36 restore • create repository (if needed) • restore PUT /_snapshot/main_backup { "type" : "fs", "settings" : { "location" : "/tmp/es-backup" } } POST /_snapshot/main_backup/snap1/_restore?wait_for_completion=true { "indices":"+person" } POST /_snapshot/main_backup/snap1/_restore { "indices":"+person", "rename_pattern": "person", "rename_replacement": "new_person" }