Slide 1

Slide 1 text

Workshop Elasticsearch Kibana David Pilato Developer | Evangelist @dadoonet

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 3 Agenda • Elasticsearch overview • Workshop 0: getting started • Workshop 1: let’s index some documents • Workshop 2: let’s search them • Workshop 3: let’s pull some analytics • Workshop 4: let’s add a powerful live UI on top • Workshop 5: snapshot and restore

Slide 4

Slide 4 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 4 fundamentals elasticsearch

Slide 5

Slide 5 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 5 a search engine • Create indices from documents • Search in indices

Slide 6

Slide 6 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 6 elasticsearch • Cloud based search engine • Based on Lucene • Hide Lucene complexity by exposing all services – HTTP / REST / JSON • Works with all technologies • Horizontal scaling, replication, fail over, load balancing • Blazing fast! • It’s a search engine! Not a search tool in a box!

Slide 7

Slide 7 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 7 think document! • Change your mindset: – Forget SQL! – Index what you want to find • A document – A JSON object – Core field types (string, numbers, booleans) – Complex field types (arrays, objects) – Additional field types (geo points, geo shapes)

Slide 8

Slide 8 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 8 organize your documents! • Documents coordinates: – index (hold setup) – type (holds mapping) – id (can be auto-generated) { "name" : "elasticsearch", "website" : "http://www.elasticsearch.com", "category" : "software", "founded_year" : 2012, "overview" : "The company behind the elasticsearch open source project", "tags" : ["search", “datastore", "analytics", "realtime", "scalability"], "location" : { "city" : "Amsterdam", "country_code" : "NL", "geo" : { "lat" : 52.370176, "lon" : 4.895008 } } }

Slide 9

Slide 9 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 9 glossary • Node – a running elasticsearch instance (JVM process) • Cluster – a group of nodes • Shard – a part of an index – a Lucene index under the hood – primary: unique in the cluster – replica: one or more copy of the primary

Slide 10

Slide 10 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 10 setup workshop 0

Slide 11

Slide 11 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 11 setup • get elasticsearch x.x.x • edit config/elasticsearch.yml • install marvel plugin bin/plugin -install marvel -url file:../binaries/marvel-latest.zip unzip binaries/elasticsearch-x.x.x.zip cd elasticsearch-x.x.x cluster.name: workshop discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["127.0.0.1"]

Slide 12

Slide 12 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 12 play with nodes • start an elasticsearch node • open marvel bin/elasticsearch open http://localhost:9200/_plugin/marvel/

Slide 13

Slide 13 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 13 we index persons workshop 1

Slide 14

Slide 14 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 14 indexing a document PUT /person/person/1 { "name":"Anaelle Alessio", "dateOfBirth":"2009-09-05", "gender":"female", "marketing":{ "shoes":1000, "fashion":1200, "music":800 }, "address":{ "country":"England", "zipcode":"5226", "city":"Plymouth", "countrycode":"GB" } } { "_index": "person", "_type": "person", "_id": "1", "_version": 1, "created": true }

Slide 15

Slide 15 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 15 getting a document GET /person/person/1 { "_index": "person", "_type": "person", "_id": "1", "_version": 1, "found": true, "_source": { "name": "Anaelle Alessio", "dateOfBirth": "2009-09-05", "gender": "female", "marketing": { "shoes": 1000, "fashion": 1200, "music": 800 }, "address": { "country": "England", "zipcode": "5226", "city": "Plymouth", "countrycode": "GB" } } }

Slide 16

Slide 16 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 16 updating a document PUT /person/person/1 { "name":"Anaelle Alessio", "dateOfBirth":"2009-09-05", "gender":"female", "marketing":{ "shoes":1001, "fashion":1200, "music":800 }, "address":{ "country":"England", "zipcode":"5226", "city":"Plymouth", "countrycode":"GB" } } { "_index": "person", "_type": "person", "_id": "1", "_version": 2, "created": false }

Slide 17

Slide 17 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 17 deleting a document DELETE /person/person/1 { "found": true, "_index": "person", "_type": "person", "_id": "1", "_version": 3 }

Slide 18

Slide 18 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 18 workshop 1: index some persons PUT /person/person/1 { "name":"Anaelle Alessio" } PUT /person/person/1 { "name":"Anaelle Alessio", "dateOfBirth":"2009-09-05" } PUT /person/person/2 { "name":"Joe Smith" } PUT /person/person/2 { "name":"Joe Smith",
 "gender":"male" }

Slide 19

Slide 19 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 19 workshop 1: 500 000 persons • use injector script • see effect in marvel • start more nodes java -jar injector-x.x.x.jar 500000 10000 workshop open http://localhost:9200/_plugin/marvel/ bin/elasticsearch bin/elasticsearch bin/elasticsearch ...

Slide 20

Slide 20 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 20 we search for persons workshop 2

Slide 21

Slide 21 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 21 searching persons in Germany GET /person/person/_search { "query": { "term": { "address.country": { "value": "Germany" } } } } { "took" : 3, "hits" : { "total" : 0, "max_score" : null, "hits" : [ ] } }

Slide 22

Slide 22 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 22 searching persons in germany GET /person/person/_search { "query": { "term": { "address.country": { "value": "germany" } } } } { "took" : 4, "hits" : { "total" : 30004, "max_score" : 2.100946, "hits" : [ { "_index" : "person", "_type" : "person", "_id" : "SUy7Py3zSvqhjQroJPVFCw", "_score" : 2.100946, "_source" : {"name":"Fadi Norah", "address":{"country":"Germany"}} }, { ... } ] } }

Slide 23

Slide 23 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 23 searching persons in Germany GET /person/person/_search { "query": { "match": { "address.country": "Germany" } } } { "took" : 4, "hits" : { "total" : 30004, "max_score" : 2.100946, "hits" : [ { "_index" : "person", "_type" : "person", "_id" : "SUy7Py3zSvqhjQroJPVFCw", "_score" : 2.100946, "_source" : {"name":"Fadi Norah", "address":{"country":"Germany"}} }, { ... } ] } }

Slide 24

Slide 24 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 24 searching for persons GET /person/person/_search { "query": { "bool": { "must": [ { "match": { "address.country": "Germany" } }, { "range": { "dateOfBirth": { "from": "1970", "to": "1971" } } } ] } } }

Slide 25

Slide 25 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 25 workshop 2: reinject with mapping • delete old data • use injector script • get mapping java -jar injector-x.x.x.jar 1000000 10000 workshop DELETE /person GET /person/person/_mapping

Slide 26

Slide 26 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 26 workshop 2: search again GET /person/person/_search { "query": { "term": { "address.country": "Germany" } } } { "took" : 4, "hits" : { "total" : 30004, "max_score" : 2.100946, "hits" : [ { "_index" : "person", "_type" : "person", "_id" : "SUy7Py3zSvqhjQroJPVFCw", "_score" : 2.100946, "_source" : {"name":"Fadi Norah", "address":{"country":"Germany"}} }, { ... } ] } }

Slide 27

Slide 27 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 27 make sense of 
 your data: aggs! workshop 3

Slide 28

Slide 28 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 28 break by country GET /person/person/_search
 { "aggs": { "by_country": { "terms": { "field": "address.country" } } } } { ..., "aggregations" : { "by_country" : { "buckets" : [ { "key" : "England", "doc_count" : 30051 }, { "key" : "Germany", "doc_count" : 30004 }, { "key" : "France", "doc_count" : 15034 }, { "key" : "Spain", "doc_count" : 14912 } ]}}} 17 % 17 % 33 % 33 % England Germany France Spain

Slide 29

Slide 29 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 29 date of birth histogram GET /person/person/_search
 { "aggs": { "by_date": { "date_histogram": { "field": "dateOfBirth", "interval": "year", "format": "yyyy" } } } } { ..., "aggregations": { "by_date": { "buckets": [ { "key_as_string": "1960", "key": -946080000000, "doc_count": 39 }, { "key_as_string": "1961", "key": -630720000000, "doc_count": 12677 }, { "key_as_string": "1962", "key": -315360000000, "doc_count": 12936 }, ... ] } }} 0 7500 15000 22500 30000 1940 1960 1980 2000

Slide 30

Slide 30 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 30 searching for persons with aggs GET /person/person/_search { "query": { "bool": { "must": [ { "match": { "address.country": "Germany"} }, { "range": { "dateOfBirth": { "from": "1970", "to": "1971" }}} ]}}, "aggs": { "by_date": { "date_histogram": { "field": "dateOfBirth", "interval": "month", "format": "yyyy-MM" }, "aggs": { "by_gender": { "terms": { "field": "gender" }, "aggs": { "children": { "stats": { "field": "children" } } } } } } } }

Slide 31

Slide 31 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 31 click and play! workshop 4

Slide 32

Slide 32 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 32 setup • get kibana • launch kibana unzip binaries/kibana-x.x.x.zip cd kibana-x.x.x bin/kibana open http://0.0.0.0:5601/

Slide 33

Slide 33 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 33 build • get kibana • launch kibana

Slide 34

Slide 34 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 34 snapshot and restore workshop 5

Slide 35

Slide 35 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 35 backup • create repository • backup • show all backups PUT /_snapshot/main_backup { "type" : "fs", "settings" : { "location" : "/tmp/es-backup" } } PUT /_snapshot/main_backup/snap1?wait_for_completion=true GET /_snapshot/main_backup/_all

Slide 36

Slide 36 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 36 restore • create repository (if needed) • restore PUT /_snapshot/main_backup { "type" : "fs", "settings" : { "location" : "/tmp/es-backup" } } POST /_snapshot/main_backup/snap1/_restore?wait_for_completion=true { "indices":"+person" } POST /_snapshot/main_backup/snap1/_restore { "indices":"+person", "rename_pattern": "person", "rename_replacement": "new_person" }

Slide 37

Slide 37 text

Thank you!
 https://www.elastic.co/subscriptions David Pilato Developer | Evangelist @dadoonet