Slide 1
Slide 1 text
Workshop
Elasticsearch
Kibana
David Pilato
Developer | Evangelist
@dadoonet
Slide 2
Slide 2 text
No content
Slide 3
Slide 3 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
3
Agenda
• Elasticsearch overview
• Workshop 0: getting started
• Workshop 1: let’s index some documents
• Workshop 2: let’s search them
• Workshop 3: let’s pull some analytics
• Workshop 4: let’s add a powerful live UI on top
• Workshop 5: snapshot and restore
Slide 4
Slide 4 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
4
fundamentals
elasticsearch
Slide 5
Slide 5 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
5
a search engine
• Create indices from documents
• Search in indices
Slide 6
Slide 6 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
6
elasticsearch
• Cloud based search engine
• Based on Lucene
• Hide Lucene complexity by exposing all services
– HTTP / REST / JSON
• Works with all technologies
• Horizontal scaling, replication, fail over, load balancing
• Blazing fast!
• It’s a search engine! Not a search tool in a box!
Slide 7
Slide 7 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
7
think document!
• Change your mindset:
– Forget SQL!
– Index what you want to find
• A document
– A JSON object
– Core field types (string, numbers, booleans)
– Complex field types (arrays, objects)
– Additional field types (geo points, geo shapes)
Slide 8
Slide 8 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
8
organize your documents!
• Documents coordinates:
– index (hold setup)
– type (holds mapping)
– id (can be auto-generated)
{
"name" : "elasticsearch",
"website" : "http://www.elasticsearch.com",
"category" : "software",
"founded_year" : 2012,
"overview" : "The company behind the elasticsearch open source project",
"tags" : ["search", “datastore", "analytics", "realtime", "scalability"],
"location" : {
"city" : "Amsterdam",
"country_code" : "NL",
"geo" : {
"lat" : 52.370176,
"lon" : 4.895008
}
}
}
Slide 9
Slide 9 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
9
glossary
• Node
– a running elasticsearch instance (JVM process)
• Cluster
– a group of nodes
• Shard
– a part of an index
– a Lucene index under the hood
– primary: unique in the cluster
– replica: one or more copy of the primary
Slide 10
Slide 10 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
10
setup
workshop 0
Slide 11
Slide 11 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
11
setup
• get elasticsearch x.x.x
• edit config/elasticsearch.yml
• install marvel plugin
bin/plugin -install marvel -url file:../binaries/marvel-latest.zip
unzip binaries/elasticsearch-x.x.x.zip
cd elasticsearch-x.x.x
cluster.name: workshop
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["127.0.0.1"]
Slide 12
Slide 12 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
12
play with nodes
• start an elasticsearch node
• open marvel
bin/elasticsearch
open http://localhost:9200/_plugin/marvel/
Slide 13
Slide 13 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
13
we index persons
workshop 1
Slide 14
Slide 14 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
14
indexing a document
PUT /person/person/1
{
"name":"Anaelle Alessio",
"dateOfBirth":"2009-09-05",
"gender":"female",
"marketing":{
"shoes":1000,
"fashion":1200,
"music":800
},
"address":{
"country":"England",
"zipcode":"5226",
"city":"Plymouth",
"countrycode":"GB"
}
}
{
"_index": "person",
"_type": "person",
"_id": "1",
"_version": 1,
"created": true
}
Slide 15
Slide 15 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
15
getting a document
GET /person/person/1
{
"_index": "person",
"_type": "person",
"_id": "1",
"_version": 1,
"found": true,
"_source": {
"name": "Anaelle Alessio",
"dateOfBirth": "2009-09-05",
"gender": "female",
"marketing": {
"shoes": 1000,
"fashion": 1200,
"music": 800
},
"address": {
"country": "England",
"zipcode": "5226",
"city": "Plymouth",
"countrycode": "GB"
}
}
}
Slide 16
Slide 16 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
16
updating a document
PUT /person/person/1
{
"name":"Anaelle Alessio",
"dateOfBirth":"2009-09-05",
"gender":"female",
"marketing":{
"shoes":1001,
"fashion":1200,
"music":800
},
"address":{
"country":"England",
"zipcode":"5226",
"city":"Plymouth",
"countrycode":"GB"
}
}
{
"_index": "person",
"_type": "person",
"_id": "1",
"_version": 2,
"created": false
}
Slide 17
Slide 17 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
17
deleting a document
DELETE /person/person/1
{
"found": true,
"_index": "person",
"_type": "person",
"_id": "1",
"_version": 3
}
Slide 18
Slide 18 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
18
workshop 1: index some persons
PUT /person/person/1
{
"name":"Anaelle Alessio"
}
PUT /person/person/1
{
"name":"Anaelle Alessio",
"dateOfBirth":"2009-09-05"
}
PUT /person/person/2
{
"name":"Joe Smith"
}
PUT /person/person/2
{
"name":"Joe Smith",
"gender":"male"
}
Slide 19
Slide 19 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
19
workshop 1: 500 000 persons
• use injector script
• see effect in marvel
• start more nodes
java -jar injector-x.x.x.jar 500000 10000 workshop
open http://localhost:9200/_plugin/marvel/
bin/elasticsearch
bin/elasticsearch
bin/elasticsearch
...
Slide 20
Slide 20 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
20
we search for persons
workshop 2
Slide 21
Slide 21 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
21
searching persons in Germany
GET /person/person/_search
{
"query": {
"term": {
"address.country": {
"value": "Germany"
}
}
}
}
{
"took" : 3,
"hits" : {
"total" : 0,
"max_score" : null,
"hits" : [ ]
}
}
Slide 22
Slide 22 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
22
searching persons in germany
GET /person/person/_search
{
"query": {
"term": {
"address.country": {
"value": "germany"
}
}
}
}
{
"took" : 4,
"hits" : {
"total" : 30004,
"max_score" : 2.100946,
"hits" : [ {
"_index" : "person",
"_type" : "person",
"_id" : "SUy7Py3zSvqhjQroJPVFCw",
"_score" : 2.100946,
"_source" : {"name":"Fadi Norah", "address":{"country":"Germany"}}
}, { ... }
]
}
}
Slide 23
Slide 23 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
23
searching persons in Germany
GET /person/person/_search
{
"query": {
"match": {
"address.country": "Germany"
}
}
}
{
"took" : 4,
"hits" : {
"total" : 30004,
"max_score" : 2.100946,
"hits" : [ {
"_index" : "person",
"_type" : "person",
"_id" : "SUy7Py3zSvqhjQroJPVFCw",
"_score" : 2.100946,
"_source" : {"name":"Fadi Norah", "address":{"country":"Germany"}}
}, { ... }
]
}
}
Slide 24
Slide 24 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
24
searching for persons
GET /person/person/_search
{
"query": {
"bool": {
"must": [ {
"match": {
"address.country": "Germany"
}
}, {
"range": {
"dateOfBirth": {
"from": "1970",
"to": "1971"
}
}
}
]
}
}
}
Slide 25
Slide 25 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
25
workshop 2: reinject with mapping
• delete old data
• use injector script
• get mapping
java -jar injector-x.x.x.jar 1000000 10000 workshop
DELETE /person
GET /person/person/_mapping
Slide 26
Slide 26 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
26
workshop 2: search again
GET /person/person/_search
{
"query": {
"term": {
"address.country": "Germany"
}
}
}
{
"took" : 4,
"hits" : {
"total" : 30004,
"max_score" : 2.100946,
"hits" : [ {
"_index" : "person",
"_type" : "person",
"_id" : "SUy7Py3zSvqhjQroJPVFCw",
"_score" : 2.100946,
"_source" : {"name":"Fadi Norah", "address":{"country":"Germany"}}
}, { ... }
]
}
}
Slide 27
Slide 27 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
27
make sense of
your data: aggs!
workshop 3
Slide 28
Slide 28 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
28
break by country
GET /person/person/_search
{
"aggs": {
"by_country": {
"terms": {
"field": "address.country"
}
}
}
} { ..., "aggregations" : {
"by_country" : {
"buckets" : [ {
"key" : "England",
"doc_count" : 30051
}, {
"key" : "Germany",
"doc_count" : 30004
}, {
"key" : "France",
"doc_count" : 15034
}, {
"key" : "Spain",
"doc_count" : 14912
} ]}}}
17 %
17 %
33 %
33 %
England
Germany
France
Spain
Slide 29
Slide 29 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
29
date of birth histogram
GET /person/person/_search
{
"aggs": {
"by_date": {
"date_histogram": {
"field": "dateOfBirth",
"interval": "year",
"format": "yyyy"
}
}
}
}
{ ..., "aggregations": {
"by_date": {
"buckets": [
{
"key_as_string": "1960",
"key": -946080000000,
"doc_count": 39
},
{
"key_as_string": "1961",
"key": -630720000000,
"doc_count": 12677
},
{
"key_as_string": "1962",
"key": -315360000000,
"doc_count": 12936
}, ...
]
}
}}
0
7500
15000
22500
30000
1940 1960 1980 2000
Slide 30
Slide 30 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
30
searching for persons with aggs
GET /person/person/_search
{
"query": { "bool": { "must": [
{ "match": { "address.country": "Germany"} },
{ "range": { "dateOfBirth": { "from": "1970", "to": "1971" }}}
]}},
"aggs": {
"by_date": {
"date_histogram": {
"field": "dateOfBirth", "interval": "month", "format": "yyyy-MM"
},
"aggs": {
"by_gender": {
"terms": {
"field": "gender"
},
"aggs": {
"children": {
"stats": {
"field": "children"
}
}
}
}
}
}
}
}
Slide 31
Slide 31 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
31
click and play!
workshop 4
Slide 32
Slide 32 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
32
setup
• get kibana
• launch kibana
unzip binaries/kibana-x.x.x.zip
cd kibana-x.x.x
bin/kibana
open http://0.0.0.0:5601/
Slide 33
Slide 33 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
33
build
• get kibana
• launch kibana
Slide 34
Slide 34 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
34
snapshot and restore
workshop 5
Slide 35
Slide 35 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
35
backup
• create repository
• backup
• show all backups
PUT /_snapshot/main_backup
{
"type" : "fs",
"settings" : {
"location" : "/tmp/es-backup"
}
}
PUT /_snapshot/main_backup/snap1?wait_for_completion=true
GET /_snapshot/main_backup/_all
Slide 36
Slide 36 text
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
36
restore
• create repository (if needed)
• restore
PUT /_snapshot/main_backup
{
"type" : "fs",
"settings" : {
"location" : "/tmp/es-backup"
}
}
POST /_snapshot/main_backup/snap1/_restore?wait_for_completion=true
{
"indices":"+person"
}
POST /_snapshot/main_backup/snap1/_restore
{
"indices":"+person",
"rename_pattern": "person",
"rename_replacement": "new_person"
}
Slide 37
Slide 37 text
Thank you!
https://www.elastic.co/subscriptions
David Pilato
Developer | Evangelist
@dadoonet