Life of a Document in Elasticsearch

by Boaz Leskes

Slide 1

Slide 1 text

Life of a document in Elasticsearch Boaz Leskes - @bleskes

Slide 2

Slide 2 text

{ } CC-BY-ND 4.0 Agenda 2 Life of a document in Elasticsearch

Slide 3

Slide 3 text

{ } CC-BY-ND 4.0 Agenda 3 Life of a document in Elasticsearch How, when and where is a document stored/processed in Elasticsearch?

Slide 4

Slide 4 text

{ } CC-BY-ND 4.0 Wait… Elasticsearch? • Distributed, highly available, data storage & retrieval • Search engine / aggregation engine / analytics • Suggestions / percolation / highlighting / geo • Document store • Horizontally distributed • High availability • Real time & near real time • Open Source • Apache License • Proprietary plugins available for security & messaging 4

Slide 5

Slide 5 text

{ } CC-BY-ND 4.0 Exhibit A: A JSON document { "name" : "Elasticsearch - The definitive guide", "pages" : 721, "isbn13" : "978-1449358549", "authors" : [ "Clinton Gormley" , "Zachary Tong" ], "publish_date" : "2015/01/31", "category" : "IT > Search Engines", "description" : "Whether you need full-text search or real-time analytics...", } 5

Slide 6

Slide 6 text

{ } CC-BY-ND 4.0 Exhibit A: A JSON document PUT /books/book/978-1449358549 { "name" : "Elasticsearch - The definitive guide", "pages" : 721, "isbn13" : "978-1449358549", "authors" : [ "Clinton Gormley" , "Zachary Tong" ], "publish_date" : "2015/01/31", "category" : "IT > Search Engines", "description" : "Whether you need full-text search or real-time analytics...", } 6

Slide 7

Slide 7 text

{ } CC-BY-ND 4.0 Indexing a document 7 Elasticsearch Client PUT /books/book/978-1449358549 200 OK

Slide 8

Slide 8 text

{ } CC-BY-ND 4.0 Indexing a document 8 Client PUT /books/book/978-1449358549 200 OK ?

Slide 9

Slide 9 text

{ } CC-BY-ND 4.0 Receiving: Where to put it? 9 node 1 PUT /books/book/978-1449358549 { }

Slide 10

Slide 10 text

{ } CC-BY-ND 4.0 Receiving: Where to put it? 10 node 1 PUT /books/book/978-1449358549 { } hash(978-1449358549) % number_of_shards

Slide 11

Slide 11 text

{ } CC-BY-ND 4.0 Receiving: Where to put it? 11 node 1 PUT /books/book/978-1449358549 { } node 2

Slide 12

Slide 12 text

{ } CC-BY-ND 4.0 Next step: Analysis 12 Each field is put into the inverted index { "name" : "Elasticsearch - The definitive guide", "pages" : 721, "isbn13" : "978-1449358549", "authors" : [ "Clinton Gormley" , "Zachary Tong" ], "publish_date" : "2015/01/31", "category" : "IT > Search Engines", "description" : "Whether you need full-text search or real-time analytics...", }

Slide 13

Slide 13 text

{ } CC-BY-ND 4.0 Analysis 13 Each field is put into the inverted index Clinton Gormley -> Input Tokenization Analysis Clinton Gormley [Clinton, Gormley] [clinton, gormley] Zachary Tong [Zachary, Tong] [zachary, tong]

Slide 14

Slide 14 text

{ } CC-BY-ND 4.0 Analysis 14 Resulting in a postings list Clinton Gormley -> Value Document ID clinton 1 gormley 1 tong 1 zachary 1

Slide 15

Slide 15 text

{ } CC-BY-ND 4.0 Lucene at work: Field by field { "name" : "Elasticsearch - The definitive guide", "pages" : 721, "isbn13" : "978-1449358549", "authors" : [ "Clinton Gormley" , "Zachary Tong" ], "publish_date" : "2015/01/31", "category" : "IT > Search Engines", "description" : "Whether you need full-text search or real-time analytics...", } 15

Slide 16

Slide 16 text

{ } CC-BY-ND 4.0 Special case: _all field { "name" : "Elasticsearch - The definitive guide", "pages" : 721, "isbn13" : "978-1449358549", "authors" : [ "Clinton Gormley" , "Zachary Tong" ], "publish_date" : "2015/01/31", "category" : "IT > Search Engines", "description" : "Whether you need full-text search or real-time analytics...", } 16

Slide 17

Slide 17 text

{ } CC-BY-ND 4.0 Special case: _source field { "name" : "Elasticsearch - The definitive guide", "pages" : 721, "isbn13" : "978-1449358549", "authors" : [ "Clinton Gormley" , "Zachary Tong" ], "publish_date" : "2015/01/31", "category" : "IT > Search Engines", "description" : "Whether you need full-text search or real-time analytics...", } 17

Slide 18

Slide 18 text

{ } CC-BY-ND 4.0 Multifields PUT /books/book/_mapping { "book" : { "properties" : { "authors": { "type": "string", "analyzer": "standard", "fields": { "raw": { "type": "string", "index": “not_analyzed" } } } } } } 18 Sorting & Aggregations

Slide 19

Slide 19 text

{ } CC-BY-ND 4.0 Multifields 19 Resulting in multiple postings lists Field: authors Value Document ID clinton 1 gormley 1 tong 1 zachary 1 Value Document ID Clinton Gormley 1 Zachary Tong 1 Field: authors.raw

Slide 20

Slide 20 text

{ } CC-BY-ND 4.0 Completion suggester • Auto-complete, search-as-you-type • Data structure is stored on disk on indexing, then loaded into memory 20

Slide 21

Slide 21 text

{ } CC-BY-ND 4.0 Term vectors • Used for term vector highlighter and more like this query • Single document inverted index • Allows to easily find sorted unique terms as well as original positions and offsets 21

Slide 22

Slide 22 text

{ } CC-BY-ND 4.0 Persisting a document 22 buffer translog

Slide 23

Slide 23 text

{ } CC-BY-ND 4.0 Indexing a document 23 buffer translog

Slide 24

Slide 24 text

{ } CC-BY-ND 4.0 Elasticsearch refresh 24 buffer translog segment

Slide 25

Slide 25 text

{ } CC-BY-ND 4.0 Elasticsearch flush 25 buffer translog segment fsync

Slide 26

Slide 26 text

{ } CC-BY-ND 4.0 Where are we now? • Document received via network • Document received on the correct node • Document analyzed, stored in translog & buffer • No reply to the client yet that everything is done 26

Slide 27

Slide 27 text

{ } CC-BY-ND 4.0 Replication on indexing 27 Document level replication node 1 a0 a1 node 2 a1 a0 node 3 a1 a0

Slide 28

Slide 28 text

{ } CC-BY-ND 4.0 Replication on indexing 28 Document level replication node 1 a0 a1 node 2 a1 a0 node 3 a1 a0

Slide 29

Slide 29 text

{ } CC-BY-ND 4.0 Replication on indexing 29 Document level replication node 1 a0 a1 node 2 a1 a0 node 3 a1 a0

Slide 30

Slide 30 text

{ } CC-BY-ND 4.0 Replication on indexing 30 Document level replication node 1 a0 a1 node 2 a1 a0 node 3 a1 a0

Slide 31

Slide 31 text

{ } CC-BY-ND 4.0 Replication on indexing 31 Document level replication node 1 a0 a1 node 2 a1 a0 node 3 a1 a0 Document got written 6 times! HTTP/1.1 200 OK

Slide 32

Slide 32 text

{ } CC-BY-ND 4.0 Searching 32

Slide 33

Slide 33 text

{ } CC-BY-ND 4.0 Searches: Collecting node 33 node 1 a0 a1 node 2 a1 a0 node 3 a1 a0 query hits node GET /books/book/_search?q=elasticsearch

Slide 34

Slide 34 text

{ } CC-BY-ND 4.0 Searches: Collecting node 34 node 1 a0 a1 node 2 a1 a0 node 3 a1 a0 QUERY phase executed on each shard

Slide 35

Slide 35 text

{ } CC-BY-ND 4.0 Searches: Collecting node 35 node 1 a0 a1 node 2 a1 a0 node 3 a1 a0 QUERY results are sorted

Slide 36

Slide 36 text

{ } CC-BY-ND 4.0 Searches: Collecting node 36 node 1 a0 a1 node 2 a1 a0 node 3 a1 a0 FETCH phase executed

Slide 37

Slide 37 text

{ } CC-BY-ND 4.0 Searches: Collecting node 37 node 1 a0 a1 node 2 a1 a0 node 3 a1 a0 FETCH results are returned

Slide 38

Slide 38 text

{ } CC-BY-ND 4.0 Searches: Collecting node 38 node 1 a0 a1 node 2 a1 a0 node 3 a1 a0

Slide 39

Slide 39 text

{ } CC-BY-ND 4.0 Search types • Full text search – Goes on disk, access segments, execute search • Aggregation – Loads data into memory from segments, executes on in-memory data structure 39

Slide 40

Slide 40 text

{ } CC-BY-ND 4.0 GET /books/book/_search?search_type=count { "aggs" : { "author-top10" : { "terms" : { "field" : "authors" } } } } 40 Memory: Fielddata

Slide 41

Slide 41 text

{ } CC-BY-ND 4.0 41 This does not work for aggs Clinton Gormley -> Term Document ID clinton 1 gormley 1 tong 1 zachary 1 Memory: Fielddata

Slide 42

Slide 42 text

{ } CC-BY-ND 4.0 Memory: Fielddata 42 Uninverting the index from a raw field Document ID Term 1 Clinton Gormley, Zachary Tong Term Document ID Clinton Gormley 1 Zachary Tong 1

Slide 43

Slide 43 text

{ } CC-BY-ND 4.0 Memory: Fielddata • Used for sorting & aggregations • All values of a field are loaded into memory • Per segment data structure • Can take a lot of memory 43

Slide 44

Slide 44 text

{ } CC-BY-ND 4.0 Doc values 44 On-Disk vs. in-memory PUT /books/book/_mapping { "book" : { "properties" : { "authors": { "type": "string", "analyzer": "standard", "fields": { "raw": { "type": "string", "index": “not_analyzed", "doc_values" : true } } } } } }

Slide 45

Slide 45 text

{ } CC-BY-ND 4.0 Where are we now? • Original source is sent to collecting node during queries • Document lives in a segment as part of a shard/index • Document fields live in memory as part of fielddata • Segments are put into the filesystem cache by the OS 45

Slide 46

Slide 46 text

{ } CC-BY-ND 4.0 Scaling your cluster 46

Slide 47

Slide 47 text

{ } CC-BY-ND 4.0 Relocation 47 Let’s offload our fast node node 1 0P STARTED 1P STARTED node 2 node.box_type: large node.box_type: small PUT /books/ { "settings" : { "index.routing.allocation.include.tag" : “large" } }

Slide 48

Slide 48 text

{ } CC-BY-ND 4.0 Relocation 48 Let’s offload our fast node node 1 0P STARTED 1P STARTED node 2 node.box_type: large node.box_type: small PUT /books/_settings { "index.routing.allocation.include.tag" : "small" }

Slide 49

Slide 49 text

{ } CC-BY-ND 4.0 Relocation 49 Let’s offload our fast node node 1 0P STARTED 1P RELOCATING node 2 node.box_type: large node.box_type: small PUT /books/_settings { "index.routing.allocation.include.tag" : "small" } 1P INITIALIZING

Slide 50

Slide 50 text

{ } CC-BY-ND 4.0 Relocation 50 Let’s offload our fast node node 1 0P STARTED node 2 node.box_type: large node.box_type: small PUT /books/_settings { "index.routing.allocation.include.tag" : "small" } 1P STARTED

Slide 51

Slide 51 text

{ } CC-BY-ND 4.0 Relocation 51 Let’s offload our fast node node 1 0P RELOCATING node 2 node.box_type: large node.box_type: small PUT /books/_settings { "index.routing.allocation.include.tag" : "small" } 1P STARTED 1P INITIALIZING

Slide 52

Slide 52 text

{ } CC-BY-ND 4.0 Relocation 52 Let’s offload our fast node node 1 node 2 node.box_type: large node.box_type: small PUT /books/_settings { "index.routing.allocation.include.tag" : "small" } 1P STARTED 1P STARTED

Slide 53

Slide 53 text

{ } CC-BY-ND 4.0 Relocation 53 What does RELOCATING mean? • Point in time snapshot is copied (segments) • Write operations occur and create new segments • All write operations are stored in translog • This translog is sent to the other node and replayed node 1 0P STARTED 1P RELOCATING

Slide 54

Slide 54 text

{ } CC-BY-ND 4.0 Relocation • Process of copying whole shards across the cluster • Triggered by – adding/removing nodes from the cluster – exceeded disk thresholds – shard allocation filtering 54

Slide 55

Slide 55 text

{ } CC-BY-ND 4.0 Where are we now? • Relocation moves shards across the cluster • Document is duplicated by being copied from one node to another, as part of a segment • Segments are kept open until operation is complete 55

Slide 56

Slide 56 text

{ } CC-BY-ND 4.0 Deleting data 56

Slide 57

Slide 57 text

{ } CC-BY-ND 4.0 Data deletion 57 node 1 b0 b1 node 2 b1 b0 node 3 b0 b1 DELETE /books/

Slide 58

Slide 58 text

{ } CC-BY-ND 4.0 Data deletion 58 node 1 node 2 node 3 DELETE /books/

Slide 59

Slide 59 text

{ } CC-BY-ND 4.0 Restore from a snapshot 59 node 1 node 2 node 3 POST /_snapshot/books/20150310/_restore

Slide 60

Slide 60 text

{ } CC-BY-ND 4.0 Restore from a snapshot 60 node 1 node 2 node 3 POST /_snapshot/books/20150310/_restore Repository S3, FS, HDFS Azure

Slide 61

Slide 61 text

{ } CC-BY-ND 4.0 Restore from a snapshot 61 node 1 node 2 node 3 POST /_snapshot/books/20150310/_restore b0 b1

Slide 62

Slide 62 text

{ } CC-BY-ND 4.0 Restore from a snapshot 62 node 1 node 2 node 3 POST /_snapshot/books/20150310/_restore b1 b0 b1 b0

Slide 63

Slide 63 text

{ } CC-BY-ND 4.0 Creating a snapshot 63 node 1 node 2 node 3 PUT /_snapshot/books/20150310 b1 b0 b1 b0

Slide 64

Slide 64 text

{ } CC-BY-ND 4.0 Data deletion 64 node 1 b0 b1 node 2 b1 b0 node 3 b0 b1 DELETE /books/book/978-1449358549

Slide 65

Slide 65 text

{ } CC-BY-ND 4.0 Document deletion • Documents are immutable • Deletes are soft, document ids are added in a tombstone file to be marked for deletion • Actual deletion happens on next merge of that segment 65

Slide 66

Slide 66 text

{ } CC-BY-ND 4.0 Lucene merges 66 segment • Segments are created all the time when indexing data • Merging creates one big segment from several smaller ones

Slide 67

Slide 67 text

{ } CC-BY-ND 4.0 Lucene merges 67 segment segment

Slide 68

Slide 68 text

{ } CC-BY-ND 4.0 Lucene merges 68 segment segment segment

Slide 69

Slide 69 text

{ } CC-BY-ND 4.0 Lucene merges 69 segment segment segment segment

Slide 70

Slide 70 text

{ } CC-BY-ND 4.0 Lucene merges 70 segment segment segment segment document exists twice!

Slide 71

Slide 71 text

{ } CC-BY-ND 4.0 Lucene merges 71 segment segment segment segment

Slide 72

Slide 72 text

{ } CC-BY-ND 4.0 Lucene merges 72 segment

Slide 73

Slide 73 text

{ } Thanks for listening! Questions? @bleskes

Slide 74

Slide 74 text

{ } This work is licensed under the Creative Commons Attribution-NoDerivatives 4.0 International License. To view a copy of this license, visit: http://creativecommons.org/licenses/by-nd/4.0/ or send a letter to: Creative Commons PO Box 1866 Mountain View, CA 94042 USA CC-BY-ND 4.0