Life of a Document in Elasticsearch

Slide 1

Slide 1 text

Life of a document in Elasticsearch Alexander Reelsen - @spinscale Boaz Leskes - @bleskes

Slide 2

Slide 2 text

{ } CC-BY-ND 4.0 Agenda 2 Life of a document in Elasticsearch

Slide 3

Slide 3 text

{ } CC-BY-ND 4.0 Agenda 3 Life of a document in Elasticsearch How, when and where is a document stored/processed in Elasticsearch?

Slide 4

Slide 4 text

{ } CC-BY-ND 4.0 About us Core developer & Marvel lead 4 Core & Shield developer Boaz Leskes Alexander Reelsen

Slide 5

Slide 5 text

{ } CC-BY-ND 4.0 Exhibit A: A JSON document { "name" : "Elasticsearch - The definitive guide", "pages" : 721, "isbn13" : "978-1449358549", "authors" : [ "Clinton Gormley" , "Zachary Tong" ], "publish_date" : "2015/01/31", "category" : "IT > Search Engines", "description" : "Whether you need full-text search or real-time analytics...", } 5

Slide 6

Slide 6 text

{ } CC-BY-ND 4.0 Exhibit A: A JSON document PUT /books/book/978-1449358549 { "name" : "Elasticsearch - The definitive guide", "pages" : 721, "isbn13" : "978-1449358549", "authors" : [ "Clinton Gormley" , "Zachary Tong" ], "publish_date" : "2015/01/31", "category" : "IT > Search Engines", "description" : "Whether you need full-text search or real-time analytics...", } 6

Slide 7

Slide 7 text

{ } CC-BY-ND 4.0 Indexing a document 7 Elasticsearch Client PUT /books/book/978-1449358549 200 OK

Slide 8

Slide 8 text

{ } CC-BY-ND 4.0 Indexing a document 8 Client PUT /books/book/978-1449358549 200 OK ?

Slide 9

Slide 9 text

{ } CC-BY-ND 4.0 Receiving: Where to store it? 9 node 1 C PUT /books/book/978-1449358549 { }

Slide 10

Slide 10 text

{ } CC-BY-ND 4.0 Receiving: Where to put it? 10 node 1 C PUT /books/book/978-1449358549 { } hash(978-1449358549) % number_of_shards

Slide 11

Slide 11 text

{ } CC-BY-ND 4.0 Receiving: Where to put it? 11 node 1 C PUT /books/book/978-1449358549 { } node 2

Slide 12

Slide 12 text

{ } CC-BY-ND 4.0 Next step: Analysis 12 Each field is put into the inverted index { "name" : "Elasticsearch - The definitive guide", "pages" : 721, "isbn13" : "978-1449358549", "authors" : [ "Clinton Gormley" , "Zachary Tong" ], "publish_date" : "2015/01/31", "category" : "IT > Search Engines", "description" : "Whether you need full-text search or real-time analytics...", }

Slide 13

Slide 13 text

{ } CC-BY-ND 4.0 Analysis 13 Each field is put into the inverted index Clinton Gormley -> Input Tokenization Analysis Clinton Gormley [Clinton, Gormley] [clinton, gormley] Zachary Tong [Zachary, Tong] [zachary, tong]

Slide 14

Slide 14 text

{ } CC-BY-ND 4.0 Analysis 14 Resulting in a postings list Clinton Gormley -> Value Document ID clinton 1 gormley 1 tong 1 zachary 1

Slide 15

Slide 15 text

{ } CC-BY-ND 4.0 Lucene at work: Field by field { "name" : "Elasticsearch - The definitive guide", "pages" : 721, "isbn13" : "978-1449358549", "authors" : [ "Clinton Gormley" , "Zachary Tong" ], "publish_date" : "2015/01/31", "category" : "IT > Search Engines", "description" : "Whether you need full-text search or real-time analytics...", } 15

Slide 16

Slide 16 text

{ } CC-BY-ND 4.0 Special case: _all field { "name" : "Elasticsearch - The definitive guide", "pages" : 721, "isbn13" : "978-1449358549", "authors" : [ "Clinton Gormley" , "Zachary Tong" ], "publish_date" : "2015/01/31", "category" : "IT > Search Engines", "description" : "Whether you need full-text search or real-time analytics...", } 16

Slide 17

Slide 17 text

{ } CC-BY-ND 4.0 Special case: _source field { "name" : "Elasticsearch - The definitive guide", "pages" : 721, "isbn13" : "978-1449358549", "authors" : [ "Clinton Gormley" , "Zachary Tong" ], "publish_date" : "2015/01/31", "category" : "IT > Search Engines", "description" : "Whether you need full-text search or real-time analytics...", } 17

Slide 18

Slide 18 text

{ } CC-BY-ND 4.0 Multifields PUT /books/book/_mapping { "book" : { "properties" : { "authors": { "type": "string", "analyzer": "standard", "fields": { "raw": { "type": "string", "index": “not_analyzed" } } } } } } 18 Sorting & Aggregations

Slide 19

Slide 19 text

{ } CC-BY-ND 4.0 Multifields 19 Resulting in multiple postings lists Field: authors Value Document ID clinton 1 gormley 1 tong 1 zachary 1 Value Document ID Clinton Gormley 1 Zachary Tong 1 Field: authors.raw

Slide 20

Slide 20 text

{ } CC-BY-ND 4.0 Completion suggester • Auto-complete, search-as-you-type • Data structure is stored on disk on indexing, then loaded into memory 20

Slide 21

Slide 21 text

{ } CC-BY-ND 4.0 Term vectors • Handy when hit highlighting huge documents • An index of a single document • Allows to easily find sorted unique terms as well as original positions and offsets 21

Slide 22

Slide 22 text

{ } CC-BY-ND 4.0 Buffering documents 22 buffer buffer

Slide 23

Slide 23 text

{ } CC-BY-ND 4.0 Buffering documents 23 buffer buffer

Slide 24

Slide 24 text

{ } CC-BY-ND 4.0 Buffering documents 24 buffer segment

Slide 25

Slide 25 text

{ } CC-BY-ND 4.0 Elasticsearch refresh 25 buffer translog

Slide 26

Slide 26 text

{ } CC-BY-ND 4.0 Elasticsearch refresh 26 buffer translog segment

Slide 27

Slide 27 text

{ } CC-BY-ND 4.0 Indexing a document 27 buffer translog

Slide 28

Slide 28 text

{ } CC-BY-ND 4.0 Elasticsearch refresh 28 buffer translog segment

Slide 29

Slide 29 text

{ } CC-BY-ND 4.0 Elasticsearch flush 29 buffer translog segment fsync

Slide 30

Slide 30 text

{ } CC-BY-ND 4.0 Where are we now? • Document received via network • Document received on the correct node • Document analyzed, stored in translog & buffer • Client is still waiting for a response 30

Slide 31

Slide 31 text

{ } CC-BY-ND 4.0 Replication on indexing 31 Document level replication node 1 b0 b1 node 2 b1 b0 node 3 b1 b0 C

Slide 32

Slide 32 text

{ } CC-BY-ND 4.0 Replication on indexing 32 Document level replication node 1 b0 b1 node 2 b1 b0 node 3 b1 b0 C

Slide 33

Slide 33 text

{ } CC-BY-ND 4.0 Replication on indexing 33 Document level replication node 1 b0 b1 node 2 b1 b0 node 3 b1 b0 C

Slide 34

Slide 34 text

{ } CC-BY-ND 4.0 Replication on indexing 34 Document level replication node 1 b0 b1 node 2 b1 b0 node 3 b1 b0 C

Slide 35

Slide 35 text

{ } CC-BY-ND 4.0 Replication on indexing 35 Document level replication node 1 b0 b1 node 2 b1 b0 node 3 b1 b0 C

Slide 36

Slide 36 text

{ } CC-BY-ND 4.0 Replication on indexing 36 Document level replication node 1 b0 b1 node 2 b1 b0 node 3 b1 b0 C Document got written 6 times! HTTP/1.1 200 OK

Slide 37

Slide 37 text

{ } CC-BY-ND 4.0 Searching 37

Slide 38

Slide 38 text

{ } CC-BY-ND 4.0 Searches: Collecting node 38 node 1 b0 b1 node 2 b1 b0 node 3 b1 b0 C query hits node GET /books/book/_search?q=elasticsearch

Slide 39

Slide 39 text

{ } CC-BY-ND 4.0 Searches: Collecting node 39 node 1 b0 b1 node 2 b1 b0 node 3 b1 b0 C QUERY phase executed on each shard

Slide 40

Slide 40 text

{ } CC-BY-ND 4.0 Searches: Collecting node 40 node 1 b0 b1 node 2 b1 b0 node 3 b1 b0 C QUERY results are sorted

Slide 41

Slide 41 text

{ } CC-BY-ND 4.0 Searches: Collecting node 41 node 1 b0 b1 node 2 b1 b0 node 3 b1 b0 C FETCH phase executed

Slide 42

Slide 42 text

{ } CC-BY-ND 4.0 Searches: Collecting node 42 node 1 b0 b1 node 2 b1 b0 node 3 b1 b0 C FETCH results are returned

Slide 43

Slide 43 text

{ } CC-BY-ND 4.0 Searches: Collecting node 43 node 1 b0 b1 node 2 b1 b0 node 3 b1 b0 C

Slide 44

Slide 44 text

{ } CC-BY-ND 4.0 Search types • Full text search – Goes on disk, access segments, execute search • Aggregation – Loads data into memory from segments, executes on in-memory data structure 44

Slide 45

Slide 45 text

{ } CC-BY-ND 4.0 Memory: Fielddata GET /books/book/_search?search_type=count { "aggs" : { "author-top10" : { "terms" : { "field" : “authors.raw" } } } } 45

Slide 46

Slide 46 text

{ } CC-BY-ND 4.0 Memory: Fielddata 46 This does not work for aggs Clinton Gormley -> Term Document ID clinton 1 gormley 1 tong 1 zachary 1

Slide 47

Slide 47 text

{ } CC-BY-ND 4.0 Memory: Fielddata 47 Uninverting the index from a raw field Document ID Term 1 Clinton Gormley, Zachary Tong Term Document ID Clinton Gormley 1 Zachary Tong 1

Slide 48

Slide 48 text

{ } CC-BY-ND 4.0 Memory: Fielddata • Used for sorting & aggregations • All values of a field are loaded into the heap • Per segment data structure • Can take a lot of memory 48

Slide 49

Slide 49 text

{ } CC-BY-ND 4.0 Doc values 49 On-Disk vs. in-memory PUT /books/book/_mapping { "book" : { "properties" : { "authors": { "type": "string", "analyzer": "standard", "fields": { "raw": { "type": "string", "index": "not_analyzed", "doc_values" : true } } } } } }

Slide 50

Slide 50 text

{ } CC-BY-ND 4.0 Where are we now? • Original source is sent to collecting node during queries • Document lives in a segment as part of a shard/index • Document fields live in memory as part of fielddata • Segments are put into the filesystem cache by the OS 50

Slide 51

Slide 51 text

{ } CC-BY-ND 4.0 Scaling your cluster 51

Slide 52

Slide 52 text

{ } CC-BY-ND 4.0 Relocation 52 Let’s offload our fast node node 1 0P STARTED 1P STARTED node 2 node.box_type: ssd node.box_type: hdd PUT /books/ { "settings" : { “index.routing.allocation.include.box_type" : “ssd" } }

Slide 53

Slide 53 text

{ } CC-BY-ND 4.0 Relocation 53 Let’s offload our fast node node 1 0P STARTED 1P STARTED node 2 node.box_type: ssd node.box_type: hdd PUT /books/_settings { "index.routing.allocation.include.box_type" : "hdd" }

Slide 54

Slide 54 text

{ } CC-BY-ND 4.0 Relocation 54 Let’s offload our fast node node 1 0P STARTED 1P RELOCATING node 2 node.box_type: ssd node.box_type: hdd PUT /books/_settings { "index.routing.allocation.include.box_type" : "hdd" } 1P INITIALIZING

Slide 55

Slide 55 text

{ } CC-BY-ND 4.0 Relocation 55 Let’s offload our fast node node 1 0P STARTED node 2 node.box_type: ssd node.box_type: hdd PUT /books/_settings { "index.routing.allocation.include.box_type" : "hdd" } 1P STARTED

Slide 56

Slide 56 text

{ } CC-BY-ND 4.0 Relocation 56 Let’s offload our fast node node 1 0P RELOCATING node 2 node.box_type: sdd node.box_type: hdd PUT /books/_settings { "index.routing.allocation.include.box_type" : "hdd" } 1P STARTED 1P INITIALIZING

Slide 57

Slide 57 text

{ } CC-BY-ND 4.0 Relocation 57 Let’s offload our fast node node 1 node 2 node.box_type: ssd node.box_type: hdd PUT /books/_settings { "index.routing.allocation.include.box_type" : "hdd" } 1P STARTED 1P STARTED

Slide 58

Slide 58 text

{ } CC-BY-ND 4.0 Relocation 58 What does RELOCATING mean? • Point in time snapshot is copied (segments) • Write operations occur and create new segments • All write operations are stored in translog • This translog is sent to the other node and replayed node 1 0P STARTED 1P RELOCATING

Slide 59

Slide 59 text

{ } CC-BY-ND 4.0 Relocation • Process of copying whole shards across the cluster • Triggered by – adding/removing nodes from the cluster – exceeded disk thresholds – shard allocation filtering 59

Slide 60

Slide 60 text

{ } CC-BY-ND 4.0 Where are we now? • Relocation moves shards across the cluster • Document is duplicated by being copied from one node to another, as part of a segment • Segments are kept open until operation is complete 60

Slide 61

Slide 61 text

{ } CC-BY-ND 4.0 Backup your data 61

Slide 62

Slide 62 text

{ } CC-BY-ND 4.0 How to backup? 62 node 1 b0 b1 node 2 b1 b0 node 3 b1 b0

Slide 63

Slide 63 text

{ } CC-BY-ND 4.0 How to backup? 63 node 1 b0 b1 node 2 b1 b0 node 3 b1 b0 Repository S3, FS, HDFS Azure

Slide 64

Slide 64 text

{ } CC-BY-ND 4.0 How to backup? 64 node 1 b0 b1 node 2 b1 b0 node 3 b0 b1 S3, FS, HDFS Azure C PUT /_snapshot/books/20150310 b1 b0

Slide 65

Slide 65 text

{ } CC-BY-ND 4.0 Restore from a snapshot 65 node 1 node 2 node 3 C POST /_snapshot/books/20150310/_restore b0 b1

Slide 66

Slide 66 text

{ } CC-BY-ND 4.0 Restore from a snapshot 66 node 1 node 2 node 3 C POST /_snapshot/books/20150310/_restore b1 b0 b1 b0

Slide 67

Slide 67 text

{ } CC-BY-ND 4.0 Deleting data 67

Slide 68

Slide 68 text

{ } CC-BY-ND 4.0 Index deletion 68 node 1 b0 b1 node 2 b1 b0 node 3 b0 b1 C DELETE /books/

Slide 69

Slide 69 text

{ } CC-BY-ND 4.0 Index deletion 69 node 1 node 2 node 3 C DELETE /books/

Slide 70

Slide 70 text

{ } CC-BY-ND 4.0 Document deletion 70 node 1 b0 b1 node 2 b1 b0 node 3 b0 b1 C DELETE /books/book/978-1449358549

Slide 71

Slide 71 text

{ } CC-BY-ND 4.0 Document deletion • Segments are immutable • Deletes are soft, document ids are added in a tombstone file to be marked for deletion • Actual deletion happens on next merge of that segment 71

Slide 72

Slide 72 text

{ } CC-BY-ND 4.0 Lucene merges 72 segment • Segments are created all the time when indexing data • Merging creates one big segment from several smaller ones

Slide 73

Slide 73 text

{ } CC-BY-ND 4.0 Lucene merges 73 segment segment

Slide 74

Slide 74 text

{ } CC-BY-ND 4.0 Lucene merges 74 segment segment segment

Slide 75

Slide 75 text

{ } CC-BY-ND 4.0 Lucene merges 75 segment segment segment segment

Slide 76

Slide 76 text

{ } CC-BY-ND 4.0 Lucene merges 76 segment segment segment segment document exists twice!

Slide 77

Slide 77 text

{ } CC-BY-ND 4.0 Lucene merges 77 segment segment segment segment

Slide 78

Slide 78 text

{ } CC-BY-ND 4.0 Lucene merges 78 segment

Slide 79

Slide 79 text

{ } CC-BY-ND 4.0 Don’t be afraid of data duplication! 79

Slide 80

Slide 80 text

{ } CC-BY-ND 4.0 Don’t be afraid of data duplication! ... if you are aware of its lifecycle! 80

Slide 81

Slide 81 text

{ } Thanks for listening! Questions? @bleskes @spinscale

Slide 82

Slide 82 text

{ } This work is licensed under the Creative Commons Attribution-NoDerivatives 4.0 International License. To view a copy of this license, visit: http://creativecommons.org/licenses/by-nd/4.0/ or send a letter to: Creative Commons PO Box 1866 Mountain View, CA 94042 USA CC-BY-ND 4.0