Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Life of a Document in Elasticsearch

Elastic Co
March 19, 2015

Life of a Document in Elasticsearch

Ever wondered about the lifecycle of a single document in Elasticsearch? What happens when you index it? How does Elasticsearch ensure a document is replicated and found across the whole cluster reliably? Does deleting a document really physically remove the document from disk? How do we get from a blob of text and keywords to near real-time search and analytics?

This talk will explain the when, where and why of your document's life inside of Elasticsearch. Alex and Boaz will take you through the journey of a document across a cluster, taking off with JSON and the curly braces, travelling through the network into the memory and all the analysis chains, heading further onto storage when writing into the transaction logs and Apache Lucene index, being read back by executing searches, all the way until the document is finally deleted.

Even though this talk will cover a lot of different aspects, it's a talk for those who may be less familiar with core searchfunctionality. You do not need to be an Apache Lucene wizard to follow along and find this session useful.

Elastic Co

March 19, 2015
Tweet

More Decks by Elastic Co

Other Decks in Technology

Transcript

  1. Life of a document in Elasticsearch
    Alexander Reelsen - @spinscale
    Boaz Leskes - @bleskes

    View Slide

  2. { } CC-BY-ND 4.0
    Agenda
    2
    Life of a document in Elasticsearch

    View Slide

  3. { } CC-BY-ND 4.0
    Agenda
    3
    Life of a document in Elasticsearch
    How, when and where is a document
    stored/processed in Elasticsearch?

    View Slide

  4. { } CC-BY-ND 4.0
    About us
    Core developer & Marvel lead
    4
    Core & Shield developer
    Boaz Leskes
    Alexander Reelsen

    View Slide

  5. { } CC-BY-ND 4.0
    Exhibit A: A JSON document
    {
    "name" : "Elasticsearch - The definitive guide",
    "pages" : 721, "isbn13" : "978-1449358549",
    "authors" : [ "Clinton Gormley" , "Zachary Tong" ],
    "publish_date" : "2015/01/31",
    "category" : "IT > Search Engines",
    "description" : "Whether you need full-text search
    or real-time analytics...",
    }
    5

    View Slide

  6. { } CC-BY-ND 4.0
    Exhibit A: A JSON document
    PUT /books/book/978-1449358549
    {
    "name" : "Elasticsearch - The definitive guide",
    "pages" : 721, "isbn13" : "978-1449358549",
    "authors" : [ "Clinton Gormley" , "Zachary Tong" ],
    "publish_date" : "2015/01/31",
    "category" : "IT > Search Engines",
    "description" : "Whether you need full-text search
    or real-time analytics...",
    }
    6

    View Slide

  7. { } CC-BY-ND 4.0
    Indexing a document
    7
    Elasticsearch
    Client
    PUT /books/book/978-1449358549
    200 OK

    View Slide

  8. { } CC-BY-ND 4.0
    Indexing a document
    8
    Client
    PUT /books/book/978-1449358549
    200 OK
    ?

    View Slide

  9. { } CC-BY-ND 4.0
    Receiving: Where to store it?
    9
    node  1
    C
    PUT /books/book/978-1449358549
    {
    }

    View Slide

  10. { } CC-BY-ND 4.0
    Receiving: Where to put it?
    10
    node  1
    C
    PUT /books/book/978-1449358549
    {
    }
    hash(978-1449358549) % number_of_shards

    View Slide

  11. { } CC-BY-ND 4.0
    Receiving: Where to put it?
    11
    node  1
    C
    PUT /books/book/978-1449358549
    {
    }
    node  2

    View Slide

  12. { } CC-BY-ND 4.0
    Next step: Analysis
    12
    Each field is put into the inverted index
    {
    "name" : "Elasticsearch - The definitive guide",
    "pages" : 721, "isbn13" : "978-1449358549",
    "authors" : [ "Clinton Gormley" , "Zachary Tong" ],
    "publish_date" : "2015/01/31",
    "category" : "IT > Search Engines",
    "description" : "Whether you need full-text search
    or real-time analytics...",
    }

    View Slide

  13. { } CC-BY-ND 4.0
    Analysis
    13
    Each field is put into the inverted index
    Clinton Gormley ->
    Input Tokenization Analysis
    Clinton  Gormley [Clinton,  Gormley] [clinton,  gormley]
    Zachary  Tong [Zachary,  Tong] [zachary,  tong]

    View Slide

  14. { } CC-BY-ND 4.0
    Analysis
    14
    Resulting in a postings list
    Clinton Gormley ->
    Value Document  ID
    clinton 1
    gormley 1
    tong 1
    zachary 1

    View Slide

  15. { } CC-BY-ND 4.0
    Lucene at work: Field by field
    {
    "name" : "Elasticsearch - The definitive guide",
    "pages" : 721, "isbn13" : "978-1449358549",
    "authors" : [ "Clinton Gormley" , "Zachary Tong" ],
    "publish_date" : "2015/01/31",
    "category" : "IT > Search Engines",
    "description" : "Whether you need full-text search
    or real-time analytics...",
    }
    15

    View Slide

  16. { } CC-BY-ND 4.0
    Special case: _all field
    {
    "name" : "Elasticsearch - The definitive guide",
    "pages" : 721, "isbn13" : "978-1449358549",
    "authors" : [ "Clinton Gormley" , "Zachary Tong" ],
    "publish_date" : "2015/01/31",
    "category" : "IT > Search Engines",
    "description" : "Whether you need full-text search
    or real-time analytics...",
    }
    16

    View Slide

  17. { } CC-BY-ND 4.0
    Special case: _source field
    {
    "name" : "Elasticsearch - The definitive guide",
    "pages" : 721, "isbn13" : "978-1449358549",
    "authors" : [ "Clinton Gormley" , "Zachary Tong" ],
    "publish_date" : "2015/01/31",
    "category" : "IT > Search Engines",
    "description" : "Whether you need full-text search
    or real-time analytics...",
    }
    17

    View Slide

  18. { } CC-BY-ND 4.0
    Multifields
    PUT /books/book/_mapping
    {
    "book" : {
    "properties" : {
    "authors": {
    "type": "string", "analyzer": "standard",
    "fields": { "raw": { "type": "string", "index": “not_analyzed" } }
    }
    }
    }
    }
    18
    Sorting & Aggregations

    View Slide

  19. { } CC-BY-ND 4.0
    Multifields
    19
    Resulting in multiple postings lists
    Field: authors Value Document  ID
    clinton 1
    gormley 1
    tong 1
    zachary 1
    Value Document  ID
    Clinton  Gormley 1
    Zachary  Tong 1
    Field: authors.raw

    View Slide

  20. { } CC-BY-ND 4.0
    Completion suggester
    • Auto-complete, search-as-you-type
    • Data structure is stored on disk on indexing, then loaded
    into memory
    20

    View Slide

  21. { } CC-BY-ND 4.0
    Term vectors
    • Handy when hit highlighting huge documents
    • An index of a single document
    • Allows to easily find sorted unique terms as well as
    original positions and offsets
    21

    View Slide

  22. { } CC-BY-ND 4.0
    Buffering documents
    22
    buffer
    buffer

    View Slide

  23. { } CC-BY-ND 4.0
    Buffering documents
    23
    buffer
    buffer

    View Slide

  24. { } CC-BY-ND 4.0
    Buffering documents
    24
    buffer segment

    View Slide

  25. { } CC-BY-ND 4.0
    Elasticsearch refresh
    25
    buffer
    translog

    View Slide

  26. { } CC-BY-ND 4.0
    Elasticsearch refresh
    26
    buffer
    translog
    segment

    View Slide

  27. { } CC-BY-ND 4.0
    Indexing a document
    27
    buffer
    translog

    View Slide

  28. { } CC-BY-ND 4.0
    Elasticsearch refresh
    28
    buffer
    translog
    segment

    View Slide

  29. { } CC-BY-ND 4.0
    Elasticsearch flush
    29
    buffer
    translog
    segment
    fsync

    View Slide

  30. { } CC-BY-ND 4.0
    Where are we now?
    • Document received via network
    • Document received on the correct node
    • Document analyzed, stored in translog & buffer
    • Client is still waiting for a response
    30

    View Slide

  31. { } CC-BY-ND 4.0
    Replication on indexing
    31
    Document level replication
    node  1
    b0 b1
    node  2
    b1 b0
    node  3
    b1 b0
    C

    View Slide

  32. { } CC-BY-ND 4.0
    Replication on indexing
    32
    Document level replication
    node  1
    b0 b1
    node  2
    b1 b0
    node  3
    b1 b0
    C

    View Slide

  33. { } CC-BY-ND 4.0
    Replication on indexing
    33
    Document level replication
    node  1
    b0 b1
    node  2
    b1 b0
    node  3
    b1 b0
    C

    View Slide

  34. { } CC-BY-ND 4.0
    Replication on indexing
    34
    Document level replication
    node  1
    b0 b1
    node  2
    b1 b0
    node  3
    b1 b0
    C

    View Slide

  35. { } CC-BY-ND 4.0
    Replication on indexing
    35
    Document level replication
    node  1
    b0 b1
    node  2
    b1 b0
    node  3
    b1 b0
    C

    View Slide

  36. { } CC-BY-ND 4.0
    Replication on indexing
    36
    Document level replication
    node  1
    b0 b1
    node  2
    b1 b0
    node  3
    b1 b0
    C
    Document got written 6 times!
    HTTP/1.1 200 OK

    View Slide

  37. { } CC-BY-ND 4.0
    Searching
    37

    View Slide

  38. { } CC-BY-ND 4.0
    Searches: Collecting node
    38
    node  1
    b0 b1
    node  2
    b1 b0
    node  3
    b1 b0
    C
    query hits node
    GET /books/book/_search?q=elasticsearch

    View Slide

  39. { } CC-BY-ND 4.0
    Searches: Collecting node
    39
    node  1
    b0 b1
    node  2
    b1 b0
    node  3
    b1 b0
    C
    QUERY phase executed on each shard

    View Slide

  40. { } CC-BY-ND 4.0
    Searches: Collecting node
    40
    node  1
    b0 b1
    node  2
    b1 b0
    node  3
    b1 b0
    C
    QUERY results are sorted

    View Slide

  41. { } CC-BY-ND 4.0
    Searches: Collecting node
    41
    node  1
    b0 b1
    node  2
    b1 b0
    node  3
    b1 b0
    C
    FETCH phase executed

    View Slide

  42. { } CC-BY-ND 4.0
    Searches: Collecting node
    42
    node  1
    b0 b1
    node  2
    b1 b0
    node  3
    b1 b0
    C
    FETCH results are returned

    View Slide

  43. { } CC-BY-ND 4.0
    Searches: Collecting node
    43
    node  1
    b0 b1
    node  2
    b1 b0
    node  3
    b1 b0
    C

    View Slide

  44. { } CC-BY-ND 4.0
    Search types
    • Full text search
    – Goes on disk, access segments, execute search
    • Aggregation
    – Loads data into memory from segments, executes on
    in-memory data structure
    44

    View Slide

  45. { } CC-BY-ND 4.0
    Memory: Fielddata
    GET /books/book/_search?search_type=count
    {
    "aggs" : {
    "author-top10" : {
    "terms" : {
    "field" : “authors.raw"
    }
    }
    }
    }
    45

    View Slide

  46. { } CC-BY-ND 4.0
    Memory: Fielddata
    46
    This does not work for aggs
    Clinton Gormley ->
    Term Document  ID
    clinton 1
    gormley 1
    tong 1
    zachary 1

    View Slide

  47. { } CC-BY-ND 4.0
    Memory: Fielddata
    47
    Uninverting the index from a raw field
    Document  ID Term
    1 Clinton  Gormley,  Zachary  Tong
    Term Document  ID
    Clinton  Gormley 1
    Zachary  Tong 1

    View Slide

  48. { } CC-BY-ND 4.0
    Memory: Fielddata
    • Used for sorting & aggregations
    • All values of a field are loaded into the heap
    • Per segment data structure
    • Can take a lot of memory
    48

    View Slide

  49. { } CC-BY-ND 4.0
    Doc values
    49
    On-Disk vs. in-memory
    PUT /books/book/_mapping
    {
    "book" : {
    "properties" : {
    "authors": {
    "type": "string", "analyzer": "standard",
    "fields": { "raw": { "type": "string", "index": "not_analyzed",
    "doc_values" : true } }
    }
    }
    }
    }

    View Slide

  50. { } CC-BY-ND 4.0
    Where are we now?
    • Original source is sent to collecting node during queries
    • Document lives in a segment as part of a shard/index
    • Document fields live in memory as part of fielddata
    • Segments are put into the filesystem cache by the OS
    50

    View Slide

  51. { } CC-BY-ND 4.0
    Scaling your cluster
    51

    View Slide

  52. { } CC-BY-ND 4.0
    Relocation
    52
    Let’s offload our fast node
    node  1
    0P  
    STARTED
    1P  
    STARTED
    node  2
    node.box_type: ssd node.box_type: hdd
    PUT /books/
    {
    "settings" : {
    “index.routing.allocation.include.box_type" : “ssd"
    }
    }

    View Slide

  53. { } CC-BY-ND 4.0
    Relocation
    53
    Let’s offload our fast node
    node  1
    0P  
    STARTED
    1P  
    STARTED
    node  2
    node.box_type: ssd node.box_type: hdd
    PUT /books/_settings
    {
    "index.routing.allocation.include.box_type" : "hdd"
    }

    View Slide

  54. { } CC-BY-ND 4.0
    Relocation
    54
    Let’s offload our fast node
    node  1
    0P  
    STARTED
    1P  
    RELOCATING
    node  2
    node.box_type: ssd node.box_type: hdd
    PUT /books/_settings
    {
    "index.routing.allocation.include.box_type" : "hdd"
    }
    1P  
    INITIALIZING

    View Slide

  55. { } CC-BY-ND 4.0
    Relocation
    55
    Let’s offload our fast node
    node  1
    0P  
    STARTED
    node  2
    node.box_type: ssd node.box_type: hdd
    PUT /books/_settings
    {
    "index.routing.allocation.include.box_type" : "hdd"
    }
    1P  
    STARTED

    View Slide

  56. { } CC-BY-ND 4.0
    Relocation
    56
    Let’s offload our fast node
    node  1
    0P  
    RELOCATING
    node  2
    node.box_type: sdd node.box_type: hdd
    PUT /books/_settings
    {
    "index.routing.allocation.include.box_type" : "hdd"
    }
    1P  
    STARTED
    1P  
    INITIALIZING

    View Slide

  57. { } CC-BY-ND 4.0
    Relocation
    57
    Let’s offload our fast node
    node  1 node  2
    node.box_type: ssd node.box_type: hdd
    PUT /books/_settings
    {
    "index.routing.allocation.include.box_type" : "hdd"
    }
    1P  
    STARTED
    1P  
    STARTED

    View Slide

  58. { } CC-BY-ND 4.0
    Relocation
    58
    What does RELOCATING mean?
    • Point in time snapshot is copied (segments)
    • Write operations occur and create new
    segments
    • All write operations are stored in translog
    • This translog is sent to the other node and
    replayed
    node  1
    0P  
    STARTED
    1P  
    RELOCATING

    View Slide

  59. { } CC-BY-ND 4.0
    Relocation
    • Process of copying whole shards across the cluster
    • Triggered by
    – adding/removing nodes from the cluster
    – exceeded disk thresholds
    – shard allocation filtering
    59

    View Slide

  60. { } CC-BY-ND 4.0
    Where are we now?
    • Relocation moves shards across the cluster
    • Document is duplicated by being copied from one node
    to another, as part of a segment
    • Segments are kept open until operation is complete
    60

    View Slide

  61. { } CC-BY-ND 4.0
    Backup your data
    61

    View Slide

  62. { } CC-BY-ND 4.0
    How to backup?
    62
    node  1
    b0 b1
    node  2
    b1 b0
    node  3
    b1 b0

    View Slide

  63. { } CC-BY-ND 4.0
    How to backup?
    63
    node  1
    b0 b1
    node  2
    b1 b0
    node  3
    b1 b0
    Repository
    S3, FS,
    HDFS
    Azure

    View Slide

  64. { } CC-BY-ND 4.0
    How to backup?
    64
    node  1
    b0 b1
    node  2
    b1 b0
    node  3
    b0 b1
    S3, FS,
    HDFS
    Azure C PUT /_snapshot/books/20150310
    b1
    b0

    View Slide

  65. { } CC-BY-ND 4.0
    Restore from a snapshot
    65
    node  1 node  2 node  3
    C POST /_snapshot/books/20150310/_restore
    b0
    b1

    View Slide

  66. { } CC-BY-ND 4.0
    Restore from a snapshot
    66
    node  1 node  2 node  3
    C POST /_snapshot/books/20150310/_restore
    b1
    b0 b1
    b0

    View Slide

  67. { } CC-BY-ND 4.0
    Deleting data
    67

    View Slide

  68. { } CC-BY-ND 4.0
    Index deletion
    68
    node  1
    b0 b1
    node  2
    b1 b0
    node  3
    b0 b1
    C
    DELETE /books/

    View Slide

  69. { } CC-BY-ND 4.0
    Index deletion
    69
    node  1 node  2 node  3
    C
    DELETE /books/

    View Slide

  70. { } CC-BY-ND 4.0
    Document deletion
    70
    node  1
    b0 b1
    node  2
    b1 b0
    node  3
    b0 b1
    C
    DELETE /books/book/978-1449358549

    View Slide

  71. { } CC-BY-ND 4.0
    Document deletion
    • Segments are immutable
    • Deletes are soft, document ids are added in a tombstone
    file to be marked for deletion
    • Actual deletion happens on next merge of that segment
    71

    View Slide

  72. { } CC-BY-ND 4.0
    Lucene merges
    72
    segment
    • Segments are created all the time when
    indexing data
    • Merging creates one big segment from
    several smaller ones

    View Slide

  73. { } CC-BY-ND 4.0
    Lucene merges
    73
    segment
    segment

    View Slide

  74. { } CC-BY-ND 4.0
    Lucene merges
    74
    segment
    segment
    segment

    View Slide

  75. { } CC-BY-ND 4.0
    Lucene merges
    75
    segment
    segment
    segment
    segment

    View Slide

  76. { } CC-BY-ND 4.0
    Lucene merges
    76
    segment
    segment
    segment
    segment document exists twice!

    View Slide

  77. { } CC-BY-ND 4.0
    Lucene merges
    77
    segment
    segment
    segment
    segment

    View Slide

  78. { } CC-BY-ND 4.0
    Lucene merges
    78
    segment

    View Slide

  79. { } CC-BY-ND 4.0
    Don’t be afraid of data duplication!
    79

    View Slide

  80. { } CC-BY-ND 4.0
    Don’t be afraid of data duplication!
    ... if you are aware of its lifecycle!
    80

    View Slide

  81. { }
    Thanks for listening!
    Questions?
    @bleskes
    @spinscale

    View Slide

  82. { }
    This work is licensed under the Creative Commons
    Attribution-NoDerivatives 4.0 International License.
    To view a copy of this license, visit:
    http://creativecommons.org/licenses/by-nd/4.0/
    or send a letter to:
    Creative Commons
    PO Box 1866
    Mountain View, CA 94042
    USA
    CC-BY-ND 4.0

    View Slide