$30 off During Our Annual Pro Sale. View Details »

Staying Ahead of Time

Boaz Leskes
January 24, 2014

Staying Ahead of Time

elasticsearch and time based data

a talk given at Search Engines Amsterdam

Boaz Leskes

January 24, 2014
Tweet

More Decks by Boaz Leskes

Other Decks in Technology

Transcript

  1. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    elasticsearch and time based data
    Staying Ahead of Time
    Boaz Leskes
    @bleskes

    View Slide

  2. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    indices,types and
    other animals
    Basics

    View Slide

  3. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    A document
    {

    "created_at": "Fri Jan 24 11:15:24 +0000 2014",
    "id": 426674590560305150,
    "text": "Prepping up for my #elasticsearch talk
    this afternoon at the UvA : http://t.co/rqhBI5zys0",
    "user": {
    "name": “Boaz Leskes",
    "screen_name": "bleskes",
    }
    }

    View Slide

  4. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    A type
    {

    "created_at": "Fri Jan 24 11:15:24 +0000 2014",
    "id": 426674590560305150,
    "text": "Prepping up for my #elasticsearch talk
    this afternoon at the UvA : http://t.co/rqhBI5zys0",
    "user": {
    "name": “Boaz Leskes",
    "screen_name": "bleskes",
    }
    }
    = docs with similar data/structure
    {

    "created_at": "Thu Jan 23 18:27:23 +0000 2014",
    "id": 426420915698544640,
    "text": "Elasticsearch es una maravilla !!!!",
    "user": {
    "name": "Abel Coronado",
    "screen_name": "abxda",
    }
    }

    View Slide

  5. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    An index
    = a collection of types
    {

    "created_at": "Thu Jan 23
    "id": 426420915698544640,
    "text": "Elasticsearch Esc
    "user": {
    "name": "Abel Coronado
    "screen_name": "abxda"
    }
    }
    {

    "id": 19726002,
    "name": "Abel Coronado
    "screen_name": "abxda"
    "location":"Aguascalientes"
    "followers_count":871
    "friends_count":1794
    "listed_count":38
    }

    View Slide

  6. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Sharding
    index

    View Slide

  7. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Sharding
    index
    shard shard
    shard shard

    View Slide

  8. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Sharding
    index
    shard shard
    shard shard
    node node

    View Slide

  9. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Sharding
    index
    node node
    shard
    shard
    shard
    shard

    View Slide

  10. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Sharding
    index
    node node
    shard
    shard
    shard
    shard
    node node
    copy
    copy
    copy
    copy

    View Slide

  11. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Sharding
    node node
    copy copy
    node node
    shard
    shard copy
    copy shard
    shard

    View Slide

  12. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Sharding
    node node
    copy copy
    node node
    node node node node
    shard
    shard copy
    copy shard
    shard

    View Slide

  13. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Sharding
    node node
    copy copy
    node node
    node node node node
    shard shard
    shard copy
    copy shard

    View Slide

  14. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Important fact for later
    indexing & searching is done on
    shards, not indices

    View Slide

  15. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    time based data
    To the subject at hand

    View Slide

  16. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

    View Slide

  17. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    easy to get, easy to index
    # curl -XPUT localhost:9200/tweets/tweet/426674590560305150 -d '{

    "created_at": "Fri Jan 24 11:15:24 +0000 2014",
    "id": 426674590560305150,
    "text": "Prepping up for my #elasticsearch talk
    this afternoon at the UvA : http://t.co/rqhBI5zys0",
    "user": {
    "name": “Boaz Leskes",
    "screen_name": "bleskes",
    }
    }'

    View Slide

  18. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Shard 1 Shard 2 Shard 3

    View Slide

  19. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Shard 1 Shard 2 Shard 3

    View Slide

  20. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Shard 1 Shard 2 Shard 3

    View Slide

  21. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Shard 1 Shard 2 Shard 3

    View Slide

  22. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Reminds of a tile at my aunt’s house
    Today is the tomorrow
    we were all afraid of
    yesterday….

    View Slide

  23. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    August
    1 2 3

    View Slide

  24. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    August
    1 2 3
    September
    1 2 3

    View Slide

  25. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    August
    1 2 3
    September
    1 2 3
    October
    1 2 3

    View Slide

  26. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    one little tweak…
    # curl -XPUT localhost:9200/tweets_201310/tweet/426674590560305150
    -d '{

    "created_at": "Fri Jan 24 11:15:24 +0000 2014",
    "id": 426674590560305150,
    "text": "Prepping up for my #elasticsearch talk
    this afternoon at the UvA : http://t.co/rqhBI5zys0",
    "user": {
    "name": “Boaz Leskes",
    "screen_name": "bleskes",
    }
    }'

    View Slide

  27. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

    View Slide

  28. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Another fact
    index is the basic unit of configuration

    View Slide

  29. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    index templates
    curl -XPUT localhost:9200/_template/twitter -d '
    {
    "template" : “twitter_*",
    "settings" : {
    "number_of_shards" : 4,
    "number_of_replicas" : 1
    }
    }'

    View Slide

  30. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    older data
    # elasticsearch.yml
    !
    node.disk: spinning_disks
    curl -XPUT localhost:9200/twitter_2012*/_settings -d '{
    "index.routing.allocation.include.disk" : “spinning_disks”,
    “index.routing.allocation.exclude.disk" : "ssd"
    }'

    View Slide

  31. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    older data
    curl -XPOST localhost:9200/twitter_2012*/_optimize
    !
    curl -XDELETE localhost:9200/twitter_201201/
    !
    curl -XPOST localhost:9200/twitter_201201/_close

    View Slide

  32. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    aliases
    curl -XPUT localhost:9200/_aliases -d ‘{
    "add": {
    "index": "twitter_201311", "alias": "last_2_months"
    },
    "remove": {
    "index": "twitter_201309", "alias": "last_2_months"
    }
    }'

    View Slide

  33. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Implications
    • Use indices to manage data as it scales

    • Use aliases to efficiently point your searches at
    the relevant shards

    View Slide

  34. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    One More Thing..
    Time is just a (strictly) monotonic function

    Primary keys are just as good

    View Slide

  35. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    thank you!
    http://elasticsearch.com/support
    @elasticsearch , @bleskes
    http://elasticsearch.org/guide

    View Slide