Staying Ahead of Time

elasticsearch and time based data

a talk given by Boaz Leskes at the Search Engines Amsterdam seminar

Transcript

1. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited elasticsearch and time based data Staying Ahead of Time Boaz Leskes @bleskes
2. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited indices,types and other animals Basics
3. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited A document {
 "created_at": "Fri Jan 24 11:15:24 +0000 2014", "id": 426674590560305150, "text": "Prepping up for my #elasticsearch talk this afternoon at the UvA : http://t.co/rqhBI5zys0", "user": { "name": “Boaz Leskes", "screen_name": "bleskes", } }
4. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited A type {
 "created_at": "Fri Jan 24 11:15:24 +0000 2014", "id": 426674590560305150, "text": "Prepping up for my #elasticsearch talk this afternoon at the UvA : http://t.co/rqhBI5zys0", "user": { "name": “Boaz Leskes", "screen_name": "bleskes", } } = docs with similar data/structure {
 "created_at": "Thu Jan 23 18:27:23 +0000 2014", "id": 426420915698544640, "text": "Elasticsearch es una maravilla !!!!", "user": { "name": "Abel Coronado", "screen_name": "abxda", } }
5. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited An index = a collection of types {
 "created_at": "Thu Jan 23 "id": 426420915698544640, "text": "Elasticsearch Esc "user": { "name": "Abel Coronado "screen_name": "abxda" } } {
 "id": 19726002, "name": "Abel Coronado "screen_name": "abxda" "location":"Aguascalientes" "followers_count":871 "friends_count":1794 "listed_count":38 }
6. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Sharding index
7. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Sharding index shard shard shard shard
8. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Sharding index shard shard shard shard node node
9. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Sharding index node node shard shard shard shard
10. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Sharding index node node shard shard shard shard node node copy copy copy copy
11. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Sharding node node copy copy node node shard shard copy copy shard shard
12. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Sharding node node copy copy node node node node node node shard shard copy copy shard shard
13. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Sharding node node copy copy node node node node node node shard shard shard copy copy shard
14. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Important fact for later indexing & searching is done on shards, not indices
15. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited time based data To the subject at hand
16. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
17. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited easy to get, easy to index # curl -XPUT localhost:9200/tweets/tweet/426674590560305150 -d '{
 "created_at": "Fri Jan 24 11:15:24 +0000 2014", "id": 426674590560305150, "text": "Prepping up for my #elasticsearch talk this afternoon at the UvA : http://t.co/rqhBI5zys0", "user": { "name": “Boaz Leskes", "screen_name": "bleskes", } }'
18. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Shard 1 Shard 2 Shard 3
19. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Shard 1 Shard 2 Shard 3
20. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Shard 1 Shard 2 Shard 3
21. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Shard 1 Shard 2 Shard 3
22. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Reminds of a tile at my aunt’s house Today is the tomorrow we were all afraid of yesterday….
23. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited August 1 2 3
24. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited August 1 2 3 September 1 2 3
25. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited August 1 2 3 September 1 2 3 October 1 2 3
26. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited one little tweak… # curl -XPUT localhost:9200/tweets_201310/tweet/426674590560305150 -d '{
 "created_at": "Fri Jan 24 11:15:24 +0000 2014", "id": 426674590560305150, "text": "Prepping up for my #elasticsearch talk this afternoon at the UvA : http://t.co/rqhBI5zys0", "user": { "name": “Boaz Leskes", "screen_name": "bleskes", } }'
27. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
28. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Another fact index is the basic unit of configuration
29. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited index templates curl -XPUT localhost:9200/_template/twitter -d ' { "template" : “twitter_*", "settings" : { "number_of_shards" : 4, "number_of_replicas" : 1 } }'
30. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited older data # elasticsearch.yml ! node.disk: spinning_disks curl -XPUT localhost:9200/twitter_2012*/_settings -d '{ "index.routing.allocation.include.disk" : “spinning_disks”, “index.routing.allocation.exclude.disk" : "ssd" }'
31. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited older data curl -XPOST localhost:9200/twitter_2012*/_optimize ! curl -XDELETE localhost:9200/twitter_201201/ ! curl -XPOST localhost:9200/twitter_201201/_close
32. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited aliases curl -XPUT localhost:9200/_aliases -d ‘{ "add": { "index": "twitter_201311", "alias": "last_2_months" }, "remove": { "index": "twitter_201309", "alias": "last_2_months" } }'
33. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Implications • Use indices to manage data as it scales • Use aliases to efficiently point your searches at the relevant shards
34. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited One More Thing.. Time is just a (strictly) monotonic function Primary keys are just as good
35. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited thank you! http://elasticsearch.com/support @elasticsearch , @bleskes http://elasticsearch.org/guide