Oslo Elasticsearch meetup

The talk about new and upcoming new elasticsearch features presented at the Oslo elasticsearch meetup.

Transcript

1. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Oslo Elasticsearch Meetup Martijn van Groningen @mvgroningen Friday, September 27, 13
2. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Recent improvements • 0.90.5 Added windows service • 0.90.4 random ordering Improved shard allocation Highlights in phrase suggestions Function scoring • 0.90.3 A new completing suggester Friday, September 27, 13
3. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Improved shard allocation a1 a1 a2 a2 a3 a4 a4 a3 b1 b2 b3 b4 b1 b2 b3 b4 Node 1 Node 2 New index ‘b’ - shards to be allocated: Friday, September 27, 13
4. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Improved shard allocation • Shard allocation based on free disk space. Disallow shard allocation a node Reallocate shards if free disk space gets low • Implemented as an allocation decider. • Options: cluster.routing.allocation.disk.threshold_enabled cluster.routing.allocation.disk.watermark.low cluster.routing.allocation.disk.watermark.high Friday, September 27, 13
5. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Function score query • To customize the score for a matching document via functions. Support for decay, script and random functions boost factor & filters. • The function_score replaces existing custom scoring queries: custom_boost_factor custom_score custom_filters_score Friday, September 27, 13
6. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Function score query curl -XGET 'localhost:9200/books/_search' -d '{ "query" : { "function_score" : { " "query" : { "match" : { "summary" : { "query" : "quick brown fox" } } }, "functions" : [ { "script_score" : "_score * doc['field'].value" } ] } } }' Custom score function Inner query Friday, September 27, 13
7. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Function score query • The function_score query have built-in decay functions. The score decays the further is gets from the origin. • There are several type of decay functions: gauss, exponential and linear • Example usage of decay functions: Restaurant near a certain location News articles near a specific date Friday, September 27, 13
8. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Function score query curl -XGET 'localhost:9200/news/_search' -d '{ "query" : { "function_score" : { " "query" : { "match" : { "body" : { "query" : "german elections" } } }, "functions" : [{ "exp" : { "origin" : "2013-09-24", "scale" : "30d" } }] } } }' Friday, September 27, 13
9. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Function score query - random curl -XGET 'localhost:9200/products/_search' -d '{ "query" : { "function_score" : { " "query" : { "match" : { "description" : { "query" : "42 inch led tv" } } }, "functions" : [{ "random_score" : { "seed" : 123456 } }] } } }' Friday, September 27, 13
10. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Completion suggester • The completion suggester is designed for super fast as you type suggestions. Aimed for sub millisecond response times. • Good suggester reduces the number of search requests. • Currently in total three suggesters: term, phrase and completion Friday, September 27, 13
11. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Completion suggester • Super fast because of in-memory FST Finite State Transducer • FSTs in CS are stored on disk and are NRT. Friday, September 27, 13
12. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Completion suggester • The completion suggester is configured via a completion field in the mapping. Analyzer, whether to store payloads • During index time define suggest phrases. Multiple inputs, unified output and weighting. • Use the suggest api! Friday, September 27, 13
13. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Completion suggester • Completion suggester can pinpoint users to the right document. • Enhancements added in 0.90.4: Fuzzy terms (edit distance based) Suggest statistics • A must read: http://www.elasticsearch.org/blog/you-complete-me/ Friday, September 27, 13
14. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Completion suggester • Indexing suggestions: curl -XPUT localhost:9200/hotels/hotel/2 -d '{ "name" : "Hotel Monaco", "city" : "Munich", "name_suggest" : { "input" : [ "Monaco Munich", "Hotel Monaco" ], "output": "Hotel Monaco", "weight": 10, "payload" : { "hotel_id": "3" } } }' Input values Unified output Weight Payload Friday, September 27, 13
15. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Completion suggester curl -X POST localhost:9200/hotels/_suggest -d '{ "hotels" : { "text" : "m", "completion" : { "field" : "name_suggest" } } }' ... "hotels" : [ { "text" : "m", "offset" : 0, "length" : 1, "options" : [ { "text" : "Mercure Hotel Munich", "payload" : { "hotel_id": "3" }, "score" : 1.0 } ] } ] Suggest api: Suggest response part: Friday, September 27, 13
16. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Upcoming new features • Snapshot and restore api • Distributed percolator • Aggregations • Lucene doc values support. Friday, September 27, 13
17. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Snapshot and restore api • Currently there is no snapshot / restore feature. There is a backup / restore workaround. • Snapshot api will allow to create repositories. Repository in a shared fs or s3. • A cluster can be snapshotted into a repository. Friday, September 27, 13
18. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Snapshot and restore api • A repository can hold multiple snapshots. A new snapshot will reuse data of previous snapshots. • The snapshot api will also allow specific indices to be snapshotted. • The restore api can restore a snapshot into a cluster. Friday, September 27, 13
19. Distributed percolator coffee OR pots Title : Coffee percolator Body : A coffee percolator is a type of pot used to brew coffee by continually cycling the boiling or nearly-boiling brew through ... Title : Coffee percolator Body : A coffee percolator is a type of pot used to brew coffee by continually cycling the boiling or nearly-boiling brew through ... Title : Coffee percolator Body : A coffee percolator is a type of pot used to brew coffee by continually cycling the boiling or nearly-boiling brew through ... Title : Coffee percolator Body : A coffee percolator is a type of pot used to brew coffee by continually cycling the boiling or nearly-boiling brew through ... 1. Coffee percolator 2. Plain old telephone service (pots) ... Hits Query Documents Friday, September 27, 13
20. Distributed percolator coffee OR pots Title : Coffee percolator Body : A coffee percolator is a type of pot used to brew coffee by continually cycling the boiling or nearly-boiling brew through ... 1. Coffee OR pots 2. boiling AND brew ... Matches Document Queries boiling AND brew other AND stuff Friday, September 27, 13
21. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Distributed percolator • Workflow: Store and index Elasticsearch queries. Searching based on documents. • Use cases: Alerting and monitoring Classifying data Advertisement feed Friday, September 27, 13
22. Distributed percolator • Current percolator doesn’t scale. (<= 0.90.x) The queries are stored in a single shard _percolator system index that is auto replicated to all data nodes. • The distributed percolator allows queries to be stored in any index. (>= 1.0.x) Queries are stored under the _percolator type that any index can have. • The percolate api is now also multi tenant. Percolate across multiple indices and aliases Friday, September 27, 13
23. Percolator - alerting use case curl -XPUT 'localhost:9200/prices/_percolator/user-1' -d '{ "query" : { "bool" : [ { "range" : { "product.price" : { "lte" : 500 } } }, { "match" : { "product.name" : "my led tv" } } ] } }' Indexing a query as a document to an index: Friday, September 27, 13
24. Percolator - alerting use case curl -XPUT 'localhost:9200/products/product/1' -d '{ "name" : "my led tv", "price" : 499, ... }' Index a new product: Index response: { "ok" : true, "_index" : "products", "_type" : "product", "_id" : "1", "_version" : 1 } Friday, September 27, 13
25. Percolator - alerting use case Matching queries as percolate response: { "took" : 19, "_shards" : { "total" : 2, "successful" : 2, "failed" : 0 }, "count" : 4, "matches" : [ { "_index" : "prices", "_id" : "user-1" }, ... ] } curl -XGET 'localhost:9200/products/product/1/_percolate' Percolate request: Friday, September 27, 13
26. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Aggregations • Facets are great! Get statistics over the complete query result set. • There are several facet types: terms, range, stats, histogram, geo_distance terms_stats • But there is often there is a need to combine different facets. So we need something more! Friday, September 27, 13
27. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Aggregations • Aggregation concepts: Aggregator & Aggregation • Two main types of aggregators: Bucket • terms, (date) histrogram, range, geo distance, filter Calculation • count, min, max, sum, avg, stats & extended stats • Bucket aggregators allow nesting of other aggregators. Friday, September 27, 13
28. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Aggregations curl -XPUT 'localhost:9200/idx/city/1' -d '{ "state" : "Florida" "city" : "Miami", ... // Other metadata like population }' curl -XGET 'localhost:9200/idx/_search' -d '{ "facet" : { "states" : { " "terms" : { "field" : "state" } } } }' Example doc: Classic facet example: Either state or city, so no hierarchical facet... Friday, September 27, 13
29. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Aggregations • Aggregation example: curl -XGET 'localhost:9200/idx/_search' -d '{ "aggs" : { "states" : { " "terms" : { "field" : "state" }, "aggs" : { "cities" : { "terms" : { "field" : "city" } } } } } }' First bucket level Second bucket level Friday, September 27, 13