Share
Embed
Oslo Elasticsearch meetup
The talk about new and upcoming new elasticsearch features presented at the Oslo elasticsearch meetup.
Martijn van Groningen
September 26, 2013
Transcript
29 slides
-
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission
is strictly prohibited Oslo Elasticsearch Meetup Martijn van Groningen @mvgroningen Friday, September 27, 13 -
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission
is strictly prohibited Recent improvements • 0.90.5 Added windows service • 0.90.4 random ordering Improved shard allocation Highlights in phrase suggestions Function scoring • 0.90.3 A new completing suggester Friday, September 27, 13 -
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission
is strictly prohibited Improved shard allocation a1 a1 a2 a2 a3 a4 a4 a3 b1 b2 b3 b4 b1 b2 b3 b4 Node 1 Node 2 New index ‘b’ - shards to be allocated: Friday, September 27, 13 -
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission
is strictly prohibited Improved shard allocation • Shard allocation based on free disk space. Disallow shard allocation a node Reallocate shards if free disk space gets low • Implemented as an allocation decider. • Options: cluster.routing.allocation.disk.threshold_enabled cluster.routing.allocation.disk.watermark.low cluster.routing.allocation.disk.watermark.high Friday, September 27, 13 -
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission
is strictly prohibited Function score query • To customize the score for a matching document via functions. Support for decay, script and random functions boost factor & filters. • The function_score replaces existing custom scoring queries: custom_boost_factor custom_score custom_filters_score Friday, September 27, 13 -
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission
is strictly prohibited Function score query curl -XGET 'localhost:9200/books/_search' -d '{ "query" : { "function_score" : { " "query" : { "match" : { "summary" : { "query" : "quick brown fox" } } }, "functions" : [ { "script_score" : "_score * doc['field'].value" } ] } } }' Custom score function Inner query Friday, September 27, 13 -
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission
is strictly prohibited Function score query • The function_score query have built-in decay functions. The score decays the further is gets from the origin. • There are several type of decay functions: gauss, exponential and linear • Example usage of decay functions: Restaurant near a certain location News articles near a specific date Friday, September 27, 13 -
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission
is strictly prohibited Function score query curl -XGET 'localhost:9200/news/_search' -d '{ "query" : { "function_score" : { " "query" : { "match" : { "body" : { "query" : "german elections" } } }, "functions" : [{ "exp" : { "origin" : "2013-09-24", "scale" : "30d" } }] } } }' Friday, September 27, 13 -
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission
is strictly prohibited Function score query - random curl -XGET 'localhost:9200/products/_search' -d '{ "query" : { "function_score" : { " "query" : { "match" : { "description" : { "query" : "42 inch led tv" } } }, "functions" : [{ "random_score" : { "seed" : 123456 } }] } } }' Friday, September 27, 13 -
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission
is strictly prohibited Completion suggester • The completion suggester is designed for super fast as you type suggestions. Aimed for sub millisecond response times. • Good suggester reduces the number of search requests. • Currently in total three suggesters: term, phrase and completion Friday, September 27, 13 -
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission
is strictly prohibited Completion suggester • Super fast because of in-memory FST Finite State Transducer • FSTs in CS are stored on disk and are NRT. Friday, September 27, 13 -
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission
is strictly prohibited Completion suggester • The completion suggester is configured via a completion field in the mapping. Analyzer, whether to store payloads • During index time define suggest phrases. Multiple inputs, unified output and weighting. • Use the suggest api! Friday, September 27, 13 -
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission
is strictly prohibited Completion suggester • Completion suggester can pinpoint users to the right document. • Enhancements added in 0.90.4: Fuzzy terms (edit distance based) Suggest statistics • A must read: http://www.elasticsearch.org/blog/you-complete-me/ Friday, September 27, 13 -
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission
is strictly prohibited Completion suggester • Indexing suggestions: curl -XPUT localhost:9200/hotels/hotel/2 -d '{ "name" : "Hotel Monaco", "city" : "Munich", "name_suggest" : { "input" : [ "Monaco Munich", "Hotel Monaco" ], "output": "Hotel Monaco", "weight": 10, "payload" : { "hotel_id": "3" } } }' Input values Unified output Weight Payload Friday, September 27, 13 -
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission
is strictly prohibited Completion suggester curl -X POST localhost:9200/hotels/_suggest -d '{ "hotels" : { "text" : "m", "completion" : { "field" : "name_suggest" } } }' ... "hotels" : [ { "text" : "m", "offset" : 0, "length" : 1, "options" : [ { "text" : "Mercure Hotel Munich", "payload" : { "hotel_id": "3" }, "score" : 1.0 } ] } ] Suggest api: Suggest response part: Friday, September 27, 13 -
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission
is strictly prohibited Upcoming new features • Snapshot and restore api • Distributed percolator • Aggregations • Lucene doc values support. Friday, September 27, 13 -
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission
is strictly prohibited Snapshot and restore api • Currently there is no snapshot / restore feature. There is a backup / restore workaround. • Snapshot api will allow to create repositories. Repository in a shared fs or s3. • A cluster can be snapshotted into a repository. Friday, September 27, 13 -
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission
is strictly prohibited Snapshot and restore api • A repository can hold multiple snapshots. A new snapshot will reuse data of previous snapshots. • The snapshot api will also allow specific indices to be snapshotted. • The restore api can restore a snapshot into a cluster. Friday, September 27, 13 -
Distributed percolator coffee OR pots Title : Coffee percolator Body
: A coffee percolator is a type of pot used to brew coffee by continually cycling the boiling or nearly-boiling brew through ... Title : Coffee percolator Body : A coffee percolator is a type of pot used to brew coffee by continually cycling the boiling or nearly-boiling brew through ... Title : Coffee percolator Body : A coffee percolator is a type of pot used to brew coffee by continually cycling the boiling or nearly-boiling brew through ... Title : Coffee percolator Body : A coffee percolator is a type of pot used to brew coffee by continually cycling the boiling or nearly-boiling brew through ... 1. Coffee percolator 2. Plain old telephone service (pots) ... Hits Query Documents Friday, September 27, 13 -
Distributed percolator coffee OR pots Title : Coffee percolator Body
: A coffee percolator is a type of pot used to brew coffee by continually cycling the boiling or nearly-boiling brew through ... 1. Coffee OR pots 2. boiling AND brew ... Matches Document Queries boiling AND brew other AND stuff Friday, September 27, 13 -
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission
is strictly prohibited Distributed percolator • Workflow: Store and index Elasticsearch queries. Searching based on documents. • Use cases: Alerting and monitoring Classifying data Advertisement feed Friday, September 27, 13 -
Distributed percolator • Current percolator doesn’t scale. (<= 0.90.x) The
queries are stored in a single shard _percolator system index that is auto replicated to all data nodes. • The distributed percolator allows queries to be stored in any index. (>= 1.0.x) Queries are stored under the _percolator type that any index can have. • The percolate api is now also multi tenant. Percolate across multiple indices and aliases Friday, September 27, 13 -
Percolator - alerting use case curl -XPUT 'localhost:9200/prices/_percolator/user-1' -d '{
"query" : { "bool" : [ { "range" : { "product.price" : { "lte" : 500 } } }, { "match" : { "product.name" : "my led tv" } } ] } }' Indexing a query as a document to an index: Friday, September 27, 13 -
Percolator - alerting use case curl -XPUT 'localhost:9200/products/product/1' -d '{
"name" : "my led tv", "price" : 499, ... }' Index a new product: Index response: { "ok" : true, "_index" : "products", "_type" : "product", "_id" : "1", "_version" : 1 } Friday, September 27, 13 -
Percolator - alerting use case Matching queries as percolate response:
{ "took" : 19, "_shards" : { "total" : 2, "successful" : 2, "failed" : 0 }, "count" : 4, "matches" : [ { "_index" : "prices", "_id" : "user-1" }, ... ] } curl -XGET 'localhost:9200/products/product/1/_percolate' Percolate request: Friday, September 27, 13 -
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission
is strictly prohibited Aggregations • Facets are great! Get statistics over the complete query result set. • There are several facet types: terms, range, stats, histogram, geo_distance terms_stats • But there is often there is a need to combine different facets. So we need something more! Friday, September 27, 13 -
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission
is strictly prohibited Aggregations • Aggregation concepts: Aggregator & Aggregation • Two main types of aggregators: Bucket • terms, (date) histrogram, range, geo distance, filter Calculation • count, min, max, sum, avg, stats & extended stats • Bucket aggregators allow nesting of other aggregators. Friday, September 27, 13 -
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission
is strictly prohibited Aggregations curl -XPUT 'localhost:9200/idx/city/1' -d '{ "state" : "Florida" "city" : "Miami", ... // Other metadata like population }' curl -XGET 'localhost:9200/idx/_search' -d '{ "facet" : { "states" : { " "terms" : { "field" : "state" } } } }' Example doc: Classic facet example: Either state or city, so no hierarchical facet... Friday, September 27, 13 -
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission
is strictly prohibited Aggregations • Aggregation example: curl -XGET 'localhost:9200/idx/_search' -d '{ "aggs" : { "states" : { " "terms" : { "field" : "state" }, "aggs" : { "cities" : { "terms" : { "field" : "city" } } } } } }' First bucket level Second bucket level Friday, September 27, 13