explore your data with elasticsearch

A talk given at RuPy 2013

Transcript

1. explore your data with elasticsearch Honza Král @honzakral
2. REST HTTP JSON distributed search analytics real-time scalable open-source Lucene …
3. setup # wget elasticsearch.tar.gz # tar xzvf elasticsearch.tar.gz # bin/elasticsearch # curl localhost:9200
4. documents # curl -XPUT localhost:9200/stack/question/42 -d '{ “some”: “json” }' # curl -XGET localhost:9200/stack/question/42 # curl -XDELETE localhost:9200/stack/question/42
5. search # curl -XGET localhost:9200/_search?q=meetup # curl -XGET localhost:9200/_search -d '{ “query”: { “query_string”: { “query”: “meetup AND title:python” } } }'
6. queries & filters # curl -XGET localhost:9200/stack/_search -d '{ “query”: { “filtered”: { “query”: { “bool”: { “must”: [ {"multi_match": { "fields": ["title^10", "body"] "query": "python" }}, ], “must_not”: [ {“match”: {“title”: “php”} ] } }, “filter”: { “range”:{"creation_date":{"from":"2013-01-01"}} } } } }'
7. filter when you can, query if you must
8. Let us pray to the demo gods!
9. mix & match curl -XGET http://localhost:9200/dba.stackexchange.com/question/_search -d ' { "query": { "custom_score": { "query": { "filtered": { "query": { "bool": { "must": [ {"multi_match": {"fields": ["title^10", "body"], "query": "mysql"}}, { "has_child": { "child_type": "answer", "query": {"match": {"body": "nosql"}} } } ], "must_not": [ {"multi_match": {"fields": ["title", "body"], "query": "nosql"}} ] } }, "filter": { "range": {"creation_date": {"from": "2012-01-01"}} } } }, "script": "(_score + 1) * doc[\"rating\"].value" } }, "fields": ["title", "rating", "creation_date"], "highlight": { "fields": { "title": {"fragment_size" : 50}, "body": {"fragment_size" : 50} } }, "facets": { "tags": { "terms": {"field": "tags"} }, "frequency": { "date_histogram": {"field": "creation_date", "interval": "month"} } } }' Find questions that • Were asked last year • Contain “mysql” in title or body • Don't contain “nosql” • Have answer that has “nosql” in title or body • Include question rating into score calculation • Highlight matches in html • Aggregate over time and tags • ….
10. percolator # curl -XPUT localhost:9200/_percolator/conf/meet -d '{ "query" : { "term" : { "tile" : "meetup" } } }' # curl -XPUT localhost:9200/conf/event/_percolate -d '{ “doc”: { “title”: “SF Python Meetup” } }'
11. suggester – auto-complete # curl -X POST 'localhost:9200/music/_suggest' -d '{ "song-suggest" : { "text" : "n", "completion" : { "field" : "song_suggest" } } }' { "text" : "Nirvana - Nevermind", "score" : 34.0, "payload" : {"artist_id":2321} }
12. suggester – did you mean? # curl -XPOST 'localhost:9200/_search' -d { "suggest" : { "text" : "Johny Walker", "simple_phrase" : { "phrase" : { ... MAGIC HERE ... "direct_generator" : [ { "field" : "body" } ] } } } }' { "text" : "Johnnie Walker", "score" : 0.314295 }
13. Is it web scale?
14. YES!
15. distributed model
16. Thanks!