Elasticsearch 1.0

Slides from Fosdem 2014 talk by Honza Král

Transcript

1. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Honza Král @honzakral Elasticsearch 1.0
2. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearch
3. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited REST HTTP JSON distributed search analytics real-time scalable open-source Lucene …
4. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited setup $ wget elasticsearch.tar.gz! $ tar xzvf elasticsearch.tar.gz! $ bin/elasticsearch! $ curl localhost:9200!
5. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited documents $ curl -XPUT localhost:9200/stack/question/42 -d ‘{! “some”: “json”! }’ $ curl -XGET localhost:9200/stack/question/42 $ curl -XDELETE localhost:9200/stack/question/42
6. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited search $ curl localhost:9200/_search?q=conference $ curl localhost:9200/_search -d ‘{! “query”: {! “query_string”: {! “query”:“conference AND topic:search”! }! }! }’
7. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited query DSL $ curl localhost:9200/meetups/_search -d '{! “query”: {! “filtered”: {! “query”: {! “bool”: {! “must”: [! {"multi_match": {! "fields": ["title^10", "description"]! "query": "search"! }},! ],! “should”: [! {“match”: {“description”: “pizza”}! ]! }! },! “filter”: {! “range”:{"creation_date":{"from":"2014-02-01"}}! } } } }'!
8. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited filter when you can, query if you must
9. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited query DSL, take II $ curl localhost:9200/dba.stackexchange.com/question/_search -d '! {! "query": {! "custom_score": {! "query": {! "filtered": {! "query": {! "bool": {! "must": [! {"multi_match": {"fields": ["title^10", "body"], "query": "mysql"}},! {! "has_child": {! "child_type": "answer",! "query": {"match": {"body": "nosql"}}! }! }! ],! "must_not": [! {"multi_match": {"fields": ["title", "body"], "query": "nosql"}}! ]! }! },! "filter": {! "range": {"creation_date": {"from": "2013-01-01"}}! }! }! },! "script": "(_score + 1) * doc[\"rating\"].value"! }! },! "fields": ["title", "rating", "creation_date"],! "highlight": { ! "fields": { ! "title": {"fragment_size" : 50},! "body": {"fragment_size" : 50}! } ! },! "aggs": {! "tags": {! "terms": {"field": "tags"}! }, ! "frequency": {! "date_histogram": {"field": "creation_date", "interval": "month"}! }! }! }'! ! Find questions that! • Were asked last year! • Contain “mysql” in title or body! • Don't contain “nosql”! • Have answer that has “nosql” in title or body! • Include question rating into score calculation! • Highlight matches in html! • Aggregate over time and tags! • ….
10. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited 1.0
11. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited aggregations
12. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited facets ? terms + statistical = terms_stats! terms + range + avg = aggregations
13. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited bucket aggregators global filter missing terms range date range ip range histogram date histogram geo distance nested
14. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited aggregators count stats extended stats avg min max sum
15. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited snapshot and restore
16. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited backup in 0.90 1. disable flush 2. find all primary shard location (optional) 3. copy files from primary shards (rsync) 4. enable flush
17. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited backup in v1.0 $ curl -XPUT localhost:9200/_snapshot/prod_s3/snapshot_20131010 snapshot name repository
18. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited snapshot and restore • incremental backup • rollback to previous index state • index copy within a cluster • copying index from one cluster to another
19. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited register repository $ curl -XPUT "localhost:9200/_snapshot/local" -d '{! "type": "fs", ! "settings": {! "location":"/tmp/es-backup"! }! }' location repository repository type
20. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited start snapshot $ curl -XPUT localhost:9200/_snapshot/local/backup_1 -d '{! “indices":"*,-test*"! }' snapshot name repository index list (optional)
21. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited restore in 0.90 1. close the index (shutdown the cluster) 2. find all existing index shards 3. replace all index shards with data from backup 4. open the index (start the cluster)
22. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited restore in 1.0 $ curl -XPOST “localhost:9200/my_*/_close" close all indices that start with my_ $ curl -XPOST localhost:9200/_snapshot/local/backup_1/_restore -d '{! "indices":"my_*"! }' repository name index list snapshot name
23. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited percolator search, reversed
24. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited registering percolator $ curl -XPUT “localhost:9200/some_index/.percolator/es-tweets” -d ‘{! “query”: {! “match”: { “body”: “elasticsearch” }! },! “alert_type”: “mention”! }’! reserved .percolator type query id any index with as many shards as you need
25. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited percolation $ curl -XGET “localhost:9200/twitter/tweet/_percolate” -d ‘{! “doc”: {! “text”: “#elasticsearch is awesome”! “nick”: “@everybody”! “name”: “Everybody Everywhere”! “date”: “2013-11-03” ! }! }’ target index percolation end point document to be percolated {! “ok”: true! “matches”: [“es-tweets”]! } matching queries
26. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited changes VS 0.90? _percolate index VS .percolator type ! sequential VS paralel execution
27. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited =^.^= Image Source: Wikipedia,
28. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited /_cat/* api It’s because humans suck at reading JSON (and YAML)
29. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Which one is the master? (v0.90) $ curl "localhost:9200/_cluster/state? pretty&filter_metadata=true&filter_routing_table=true"! {! "cluster_name" : "elasticsearch",! "master_node" : "GNf0hEXlTfaBvQXKBF300A",! "blocks" : { },! "nodes" : {! "ObdRqLHGQ6CMI5rOEstA5A" : {! "name" : "Triton",! "transport_address" : “inet[/10.0.1.11:9300]”,! "attributes" : { }! },! "4C7pKbfhTvu0slcSy_G4_w" : {! "name" : "Kid Colt",! "transport_address" : "inet[/10.0.1.12:9300]",! "attributes" : { }! },! "GNf0hEXlTfaBvQXKBF300A" : {! "name" : "Lang, Steven",! "transport_address" : "inet[/10.0.1.13:9300]",! "attributes" : { }! }! }! } master IP master ID
30. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Which one is the master? (v1.0) $ curl localhost:9200/_cat/master?v id host ip node GNf0hEXlTfaBvQXKBF300A MyHost.local 10.0.1.13 Honza
31. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited /_cat/count $ curl localhost:9200/_cat/count?v! epoch timestamp count! 1383501234301 12:53:54 3344067 count
32. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited _cat/* api /_cat/allocation /_cat/count /_cat/health /_cat/master /_cat/nodes /_cat/recovery /_cat/shards /_cat/indices
33. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited thank you! Honza Král twitter: @honzakral email: honza.kral@elasticsearch.com ! ! ! ! ! ! ! ! • Support: http://elasticsearch.com/support • Training: http://training.elasticsearch.com/ • We are hiring: http://elasticsearch.com/about/jobs/