Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Elasticsearch 1.0

Elasticsearch 1.0

Slides from Fosdem 2014 talk by Honza Král

Elasticsearch Inc

February 02, 2014
Tweet

More Decks by Elasticsearch Inc

Other Decks in Technology

Transcript

  1. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited Honza Král @honzakral Elasticsearch 1.0
  2. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited Elasticsearch
  3. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited REST HTTP JSON distributed search analytics real-time scalable open-source Lucene …
  4. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited setup $ wget elasticsearch.tar.gz! $ tar xzvf elasticsearch.tar.gz! $ bin/elasticsearch! $ curl localhost:9200!
  5. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited documents $ curl -XPUT localhost:9200/stack/question/42 -d ‘{! “some”: “json”! }’ $ curl -XGET localhost:9200/stack/question/42 $ curl -XDELETE localhost:9200/stack/question/42
  6. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited search $ curl localhost:9200/_search?q=conference $ curl localhost:9200/_search -d ‘{! “query”: {! “query_string”: {! “query”:“conference AND topic:search”! }! }! }’
  7. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited query DSL $ curl localhost:9200/meetups/_search -d '{! “query”: {! “filtered”: {! “query”: {! “bool”: {! “must”: [! {"multi_match": {! "fields": ["title^10", "description"]! "query": "search"! }},! ],! “should”: [! {“match”: {“description”: “pizza”}! ]! }! },! “filter”: {! “range”:{"creation_date":{"from":"2014-02-01"}}! } } } }'!
  8. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited filter when you can, query if you must
  9. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited query DSL, take II $ curl localhost:9200/dba.stackexchange.com/question/_search -d '! {! "query": {! "custom_score": {! "query": {! "filtered": {! "query": {! "bool": {! "must": [! {"multi_match": {"fields": ["title^10", "body"], "query": "mysql"}},! {! "has_child": {! "child_type": "answer",! "query": {"match": {"body": "nosql"}}! }! }! ],! "must_not": [! {"multi_match": {"fields": ["title", "body"], "query": "nosql"}}! ]! }! },! "filter": {! "range": {"creation_date": {"from": "2013-01-01"}}! }! }! },! "script": "(_score + 1) * doc[\"rating\"].value"! }! },! "fields": ["title", "rating", "creation_date"],! "highlight": { ! "fields": { ! "title": {"fragment_size" : 50},! "body": {"fragment_size" : 50}! } ! },! "aggs": {! "tags": {! "terms": {"field": "tags"}! }, ! "frequency": {! "date_histogram": {"field": "creation_date", "interval": "month"}! }! }! }'! ! Find questions that! • Were asked last year! • Contain “mysql” in title or body! • Don't contain “nosql”! • Have answer that has “nosql” in title or body! • Include question rating into score calculation! • Highlight matches in html! • Aggregate over time and tags! • ….
  10. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited 1.0
  11. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited aggregations
  12. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited facets ? terms + statistical = terms_stats! terms + range + avg = aggregations
  13. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited bucket aggregators global filter missing terms range date range ip range histogram date histogram geo distance nested
  14. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited aggregators count stats extended stats avg min max sum
  15. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited snapshot and restore
  16. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited backup in 0.90 1. disable flush 2. find all primary shard location (optional) 3. copy files from primary shards (rsync) 4. enable flush
  17. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited backup in v1.0 $ curl -XPUT localhost:9200/_snapshot/prod_s3/snapshot_20131010 snapshot name repository
  18. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited snapshot and restore • incremental backup • rollback to previous index state • index copy within a cluster • copying index from one cluster to another
  19. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited register repository $ curl -XPUT "localhost:9200/_snapshot/local" -d '{! "type": "fs", ! "settings": {! "location":"/tmp/es-backup"! }! }' location repository repository type
  20. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited start snapshot $ curl -XPUT localhost:9200/_snapshot/local/backup_1 -d '{! “indices":"*,-test*"! }' snapshot name repository index list (optional)
  21. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited restore in 0.90 1. close the index (shutdown the cluster) 2. find all existing index shards 3. replace all index shards with data from backup 4. open the index (start the cluster)
  22. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited restore in 1.0 $ curl -XPOST “localhost:9200/my_*/_close" close all indices that start with my_ $ curl -XPOST localhost:9200/_snapshot/local/backup_1/_restore -d '{! "indices":"my_*"! }' repository name index list snapshot name
  23. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited percolator search, reversed
  24. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited registering percolator $ curl -XPUT “localhost:9200/some_index/.percolator/es-tweets” -d ‘{! “query”: {! “match”: { “body”: “elasticsearch” }! },! “alert_type”: “mention”! }’! reserved .percolator type query id any index with as many shards as you need
  25. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited percolation $ curl -XGET “localhost:9200/twitter/tweet/_percolate” -d ‘{! “doc”: {! “text”: “#elasticsearch is awesome”! “nick”: “@everybody”! “name”: “Everybody Everywhere”! “date”: “2013-11-03” ! }! }’ target index percolation end point document to be percolated {! “ok”: true! “matches”: [“es-tweets”]! } matching queries
  26. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited changes VS 0.90? _percolate index VS .percolator type ! sequential VS paralel execution
  27. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited =^.^= Image Source: Wikipedia,
  28. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited /_cat/* api It’s because humans suck at reading JSON (and YAML)
  29. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited Which one is the master? (v0.90) $ curl "localhost:9200/_cluster/state? pretty&filter_metadata=true&filter_routing_table=true"! {! "cluster_name" : "elasticsearch",! "master_node" : "GNf0hEXlTfaBvQXKBF300A",! "blocks" : { },! "nodes" : {! "ObdRqLHGQ6CMI5rOEstA5A" : {! "name" : "Triton",! "transport_address" : “inet[/10.0.1.11:9300]”,! "attributes" : { }! },! "4C7pKbfhTvu0slcSy_G4_w" : {! "name" : "Kid Colt",! "transport_address" : "inet[/10.0.1.12:9300]",! "attributes" : { }! },! "GNf0hEXlTfaBvQXKBF300A" : {! "name" : "Lang, Steven",! "transport_address" : "inet[/10.0.1.13:9300]",! "attributes" : { }! }! }! } master IP master ID
  30. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited Which one is the master? (v1.0) $ curl localhost:9200/_cat/master?v id host ip node GNf0hEXlTfaBvQXKBF300A MyHost.local 10.0.1.13 Honza
  31. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited /_cat/count $ curl localhost:9200/_cat/count?v! epoch timestamp count! 1383501234301 12:53:54 3344067 count
  32. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited _cat/* api /_cat/allocation /_cat/count /_cat/health /_cat/master /_cat/nodes /_cat/recovery /_cat/shards /_cat/indices
  33. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission

    is strictly prohibited thank you! Honza Král twitter: @honzakral email: [email protected] ! ! ! ! ! ! ! ! • Support: http://elasticsearch.com/support • Training: http://training.elasticsearch.com/ • We are hiring: http://elasticsearch.com/about/jobs/