New features in Elasticsearch 1.0

Slide 1

Slide 1 text

Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearch 1.0 New features in @lucacavanna

Slide 2

Slide 2 text

Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited elasticsearch? what is

Slide 3

Slide 3 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited JSON distributed real-time analytics RESTful Lucene open source schema-free document oriented search

Slide 4

Slide 4 text

Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited Setup $ wget https://download.elasticsearch.org/ elasticsearch/elasticsearch/elasticsearch-1.0.1.zip ! $ unzip elasticsearch-1.0.1.zip ! $ cd elasticsearch-1.0.1 ! $ bin/elasticsearch

Slide 5

Slide 5 text

Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited Setup $ curl localhost:9200 ! { "status" : 200, "name" : "Moondark", "version" : { "number" : "1.0.1", "build_hash" : "5c03844e1978e5cc924dab2a423dc63ce881c42b", "build_timestamp" : "2014-02-25T15:52:53Z", "build_snapshot" : false, "lucene_version" : "4.6" }, "tagline" : "You Know, for Search" }

Slide 6

Slide 6 text

Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited Index $ curl -XPUT localhost:9200/twitter/tweet/1 -d ' { "tweet" : "New features in elasticsearch 1.0", "name" : "Luca Cavanna", "nick" : "lucacavanna", "date" : "2014-03-18", "location" : { "lat" : "13.4", "lon" : "52.5" }, "retweets" : 50 } '

Slide 7

Slide 7 text

Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited Get $ curl -XGET localhost:9200/twitter/tweet/1 Delete $ curl -XDELETE localhost:9200/twitter/tweet/1

Slide 8

Slide 8 text

Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited Search $ curl -XGET localhost:9200/_search?q=elasticsearch $ curl -XGET localhost:9200/_search -d ' { "query" : { "query_string" : { "query" : "elasticsearch AND features" } } } '

Slide 9

Slide 9 text

Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited Search - query DSL $ curl -XGET localhost:9200/_search -d ' { "query" : { "filtered" : { "query" : { "bool" : { "must" : [ { "match" : { "tweet" : { "query" : "elasticsearch features", "operator" : "AND" }}} ], "should" : [ { "match" : {"tweet" : "pizza"} } ] } }, "filter" : { "range" : { "date" : {"from" : "2014-03-01"} } } } } } '

Slide 10

Slide 10 text

Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited 12 Feb 2014 v1.0

Slide 11

Slide 11 text

Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited snapshot & restore Photo by John http://www.ﬂickr.com/people/60026579@N00

Slide 12

Slide 12 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited backup in 0.90 • disable flush • find all primary shards location (optional) • copy files (rsync) • re-enable flush

Slide 13

Slide 13 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited backup in 1.0 - repository $ curl -XPUT localhost:9200/_snapshot/local -d ' { "type" : "fs", "settings" : { "location" : "/data/es/backup" } } '

Slide 14

Slide 14 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited backup in 1.0 - snapshot $ curl -XPUT localhost:9200/_snapshot/local/backup_1 -d ' { "indices" : "*,-twitter*" } '

Slide 15

Slide 15 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited restore in 0.90 • close the index • ﬁnd all existing shards • replace ﬁles with ones from backup • re-open the index

Slide 16

Slide 16 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited restore in 1.0 $ curl -XPOST localhost:9200/2014-*/_close • close the index/indices $ curl -XPOST localhost:9200/_snapshot/local/backup_1/_restore -d ' { "indices" : "2014-*" } ' • restore existing snapshot

Slide 17

Slide 17 text

Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited aggregations

Slide 18

Slide 18 text

Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited Facets in 0.90

Slide 19

Slide 19 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Facets in 0.90 • terms / terms stats • range • histogram / date histogram • statistical • geo distance • ﬁlter / query

Slide 20

Slide 20 text

Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited retweets stats per user $ curl -XGET localhost:9200/twitter/_search -d ' { "facets" : { "retweets_per_user" : { "terms_stats" : { "key_field" : "nick", "value_field" : "retweets" } } } } '

Slide 21

Slide 21 text

Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited retweets stats per user { "facets" : { "retweets_per_user" : { "_type" : "terms_stats", "missing" : 0, "terms" : [{ "term" : “lucacavanna”, "count" : 1, "total_count" : 1, "min" : 50.0, "max" : 50.0, "total" : 50.0, "mean" : 50.0 }] } } }

Slide 22

Slide 22 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Nothing! What’s wrong?

Slide 23

Slide 23 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited give me the retweets stats per user, grouped by month… cool, then…

Slide 24

Slide 24 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited aggregations The answer is…

Slide 25

Slide 25 text

Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited retweets stats per user per month $ curl -XGET localhost:9200/twitter/_search -d ' { "aggs" : { "month" : { "date_histogram" : { "field" : "date", "interval" : "month" }, "aggs" : { "user" : { "terms" : { "field" : "nick" } }, "aggs" : { "retweets" : { "stats" : { "field" : "retweets" } } } } } } } '

Slide 26

Slide 26 text

Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited retweets stats per user per month { "aggregations" : { "month" : { "buckets" : [ { "key" : 1393632000000, "doc_count" : 1, "user" : { "buckets" : [ { "key" : "lucacavanna", "doc_count" : 1, "retweets" : { "count" : 1, "min" : 50, "max" : 50, "avg" : 50, "sum" : 50 } } ] } } ] } } }

Slide 27

Slide 27 text

Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited aggregations buckets metrics

Slide 28

Slide 28 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited buckets • global • ﬁlter • missing • terms • range • date_range • ip_range • histogram • date_histogram • geo_distance • nested

Slide 29

Slide 29 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited metrics • value_count • stats • extended_stats • avg • min • max • sum

Slide 30

Slide 30 text

Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited distributed percolator

Slide 31

Slide 31 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited register query $ curl -XPUT localhost:9200/twitter/.percolator/es-features -d ' { "query" : { "query_string" : { "query" : "elasticsearch AND features" } }, "alert_type" : "mention" } '

Slide 32

Slide 32 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited percolate document $ curl -XGET localhost:9200/twitter/tweet/_percolate -d ' { "doc" : { "tweet" : "New features in elasticsearch 1.0", "name" : "Luca Cavanna", "nick" : "lucacavanna", "date" : "2014-03-18", "retweets" : 50 } } ' { … "total" : 1, "matches" : [{ "_index" : "twitter", "_id" : "es-features" }] }

Slide 33

Slide 33 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited 0.90 VS 1.0 • single shard • sequential execution • _percolator index • single index percolation • arbitrary number of shards • parallel execution • .percolator type (any index) • multi index percolation

Slide 34

Slide 34 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited new percolation features in 1.0 • percolate existing documents • percolate count api • ﬁlter support (in addition to queries) • highlighting • scoring • multi percolate • support for aggregations

Slide 35

Slide 35 text

Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited cat/* api

Slide 36

Slide 36 text

Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited Which node is the master? $ curl localhost:9200/cluster/_state/nodes,master_node?pretty ! { "cluster_name" : "elasticsearch", "master_node" : "yT4GUfIWTY6aJdQtWVEFpw", "nodes” : { "R-5_0LiORAWmr_cYLXO69Q" : { "name" : "Woodgod", "transport_address" : "inet[/192.168.0.12:9302]", "attributes" : {} }, "yT4GUfIWTY6aJdQtWVEFpw" : { "name” : "Moondark", "transport_address" : "inet[/192.168.0.12:9300]", "attributes" : {} }, "pR0NmKeGTVGget2O1qSqCQ" : { "name" : "Adaptoid", "transport_address" : "inet[/192.168.0.12:9301]", "attributes" : {} } } }

Slide 37

Slide 37 text

Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited Which node is the master? (0.90) $ curl localhost:9200/cluster/_state/nodes,master_node?pretty ! { "cluster_name" : "elasticsearch", "master_node" : "yT4GUfIWTY6aJdQtWVEFpw", "nodes” : { "R-5_0LiORAWmr_cYLXO69Q" : { "name" : "Woodgod", "transport_address" : "inet[/192.168.0.12:9302]", "attributes" : {} }, "yT4GUfIWTY6aJdQtWVEFpw" : { "name” : "Moondark", "transport_address" : "inet[/192.168.0.12:9300]", "attributes" : {} }, "pR0NmKeGTVGget2O1qSqCQ" : { "name" : "Adaptoid", "transport_address" : "inet[/192.168.0.12:9301]", "attributes" : {} } } }

Slide 38

Slide 38 text

Slide 39

Slide 39 text

Slide 40

Slide 40 text

Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited Which node is the master? (1.0) $ curl localhost:9200/_cat/master ! ! yT4GUfIWTY6aJdQtWVEFpw Lucas-MacBook-Air.local 192.168.0.12 Moondark

Slide 41

Slide 41 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited _cat*/api • /_cat/allocation • /_cat/aliases • /_cat/count • /_cat/indices • /_cat/recovery • /_cat/shards • /_cat/health • /_cat/thread_pool • /_cat/pending_tasks • /_cat/master • /_cat/nodes

Slide 42

Slide 42 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited thank you! Support: http://elasticsearch.com/support Training: http://training.elasticsearch.com ! We are hiring: http://elasticsearch.com/about/jobs/ @lucacavanna