Whats new in Elasticsearch?

This presentation covers the most interesting features of Elasticsearch 1.0 and 1.1 and gives a short forecast on the most interesting features of the upcoming release.

1. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Whats new in Elasticsearch? Covering 1.0, 1.1, 1.2 & 1.x Alexander Reelsen @spinscale alexander.reelsen@elasticsearch.com
2. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Agenda • Introduction • Elasticsearch • 1.0: Aggregations, Snapshot/Restore, Percolator, cat API • 1.1: More Aggregations, Search template, Recovery • 1.2: More Aggregations, Context Suggester, global ordinals • 1.3: More Aggregations, Benchmark API • Q & A
3. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited about • Me Interested in metrics, ops and the web Likes the JVM Working with elasticsearch since 2011 • Elasticsearch, founded in 2012 Products: Elasticsearch, Logstash, Kibana, Marvel Professional services: Support & development subscriptions Trainings
4. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Introduction
5. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited http://www.elasticsearch.org/blog/elasticsearch-definitive-guide/
6. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Unstructured search
7. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Structured search
8. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Enrichment
9. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Sorting
10. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Pagination
11. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Aggregation
12. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Suggestions
13. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearch in 10 seconds • Schema-free, REST & JSON based distributed document store • Open Source: Apache License 2.0 • Zero configuration • Written in Java, extensible
14. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
15. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Installation & first steps
16. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Zero configuration $ wget https://download.elasticsearch.org/... $ tar -xf elasticsearch-1.2.1.tar.gz $ ./elasticsearch-1.2.1/bin/elasticsearch ... [2014-01-19 14:53:11,508][INFO ][node] [Scanner] started ...
17. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Is it alive? » curl localhost:9200 { "status" : 200, "name" : "Scanner", "version" : { "number" : “1.2.1", "build_hash" : "e018cda7e7a32643d59e0ac3cdb412ccc239af04", "build_timestamp" : "2014-03-25T15:11:47Z", "build_snapshot" : true, "lucene_version" : “4.8.1" }, "tagline" : "You Know, for Search" }
18. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited » curl -XPUT localhost:9200/books/book/1 -d ' { "title" : "Elasticsearch - The definitive guide", "authors" : "Clinton Gormley", "started" : "2013-02-04", "pages" : 230 }' Create…
19. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited » curl -XPUT localhost:9200/books/book/1 -d ' { "title" : "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : "2013-02-04", "pages" : 230 }' Update…
20. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Delete… » curl -X DELETE localhost:9200/books/book/1 Realtime GET… » curl —X GET localhost:9200/books/book/1 » curl —X GET localhost:9200/books/book/1/_source
21. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Search » curl -XGET localhost:9200/books/_search?q=elasticsearch { "took" : 2, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.076713204, "hits" : [ { "_index" : “books", "_type" : “book", "_id" : "1", "_score" : 0.076713204, "_source" : { "title" : "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : “2013-02-04", "pages" : 230 } } ] } }
22. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited » curl -XGET ‘localhost:9200/books/book/_search' -d '{ "query": { "filtered" : { "query" : { "match": { "text" : { "query" : “To Be Or Not To Be", "cutoff_frequency" : 0.01 } } }, "filter" : { "range": { "price": { "gte": 20.0 "lte": 50.0 ... } }' Search - Query DSL
23. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Scalability
24. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Distributed & scalable • Replication Read scalability Removing SPOF • Sharding Split logical data over several machines Write scalability Control data flows
25. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Distributed & scalable node 1 (m) orders products 1 4 1 2 2 2 curl  -­‐X  PUT  localhost:9200/orders  -­‐d  '{      "settings.index.number_of_shards"  :  4      "settings.index.number_of_replicas"  :  1   }' curl  -­‐X  PUT  localhost:9200/products  -­‐d  '{      "settings.index.number_of_shards"  :  2      "settings.index.number_of_replicas"  :  0   }'
26. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Distributed and scalable node 1 (m) orders products 2 1 4 1 node 2 orders products 2 2 3 3 4 1
27. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Distributed & scalable node 1 (m) orders products 2 1 4 1 node 2 orders products 2 2 node 3 orders products 3 4 1 3
28. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited A request under the hood REST Event Loop Transport Event Loop Action Event Loop Request Response
29. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Think async! • Enforces event driven architecture • Support for non-blocking model • Enforce loose coupling • Prefers push over pull • Callback based concurrency • Helps to avoid contention on resources / threads
30. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearch 1.0 - 1.x
31. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearch • Aggregations • Snapshot/Restore • Distributed percolator • Cat API • ... and more • Cardinality Agg • Percentiles Agg • Significant Terms Agg • Search Templates • Cross fields search 1.0 1.1
32. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearch • Java 7 required • Context suggester • reverse_nested Agg • Indexing/Merging • Benchmark API • Geo bounds agg • Lucene checksums 1.2 1.x
33. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Release map • 0.4.0 - 2010-02 • 0.5.0 - 2010-03 • … • 0.19.0 - 2012-03 • 0.20.0 - 2012-12 • 0.90.0 - 2013-04 • v1.0.0 - 2014-02 • v1.1.0 - 2014-03 • v1.2.0 - 2014-05 • v1.3.0 - 2014-0?
34. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearch 1.0
35. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Aggregations
36. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Aggregations • Aggregation of information • Facets are one dimensional Categories/brands/material of all results of this query • Questions are multidimensional Average revenue per category id per day • What is the average shopping cart size per order per hour?
37. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Aggregations Documents
38. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Aggregations Documents Query
39. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Aggregations Documents Query Buckets
40. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Aggregations Documents Query Buckets
41. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Aggregations Documents Query Buckets Metrics 123 123 243 185
42. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited bucket aggregators • global • filter • missing • terms • range • date range • ip range • histogram • date histogram • geo distance • nested
43. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited metrics aggregators • count • stats • extended stats • avg • max • min • sum
44. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Order average » curl -XGET 'localhost:9200/orders/order/_search' -d ' { "aggs": { "average_order_size" : { "avg" : { "field" : "total" } } } } ' ... "aggregations": { "average_order_size" : { "value" : 658.369 } } ...
45. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Order average - filters { "aggs": { “average_order_size_january" : { "filter" : { "range" : { "created_at" : { "gte" : "2014-01-01", "lt" : "2014-02-01" } } }, "aggs" : { "avg" : { "field" : "total" } } } } } ... "aggregations": { "average_order_size_january" : { "doc_count" : 8, "value" : 540.89754 } } ...
46. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Order average - by day { "aggs": { "by_day" : { "filter" : { "range" : { "created_at" : { "gte" : "2014-01-01", "lt" : "2014-02-01" } } }, "aggs" : { "daily_filter" : { "date_histogram" : { "field" : "created_at", "interval" : "day", "format" : "yyyy-MM-dd" }, "aggs" : { "average_order_size" : { "avg" : { "field" : "total" } } } } } } } }
47. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Order average - by day ... "aggregations": { "by_day" : { "doc_count" : 32422, "daily_filter" : [ { "key_as_string" : "2014-01-01", "key" : 1388534400000 "doc_count" : 423, "average_order_size" : { "value" : 380.0 } }, { "key_as_string" : "2014-01-02", "key" : 1388534400000 "doc_count" : 543, "average_order_size" : { "value" : 323.432 } }, { ... ] ...
48. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Order average - by hour { "aggs": { "by_day" : { "filter" : { "range" : { "created_at" : { "gte" : "2014-01-01", "lt" : "2014-02-01" } } }, "aggs" : { "hourly_filter" : { "histogram" : { "script" : "doc[\0027created_at\0027].date.hourOfDay", "interval" : 1 }, "aggs" : { "average_order_size" : { "avg" : { "field" : "total" } } } } } } } }
49. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Order average - by hour ... "aggregations": { "by_day" : { "doc_count" : 32422, "daily_filter" : [ { "key" : "11", "doc_count" : 1534, "average_order_size" : { "value" : 380.0 } }, { "key" : "18", "doc_count" : 8923, "average_order_size" : { "value" : 485.4323 } }, { ... ] ...
50. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Snapshot/Restore http://www.elasticsearch.org/blog/introducing-snapshot-restore/
51. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Backup made easy • Several shell commands + login were needed for pre 1.0 backups, but not via API $ curl -XPUT "localhost:9200/_snapshot/my_backup" -d '{! "type": "fs", ! "settings": {! "location":"/mnt/es-test-repo"! }! }' location repository repository type
52. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Start snapshot $ curl -XPUT "localhost:9200/_snapshot/my_backup/snapshot_20131010" -d '{! "indices":"+test_*,-test_4"! }' snapshot name repository index list (optional)
53. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Restore snapshot $ curl -XPOST "localhost:9200/test_*/_close" snapshot name close all indices that start with test_ $ curl -XPOST "localhost:9200/_snapshot/my_backup/snapshot_20131010" -d '{! "indices":"test_*"! }' repository name index list
54. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Distributed & scalable Percolator http://www.elasticsearch.org/blog/percolator-redesign-blog-post/
55. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited registering percolator in 1.0 $ curl -XPUT “localhost:9200/some_index/.percolator/es-tweets” -d ‘{! “query”: {! “match”: { “body”: “elasticsearch” }! }! }’! reserved percolator type query id any index with as many shards as you need
56. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Percolate $ curl -XGET “localhost:9200/some_index/some-type/_percolate” -d ‘{! “doc”: {! “body”: “#elasticsearch is awesome”! “nick”: “@imotov”! “name”: “Igor Motov”! “date”: “2013-11-03” ! }! }’ document to be percolated {! "took": 38,! "_shards": { "total": 5, "successful": 5, "failed": 0 },! "total": 1,! "matches": [! { "_index": “some_index", "_id": “es-tweets" }! ]! }
57. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited other features • percolation of existing document • percolate count api • filter support (in addition to queries in 0.90) • highlighting, scoring • multi-index, aliases support • multi percolate (bulk percolation)
58. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Cat API http://www.elasticsearch.org/blog/introducing-cat-api/
59. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Helping sysadmins • Elasticsearch is full of monitoring APIs Everything is returned as JSON • Humans are not the world’s best JSON parsers • What if elasticsearch had an easy to use interface from the commandline?
60. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Which one is the master? $ curl "localhost:9200/_cluster/state?pretty&filter_metadata=true&! filter_routing_table=true"! {! "cluster_name" : "elasticsearch",! "master_node" : "GNf0hEXlTfaBvQXKBF300A",! "blocks" : { },! "nodes" : {! "ObdRqLHGQ6CMI5rOEstA5A" : {! "name" : "Triton",! "transport_address" : “inet[/10.0.1.11:9300]”,! "attributes" : { }! },! "4C7pKbfhTvu0slcSy_G4_w" : {! "name" : "Kid Colt",! "transport_address" : "inet[/10.0.1.12:9300]",! "attributes" : { }! },! "GNf0hEXlTfaBvQXKBF300A" : {! "name" : "Lang, Steven",! "transport_address" : "inet[/10.0.1.13:9300]",! "attributes" : { }! }! }! }
61. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited $ curl "localhost:9200/_cluster/state? pretty&filter_metadata=true&filter_routing_table=true"! {! "cluster_name" : "elasticsearch",! "master_node" : "GNf0hEXlTfaBvQXKBF300A",! "blocks" : { },! "nodes" : {! "ObdRqLHGQ6CMI5rOEstA5A" : {! "name" : "Triton",! "transport_address" : “inet[/10.0.1.11:9300]”,! "attributes" : { }! },! "4C7pKbfhTvu0slcSy_G4_w" : {! "name" : "Kid Colt",! "transport_address" : "inet[/10.0.1.12:9300]",! "attributes" : { }! },! "GNf0hEXlTfaBvQXKBF300A" : {! "name" : "Lang, Steven",! "transport_address" : "inet[/10.0.1.13:9300]",! "attributes" : { }! }! }! } Which one is the master? (v0.90)
62. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Which one is the master? (v1.0) $ curl localhost:9200/_cat/master GNf0hEXlTfaBvQXKBF300A 10.0.1.13 Lang, Steven
63. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited /cat/* » curl 'localhost:9200/_cat?v'! =^.^=! /_cat/allocation! /_cat/shards! /_cat/shards/{index}! /_cat/master! /_cat/nodes! /_cat/indices! /_cat/indices/{index}! /_cat/segments! /_cat/segments/{index}! /_cat/count! /_cat/count/{index}! /_cat/recovery! /_cat/recovery/{index}! /_cat/health! /_cat/pending_tasks! /_cat/aliases! /_cat/aliases/{alias}! /_cat/thread_pool! /_cat/plugins
64. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited /cat/indices » curl 'localhost:9200/_cat/indices?help'! health | h | current health status! index | i,idx | index name! ...! ! ! » curl 'localhost:9200/_cat/indices?h=i,p,dc,fm'! project 5 8 0b! .marvel-2014.05.06 1 1069 0b! some_index 5 1 8.4kb! ! ! » curl 'localhost:9200/_cat/indices?h=i,p,dc,fm&v'! i p dc fm ! project 5 8 0b ! .marvel-2014.05.06 1 1099 0b ! some_index 5 1 8.4kb !
65. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited And more…
66. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited And more… • Disk-based fielddata http://www.elasticsearch.org/blog/disk-based-field-data-a-k-a-doc-values/ • Fielddata circuit breaker • Federated search http://www.elasticsearch.org/blog/tribe-node/
67. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearch 1.1
68. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearch 1.1 • Aggregations Percentile http://www.elasticsearch.org/blog/averages-can-dangerous-use-percentile/ Significant terms http://www.elasticsearch.org/blog/significant-terms-aggregation/ Cardinality http://www.elasticsearch.org/blog/count-elasticsearch/ • Improved multi field search http://www.elasticsearch.org/blog/multi-field-search-just-got- better/
69. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearch 1.1 • Search Templates http://www.elasticsearch.org/blog/dry-keeping-your-queries- short/ • Create index/template: Support for aliases http://www.elasticsearch.org/blog/aliases-ftw/ • Recovery API http://www.elasticsearch.org/blog/aliases-ftw/
70. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Significant Terms • Finding the uncommonly common http://www.infoq.com/presentations/elasticsearch-revealing- uncommonly-common • Use cases Finding mis-categorized content using the “like this but not this” pattern Training a classifier Root cause analysis Geographic anomalies Detecting credit card fraud
71. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Significant Terms Source https://xkcd.com/1138/
72. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Template Query GET /_search! {! "query": {! "template": {! "query": {"match_{{template}}": {}},! "params" : {! "template" : "all"! }! }! }! }
73. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Template Query GET /_search! {! "query": {! "template": {! "query": “storedTemplate",! "params" : {! "template" : "all"! }! }! }! } # cat config/scripts/storedTemplate.mustache! ! {"match_{{template}}": {}}
74. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Search Templates GET /_search/template! {! "template" : {! "query": { ! "match" : { "{{my_field}}" : "{{my_value}}" }! },! "size" : "{{my_size}}"! },! "params" : {! "my_field" : "user",! "my_value" : "alr",! "my_size" : 50! }! }
75. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Search Templates GET /_search/template! {! "template": "storedTemplate",! "params": {! “my_field": “user”,! “my_value": “alr”,! “my_size": 50,! }! } # cat config/scripts/storedTemplate.mustache! ! {! "query": { ! "match" : { "{{my_field}}" : "{{my_value}}" }! },! "size" : "{{my_size}}"! }
76. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearch 1.2
77. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearch 1.2 • Aggregations: global ordinals, reverse_nested • Context Suggester • Java7 required
78. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Context suggester POST services/_suggest! {! "suggest" : {! "text" : "m",! "completion" : {! "field" : "suggest_field",! "size": 10,! "context": {! "location": {! "lat": 51,! "lon": 0! }! }! }! }! }
79. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearch 1.3
80. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearch 1.x • Resiliency & Stability Checksums in Lucene http://www.elasticsearch.org/blog/investing-apache-lucene/ ! http://www.elasticsearch.org/blog/resiliency-elasticsearch/ • Benchmark API • Geo bounds Agg • Lucene checksums
81. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Thanks for listening
82. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Q & A Alexander Reelsen @spinscale alexander.reelsen@elasticsearch.com P.S. We’re hiring http://elasticsearch.com/about/jobs http://elasticsearch.com/support