Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Whats new in Elasticsearch?

Whats new in Elasticsearch?

This presentation covers the most interesting features of Elasticsearch 1.0 and 1.1 and gives a short forecast on the most interesting features of the upcoming release.

Elasticsearch Inc

May 06, 2014
Tweet

More Decks by Elasticsearch Inc

Other Decks in Technology

Transcript

  1. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Whats new in Elasticsearch?
    Covering 1.0, 1.1, 1.2 & 1.x
    Alexander Reelsen
    @spinscale
    [email protected]

    View full-size slide

  2. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Agenda
    • Introduction
    • Elasticsearch
    • 1.0: Aggregations, Snapshot/Restore, Percolator, cat API
    • 1.1: More Aggregations, Search template, Recovery
    • 1.2: More Aggregations, Context Suggester, global ordinals
    • 1.3: More Aggregations, Benchmark API
    • Q & A

    View full-size slide

  3. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    about
    • Me
    Interested in metrics, ops and the web
    Likes the JVM
    Working with elasticsearch since 2011
    • Elasticsearch, founded in 2012
    Products: Elasticsearch, Logstash, Kibana, Marvel
    Professional services: Support & development subscriptions
    Trainings

    View full-size slide

  4. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Introduction

    View full-size slide

  5. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    http://www.elasticsearch.org/blog/elasticsearch-definitive-guide/

    View full-size slide

  6. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Unstructured search

    View full-size slide

  7. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Structured search

    View full-size slide

  8. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Enrichment

    View full-size slide

  9. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Sorting

    View full-size slide

  10. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Pagination

    View full-size slide

  11. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Aggregation

    View full-size slide

  12. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Suggestions

    View full-size slide

  13. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Elasticsearch in 10 seconds
    • Schema-free, REST & JSON based distributed
    document store
    • Open Source: Apache License 2.0
    • Zero configuration
    • Written in Java, extensible

    View full-size slide

  14. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

    View full-size slide

  15. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Installation & first steps

    View full-size slide

  16. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Zero configuration
    $ wget https://download.elasticsearch.org/...
    $ tar -xf elasticsearch-1.2.1.tar.gz
    $ ./elasticsearch-1.2.1/bin/elasticsearch
    ...
    [2014-01-19 14:53:11,508][INFO ][node] [Scanner] started
    ...

    View full-size slide

  17. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Is it alive?
    » curl localhost:9200
    {
    "status" : 200,
    "name" : "Scanner",
    "version" : {
    "number" : “1.2.1",
    "build_hash" : "e018cda7e7a32643d59e0ac3cdb412ccc239af04",
    "build_timestamp" : "2014-03-25T15:11:47Z",
    "build_snapshot" : true,
    "lucene_version" : “4.8.1"
    },
    "tagline" : "You Know, for Search"
    }

    View full-size slide

  18. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    » curl -XPUT localhost:9200/books/book/1 -d '
    {
    "title" : "Elasticsearch - The definitive guide",
    "authors" : "Clinton Gormley",
    "started" : "2013-02-04",
    "pages" : 230
    }'
    Create…

    View full-size slide

  19. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    » curl -XPUT localhost:9200/books/book/1 -d '
    {
    "title" : "Elasticsearch - The definitive guide",
    "authors" : [ "Clinton Gormley", "Zachary Tong" ],
    "started" : "2013-02-04",
    "pages" : 230
    }'
    Update…

    View full-size slide

  20. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Delete…
    » curl -X DELETE localhost:9200/books/book/1
    Realtime GET…
    » curl —X GET localhost:9200/books/book/1
    » curl —X GET localhost:9200/books/book/1/_source

    View full-size slide

  21. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Search
    » curl -XGET localhost:9200/books/_search?q=elasticsearch
    {
    "took" : 2, "timed_out" : false,
    "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 },
    "hits" : {
    "total" : 1, "max_score" : 0.076713204,
    "hits" : [ {
    "_index" : “books", "_type" : “book", "_id" : "1",
    "_score" : 0.076713204, "_source" : {
    "title" : "Elasticsearch - The definitive guide",
    "authors" : [ "Clinton Gormley", "Zachary Tong" ],
    "started" : “2013-02-04", "pages" : 230
    }
    } ]
    }
    }

    View full-size slide

  22. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    » curl -XGET ‘localhost:9200/books/book/_search' -d '{
    "query": {
    "filtered" : {
    "query" : {
    "match": {
    "text" : {
    "query" : “To Be Or Not To Be",
    "cutoff_frequency" : 0.01
    }
    }
    },
    "filter" : {
    "range": {
    "price": {
    "gte": 20.0
    "lte": 50.0
    ...
    }
    }'
    Search - Query DSL

    View full-size slide

  23. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Scalability

    View full-size slide

  24. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Distributed & scalable
    • Replication
    Read scalability
    Removing SPOF
    • Sharding
    Split logical data over several machines
    Write scalability
    Control data flows

    View full-size slide

  25. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Distributed & scalable
    node 1 (m)
    orders
    products
    1
    4
    1 2
    2
    2
    curl  -­‐X  PUT  localhost:9200/orders  -­‐d  '{  
       "settings.index.number_of_shards"  :  4  
       "settings.index.number_of_replicas"  :  1  
    }'
    curl  -­‐X  PUT  localhost:9200/products  -­‐d  '{  
       "settings.index.number_of_shards"  :  2  
       "settings.index.number_of_replicas"  :  0  
    }'

    View full-size slide

  26. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Distributed and scalable
    node 1 (m)
    orders
    products
    2
    1
    4
    1
    node 2
    orders
    products
    2
    2
    3
    3 4
    1

    View full-size slide

  27. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Distributed & scalable
    node 1 (m)
    orders
    products
    2
    1
    4
    1
    node 2
    orders
    products
    2
    2
    node 3
    orders
    products
    3 4
    1
    3

    View full-size slide

  28. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    A request under the hood
    REST Event Loop
    Transport Event Loop
    Action Event Loop
    Request
    Response

    View full-size slide

  29. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Think async!
    • Enforces event driven architecture
    • Support for non-blocking model
    • Enforce loose coupling
    • Prefers push over pull
    • Callback based concurrency
    • Helps to avoid contention on resources / threads

    View full-size slide

  30. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Elasticsearch 1.0 - 1.x

    View full-size slide

  31. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Elasticsearch
    • Aggregations
    • Snapshot/Restore
    • Distributed percolator
    • Cat API
    • ... and more
    • Cardinality Agg
    • Percentiles Agg
    • Significant Terms Agg
    • Search Templates
    • Cross fields search
    1.0 1.1

    View full-size slide

  32. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Elasticsearch
    • Java 7 required
    • Context suggester
    • reverse_nested Agg
    • Indexing/Merging
    • Benchmark API
    • Geo bounds agg
    • Lucene checksums
    1.2 1.x

    View full-size slide

  33. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Release map
    • 0.4.0 - 2010-02
    • 0.5.0 - 2010-03
    • …
    • 0.19.0 - 2012-03
    • 0.20.0 - 2012-12
    • 0.90.0 - 2013-04
    • v1.0.0 - 2014-02
    • v1.1.0 - 2014-03
    • v1.2.0 - 2014-05
    • v1.3.0 - 2014-0?

    View full-size slide

  34. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Elasticsearch 1.0

    View full-size slide

  35. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Aggregations

    View full-size slide

  36. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Aggregations
    • Aggregation of information
    • Facets are one dimensional
    Categories/brands/material of all results of this query
    • Questions are multidimensional
    Average revenue per category id per day
    • What is the average shopping cart size per order
    per hour?

    View full-size slide

  37. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Aggregations
    Documents

    View full-size slide

  38. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Aggregations
    Documents
    Query

    View full-size slide

  39. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Aggregations
    Documents
    Query
    Buckets

    View full-size slide

  40. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Aggregations
    Documents
    Query
    Buckets

    View full-size slide

  41. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Aggregations
    Documents
    Query
    Buckets
    Metrics 123
    123
    243 185

    View full-size slide

  42. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    bucket aggregators
    • global
    • filter
    • missing
    • terms
    • range
    • date range
    • ip range
    • histogram
    • date histogram
    • geo distance
    • nested

    View full-size slide

  43. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    metrics aggregators
    • count
    • stats
    • extended stats
    • avg
    • max
    • min
    • sum

    View full-size slide

  44. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Order average
    » curl -XGET 'localhost:9200/orders/order/_search' -d '
    {
    "aggs": {
    "average_order_size" : {
    "avg" : { "field" : "total" }
    }
    }
    }
    '
    ...
    "aggregations": {
    "average_order_size" : {
    "value" : 658.369
    }
    }
    ...

    View full-size slide

  45. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Order average - filters
    {
    "aggs": {
    “average_order_size_january" : {
    "filter" : {
    "range" : { "created_at" : { "gte" : "2014-01-01", "lt" :
    "2014-02-01" } } },
    "aggs" : {
    "avg" : { "field" : "total" }
    }
    }
    }
    }
    ...
    "aggregations": {
    "average_order_size_january" : {
    "doc_count" : 8,
    "value" : 540.89754
    }
    }
    ...

    View full-size slide

  46. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Order average - by day
    {
    "aggs": {
    "by_day" : {
    "filter" : {
    "range" : {
    "created_at" : {
    "gte" : "2014-01-01", "lt" : "2014-02-01"
    }
    }
    },
    "aggs" : {
    "daily_filter" : {
    "date_histogram" : {
    "field" : "created_at",
    "interval" : "day",
    "format" : "yyyy-MM-dd"
    },
    "aggs" : {
    "average_order_size" : { "avg" : { "field" : "total" } }
    }
    } } } } }

    View full-size slide

  47. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Order average - by day
    ...
    "aggregations": {
    "by_day" : {
    "doc_count" : 32422,
    "daily_filter" : [ {
    "key_as_string" : "2014-01-01",
    "key" : 1388534400000
    "doc_count" : 423,
    "average_order_size" : {
    "value" : 380.0
    }
    }, {
    "key_as_string" : "2014-01-02",
    "key" : 1388534400000
    "doc_count" : 543,
    "average_order_size" : {
    "value" : 323.432
    }
    }, {
    ...
    ]
    ...

    View full-size slide

  48. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Order average - by hour
    {
    "aggs": {
    "by_day" : {
    "filter" : {
    "range" : {
    "created_at" : {
    "gte" : "2014-01-01", "lt" : "2014-02-01"
    }
    }
    },
    "aggs" : {
    "hourly_filter" : {
    "histogram" : {
    "script" : "doc[\0027created_at\0027].date.hourOfDay",
    "interval" : 1
    },
    "aggs" : {
    "average_order_size" : { "avg" : { "field" : "total" } }
    }
    } } } } }

    View full-size slide

  49. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Order average - by hour
    ...
    "aggregations": {
    "by_day" : {
    "doc_count" : 32422,
    "daily_filter" : [ {
    "key" : "11",
    "doc_count" : 1534,
    "average_order_size" : {
    "value" : 380.0
    }
    }, {
    "key" : "18",
    "doc_count" : 8923,
    "average_order_size" : {
    "value" : 485.4323
    }
    }, {
    ...
    ]
    ...

    View full-size slide

  50. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Snapshot/Restore
    http://www.elasticsearch.org/blog/introducing-snapshot-restore/

    View full-size slide

  51. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Backup made easy
    • Several shell commands + login were needed for
    pre 1.0 backups, but not via API
    $ curl -XPUT "localhost:9200/_snapshot/my_backup" -d '{!
    "type": "fs", !
    "settings": {!
    "location":"/mnt/es-test-repo"!
    }!
    }'
    location
    repository
    repository

    type

    View full-size slide

  52. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Start snapshot
    $ curl -XPUT "localhost:9200/_snapshot/my_backup/snapshot_20131010" -d '{!
    "indices":"+test_*,-test_4"!
    }'
    snapshot

    name
    repository
    index list

    (optional)

    View full-size slide

  53. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Restore snapshot
    $ curl -XPOST "localhost:9200/test_*/_close"
    snapshot

    name
    close all indices

    that start with test_
    $ curl -XPOST "localhost:9200/_snapshot/my_backup/snapshot_20131010" -d
    '{!
    "indices":"test_*"!
    }'
    repository

    name
    index

    list

    View full-size slide

  54. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Distributed & scalable Percolator
    http://www.elasticsearch.org/blog/percolator-redesign-blog-post/

    View full-size slide

  55. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    registering percolator in 1.0
    $ curl -XPUT “localhost:9200/some_index/.percolator/es-tweets” -d ‘{!
    “query”: {!
    “match”: { “body”: “elasticsearch” }!
    }!
    }’!
    reserved percolator

    type
    query id
    any index with as
    many shards as you
    need

    View full-size slide

  56. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Percolate
    $ curl -XGET “localhost:9200/some_index/some-type/_percolate” -d ‘{!
    “doc”: {!
    “body”: “#elasticsearch is awesome”!
    “nick”: “@imotov”!
    “name”: “Igor Motov”!
    “date”: “2013-11-03” !
    }!
    }’
    document

    to be percolated
    {!
    "took": 38,!
    "_shards": { "total": 5, "successful": 5, "failed": 0 },!
    "total": 1,!
    "matches": [!
    { "_index": “some_index", "_id": “es-tweets" }!
    ]!
    }

    View full-size slide

  57. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    other features
    • percolation of existing document
    • percolate count api
    • filter support (in addition to queries in 0.90)
    • highlighting, scoring
    • multi-index, aliases support
    • multi percolate (bulk percolation)

    View full-size slide

  58. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Cat API
    http://www.elasticsearch.org/blog/introducing-cat-api/

    View full-size slide

  59. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Helping sysadmins
    • Elasticsearch is full of monitoring APIs
    Everything is returned as JSON
    • Humans are not the world’s best JSON parsers
    • What if elasticsearch had an easy to use interface
    from the commandline?

    View full-size slide

  60. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Which one is the master?
    $ curl "localhost:9200/_cluster/state?pretty&filter_metadata=true&!
    filter_routing_table=true"!
    {!
    "cluster_name" : "elasticsearch",!
    "master_node" : "GNf0hEXlTfaBvQXKBF300A",!
    "blocks" : { },!
    "nodes" : {!
    "ObdRqLHGQ6CMI5rOEstA5A" : {!
    "name" : "Triton",!
    "transport_address" : “inet[/10.0.1.11:9300]”,!
    "attributes" : { }!
    },!
    "4C7pKbfhTvu0slcSy_G4_w" : {!
    "name" : "Kid Colt",!
    "transport_address" : "inet[/10.0.1.12:9300]",!
    "attributes" : { }!
    },!
    "GNf0hEXlTfaBvQXKBF300A" : {!
    "name" : "Lang, Steven",!
    "transport_address" : "inet[/10.0.1.13:9300]",!
    "attributes" : { }!
    }!
    }!
    }

    View full-size slide

  61. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    $ curl "localhost:9200/_cluster/state?
    pretty&filter_metadata=true&filter_routing_table=true"!
    {!
    "cluster_name" : "elasticsearch",!
    "master_node" : "GNf0hEXlTfaBvQXKBF300A",!
    "blocks" : { },!
    "nodes" : {!
    "ObdRqLHGQ6CMI5rOEstA5A" : {!
    "name" : "Triton",!
    "transport_address" : “inet[/10.0.1.11:9300]”,!
    "attributes" : { }!
    },!
    "4C7pKbfhTvu0slcSy_G4_w" : {!
    "name" : "Kid Colt",!
    "transport_address" : "inet[/10.0.1.12:9300]",!
    "attributes" : { }!
    },!
    "GNf0hEXlTfaBvQXKBF300A" : {!
    "name" : "Lang, Steven",!
    "transport_address" : "inet[/10.0.1.13:9300]",!
    "attributes" : { }!
    }!
    }!
    }
    Which one is the master? (v0.90)

    View full-size slide

  62. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Which one is the master? (v1.0)
    $ curl localhost:9200/_cat/master
    GNf0hEXlTfaBvQXKBF300A 10.0.1.13 Lang, Steven

    View full-size slide

  63. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    /cat/*
    » curl 'localhost:9200/_cat?v'!
    =^.^=!
    /_cat/allocation!
    /_cat/shards!
    /_cat/shards/{index}!
    /_cat/master!
    /_cat/nodes!
    /_cat/indices!
    /_cat/indices/{index}!
    /_cat/segments!
    /_cat/segments/{index}!
    /_cat/count!
    /_cat/count/{index}!
    /_cat/recovery!
    /_cat/recovery/{index}!
    /_cat/health!
    /_cat/pending_tasks!
    /_cat/aliases!
    /_cat/aliases/{alias}!
    /_cat/thread_pool!
    /_cat/plugins

    View full-size slide

  64. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    /cat/indices
    » curl 'localhost:9200/_cat/indices?help'!
    health | h | current health status!
    index | i,idx | index name!
    ...!
    !
    !
    » curl 'localhost:9200/_cat/indices?h=i,p,dc,fm'!
    project 5 8 0b!
    .marvel-2014.05.06 1 1069 0b!
    some_index 5 1 8.4kb!
    !
    !
    » curl 'localhost:9200/_cat/indices?h=i,p,dc,fm&v'!
    i p dc fm !
    project 5 8 0b !
    .marvel-2014.05.06 1 1099 0b !
    some_index 5 1 8.4kb !

    View full-size slide

  65. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    And more…

    View full-size slide

  66. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    And more…
    • Disk-based fielddata
    http://www.elasticsearch.org/blog/disk-based-field-data-a-k-a-doc-values/
    • Fielddata circuit breaker
    • Federated search
    http://www.elasticsearch.org/blog/tribe-node/

    View full-size slide

  67. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Elasticsearch 1.1

    View full-size slide

  68. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Elasticsearch 1.1
    • Aggregations
    Percentile
    http://www.elasticsearch.org/blog/averages-can-dangerous-use-percentile/
    Significant terms
    http://www.elasticsearch.org/blog/significant-terms-aggregation/
    Cardinality
    http://www.elasticsearch.org/blog/count-elasticsearch/
    • Improved multi field search
    http://www.elasticsearch.org/blog/multi-field-search-just-got-
    better/

    View full-size slide

  69. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Elasticsearch 1.1
    • Search Templates
    http://www.elasticsearch.org/blog/dry-keeping-your-queries-
    short/
    • Create index/template: Support for aliases
    http://www.elasticsearch.org/blog/aliases-ftw/
    • Recovery API
    http://www.elasticsearch.org/blog/aliases-ftw/

    View full-size slide

  70. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Significant Terms
    • Finding the uncommonly common
    http://www.infoq.com/presentations/elasticsearch-revealing-
    uncommonly-common
    • Use cases
    Finding mis-categorized content using the “like this but not this”
    pattern
    Training a classifier
    Root cause analysis
    Geographic anomalies
    Detecting credit card fraud

    View full-size slide

  71. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Significant Terms
    Source

    https://xkcd.com/1138/

    View full-size slide

  72. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Template Query
    GET /_search!
    {!
    "query": {!
    "template": {!
    "query": {"match_{{template}}": {}},!
    "params" : {!
    "template" : "all"!
    }!
    }!
    }!
    }

    View full-size slide

  73. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Template Query
    GET /_search!
    {!
    "query": {!
    "template": {!
    "query": “storedTemplate",!
    "params" : {!
    "template" : "all"!
    }!
    }!
    }!
    }
    # cat config/scripts/storedTemplate.mustache!
    !
    {"match_{{template}}": {}}

    View full-size slide

  74. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Search Templates
    GET /_search/template!
    {!
    "template" : {!
    "query": { !
    "match" : { "{{my_field}}" : "{{my_value}}" }!
    },!
    "size" : "{{my_size}}"!
    },!
    "params" : {!
    "my_field" : "user",!
    "my_value" : "alr",!
    "my_size" : 50!
    }!
    }

    View full-size slide

  75. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Search Templates
    GET /_search/template!
    {!
    "template": "storedTemplate",!
    "params": {!
    “my_field": “user”,!
    “my_value": “alr”,!
    “my_size": 50,!
    }!
    }
    # cat config/scripts/storedTemplate.mustache!
    !
    {!
    "query": { !
    "match" : { "{{my_field}}" : "{{my_value}}" }!
    },!
    "size" : "{{my_size}}"!
    }

    View full-size slide

  76. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Elasticsearch 1.2

    View full-size slide

  77. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Elasticsearch 1.2
    • Aggregations: global ordinals, reverse_nested
    • Context Suggester
    • Java7 required

    View full-size slide

  78. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Context suggester
    POST services/_suggest!
    {!
    "suggest" : {!
    "text" : "m",!
    "completion" : {!
    "field" : "suggest_field",!
    "size": 10,!
    "context": {!
    "location": {!
    "lat": 51,!
    "lon": 0!
    }!
    }!
    }!
    }!
    }

    View full-size slide

  79. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Elasticsearch 1.3

    View full-size slide

  80. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Elasticsearch 1.x
    • Resiliency & Stability
    Checksums in Lucene
    http://www.elasticsearch.org/blog/investing-apache-lucene/
    !
    http://www.elasticsearch.org/blog/resiliency-elasticsearch/
    • Benchmark API
    • Geo bounds Agg
    • Lucene checksums

    View full-size slide

  81. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Thanks for listening

    View full-size slide

  82. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Q & A
    Alexander Reelsen
    @spinscale
    [email protected]
    P.S. We’re hiring
    http://elasticsearch.com/about/jobs
    http://elasticsearch.com/support

    View full-size slide