Upgrade to Pro — share decks privately, control downloads, hide ads and more …

1/10 of a version, 10x the punch - coming features in ES 1.0

Boaz Leskes
January 15, 2014

1/10 of a version, 10x the punch - coming features in ES 1.0

Presented at the Atlanta Elasticsearch meetup on Jan 15th 2014

Boaz Leskes

January 15, 2014
Tweet

More Decks by Boaz Leskes

Other Decks in Technology

Transcript

  1. Boaz Leskes @bleskes ♎ of a version, 10x the punch

    coming features in 1.0
  2. 1.0 RC1

  3. So…. what’s coming? Aggregations (best thing since lego blocks) _cat

    API (feline love for the dev op) Distributed Percolation (put some nitro in your coffee) Snapshot & Restore (point in time, API driven backup) Federated search (get your results from multiple clusters) many, many more (memory circuit breaker, geo points compression, major improvement in allocation decision speed, ….)
  4. Aggregations "

  5. What’s wrong with facets? nothing it’s just that we want

    more…
  6. curl -X GET 'localhost:9200/scores/_search/' -d '{
 "query" : {
 "match"

    : {
 "student" : "john"
 }
 },
 "facets": {
 "subjects" : {
 "terms" : {
 "field" : "subject",
 }
 }
 }
 }'
 John’s report card curl -X GET 'localhost:9200/scores/_search/' -d '{
 "query" : {
 "match" : {
 "student" : "john"
 }
 },
 "facets": {
 "scores" : {
 "statistical" : {
 "field" : "score",
 }
 }
 }
 }'

  7. curl -X GET 'localhost:9200/scores/_search/?search_type=count&pretty' -d '{
 "query" : {
 "match"

    : {
 "student" : "john"
 }
 },
 "facets": {
 "scores-per-subject" : {
 "terms_stats" : {
 "key_field" : "subject",
 "value_field" : "score"
 }
 }
 }
 }'
 "facets" : { "scores-per-subject" : { "_type" : "terms_stats", "missing" : 0, "terms" : [ { "term" : "math", "count" : 1, "total_count" : 1, "min" : 85.0, "max" : 85.0, "total" : 85.0, "mean" : 85.0 }, ... ] } } John’s report card
  8. curl -X GET 'localhost:9200/scores/_search/' -d '{
 "query" : {
 "match"

    : {
 "student" : "john"
 }
 },
 "aggs": {
 "scores-per-subject" : {
 "terms" : { "field" : “subject” }, "aggs" : { “avg_score” : { "avg" : { "field" : "score"
 } } } }
 }
 }'
 "aggregations" : { "scores-per-subject" : { "terms" : [ { "term" : "math", "doc_count" : 1, "avg_score" : { “value": 85.0 } }, ... ] } } John’s report card, agg style
  9. curl -X GET 'localhost:9200/scores/_search/' -d '{
 "query" : {
 "match"

    : {
 "student" : "john"
 }
 },
 "aggs": {
 "scores-per-subject" : {
 "terms" : { "field" : “subject” }, "aggs" : { "avg_score_by_year”: { “date_histogram”: { "field" : "date", "interval" : "year", "format" : "yyyy" } "aggs": { "avg_score" : { "avg": { "field" : "score"
 } } } "aggregations" : { "scores-per-subject" : { "terms" : [ { "term" : "math", "doc_count" : 1, "avg_score_by_year" : [{ "key_as_string": "2013", "avg_score": { “value”: 85.0 } }… ] }, ... ] } } John has graduated…
  10. None
  11. _cat API "

  12. what’s wrong with JSON? nothing it’s just that we are

    not smart enough to read it
  13. { "cluster_name" : "elasticsearch", "master_node" : "GNf0hEXlTfaBvQXKBF300A", "blocks" : {

    }, "nodes" : { "ObdRqLHGQ6CMI5rOEstA5A" : { "name" : "Triton", … }, "4C7pKbfhTvu0slcSy_G4_w" : { "name" : "Kid Colt", … }, "GNf0hEXlTfaBvQXKBF300A" : { "name" : "Lang, Steven", … } } { "cluster_name" : "elasticsearch", "master_node" : "GNf0hEXlTfaBvQXKBF300A", "blocks" : { }, "nodes" : { "ObdRqLHGQ6CMI5rOEstA5A" : { "name" : "Triton", … }, "4C7pKbfhTvu0slcSy_G4_w" : { "name" : "Kid Colt", … }, "GNf0hEXlTfaBvQXKBF300A" : { "name" : "Lang, Steven", … } } who is the master? curl "localhost:9200/_cluster/state? pretty&filter_metadata=true&filter_routing_table=true"
  14. who is the master? _cat style boaz-air:elasticsearch$: curl localhost:9200/_cat/master !

    GNf0hEXlTfaBvQXKBF300A 10.0.1.13 Lang, Steven ! boaz-air:elasticsearch$:
  15. Distributed Percolation "

  16. curl -XPUT “localhost:9200/twitter/.percolator/es-tweets” -d ‘{ “query”: { “match”: { “body”:

    “elasticsearch” } } }’ $ curl -XGET “localhost:9200/twitter/_percolate” -d ‘{ “doc”: { “body”: “#elasticsearch is awesome” “nick”: “@imotov” “name”: “Igor Motov” “date”: “2013-11-03” } }’ { … “matches”: [ { “_index”: “twitter”, “_id”: “es-tweets” } ] }
  17. Huh? why is it useful? •Alerting •Price pointing •Contextual advertisement

    •Classifications what our users do:
  18. So what’s in distribution? •Highlighting •Sorting •Multi-Index support •Aggregations •Multi-Percolate

  19. Snapshot & Restore "

  20. Backup, 0.90 style 1. disable flush 2. find all primary

    shard location (optional) 3. copy files from primary shards (rsync) 4. enable flush
  21. curl -XPUT “localhost:9200/_snapshot/my_backup/snapshot_20140101” Backup, 1.0 style

  22. Register a repository curl -XPUT "localhost:9200/_snapshot/my_backup" -d '{ "type": "fs",

    "settings": { "location":"/mnt/es-test-repo" } }'
  23. curl -XPUT “localhost:9200/_snapshot/my_backup/snapshot_20140101” -d ‘{ "indices":"+test_*,-test_4" }’ Creating a Snapshot

  24. Restore, 0.90 style 1. close the index (shutdown the cluster)

    2. find all existing index shards 3. replace all index shards with data from backup 4. open the index (start the cluster)
  25. curl -XPOST "localhost:9200/test_*/_close" Restore, 1.0 style curl -XPOST "localhost:9200/_snapshot/my_backup/snapshot_20140101" -d

    '{ "indices":"test_*" }'
  26. thanks!