Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Consistency & Elasticsearch

Consistency & Elasticsearch

Fun with working around eventual consistency!

Elasticsearch Inc

November 04, 2013
Tweet

More Decks by Elasticsearch Inc

Other Decks in Technology

Transcript

  1. Elasticsearch & Data Consistency @drewr Tuesday, November 5, 13

  2. Search Tuesday, November 5, 13

  3. Data Tuesday, November 5, 13

  4. Fast Tuesday, November 5, 13

  5. Distributed Tuesday, November 5, 13

  6. Tuesday, November 5, 13

  7. Tuesday, November 5, 13

  8. Tuesday, November 5, 13

  9. Tuesday, November 5, 13

  10. Tuesday, November 5, 13

  11. Tuesday, November 5, 13

  12. Tuesday, November 5, 13

  13. Tuesday, November 5, 13

  14. Tuesday, November 5, 13

  15. Tuesday, November 5, 13

  16. Tuesday, November 5, 13

  17. Tuesday, November 5, 13

  18. Tuesday, November 5, 13

  19. Tuesday, November 5, 13

  20. Tuesday, November 5, 13

  21. P R R P Tuesday, November 5, 13

  22. Requests Tuesday, November 5, 13

  23. Tuesday, November 5, 13

  24. Tuesday, November 5, 13

  25. Tuesday, November 5, 13

  26. Tuesday, November 5, 13

  27. ? ? ? ? Tuesday, November 5, 13

  28. Consistency Tuesday, November 5, 13

  29. PUT /foo/t/one { "age": 35, "bio": "Some long text", "name":

    "Foo" } Tuesday, November 5, 13
  30. Update http://www.elasticsearch.org/guide/reference/api/update/ Tuesday, November 5, 13

  31. POST /foo/t/one/_update { "doc": { "name": "Bar" } } Tuesday,

    November 5, 13
  32. POST /foo/t/one/_update { "params": { "ageinc": 1 }, "script": "ctx._source.age

    += ageinc" } Tuesday, November 5, 13
  33. PUT /foo/t/one { "age": 36, "bio": "Some long text", "name":

    "Bar" } Tuesday, November 5, 13
  34. A write is a write... Tuesday, November 5, 13

  35. { "_id": "one", "_index": "foo", "_type": "t", "_version": 1, "ok":

    true } Tuesday, November 5, 13
  36. { "_id": "one", "_index": "foo", "_type": "t", "_version": 1, "ok":

    true } Tuesday, November 5, 13
  37. PUT /foo/t/one { "age": 36, "bio": "Some long text", "name":

    "Baz" } Tuesday, November 5, 13
  38. { "_id": "one", "_index": "foo", "_type": "t", "_version": 2, "ok":

    true } Tuesday, November 5, 13
  39. PUT /foo/t/one?version=1 { "age": 36, "bio": "Some long text", "name":

    "Baz" } Tuesday, November 5, 13
  40. { "error": "VersionConflict... current [2], provided [1]]", "status": 409 }

    Tuesday, November 5, 13
  41. PUT /foo/t/one?version=2 { "age": 36, "bio": "Some long text", "name":

    "Baz" } Tuesday, November 5, 13
  42. { "_id": "one", "_index": "foo", "_type": "t", "_version": 3, "ok":

    true } Tuesday, November 5, 13
  43. GET/Search Tuesday, November 5, 13

  44. PUT /foo/t/one?version=3&refresh=true { "age": 36, "bio": "Some long text", "name":

    "Quux" } Tuesday, November 5, 13
  45. { "_id": "one", "_index": "foo", "_type": "t", "_version": 4, "ok":

    true } Tuesday, November 5, 13
  46. ?consistency=... Tuesday, November 5, 13

  47. ?replication=... Tuesday, November 5, 13

  48. Case Study Tuesday, November 5, 13

  49. Tuesday, November 5, 13

  50. Tuesday, November 5, 13

  51. No New Deployment Tuesday, November 5, 13

  52. No New Deployment Consume perf loss OK Tuesday, November 5,

    13
  53. No New Deployment Consume perf loss OK Don’t really care

    about AMQP Tuesday, November 5, 13
  54. Rabbit Elasticsearch Exchange Index Queue Type Routing key ? Tuesday,

    November 5, 13
  55. news.politics.uk news.sports news.politics.usa news.* trade.dow trade.nasdaq Tuesday, November 5, 13

  56. Search! Tuesday, November 5, 13

  57. POST /exch/test.foo { "op": "frob", "thing": "/over/there" } Publish... Tuesday,

    November 5, 13
  58. GET /exch/test.foo/_search { "query": { "bool": { "must": [ {

    "constant_score": { "filter": { "missing": { "existence": true, "field": "__q_status", "null_value": true } } } }, { "prefix": { "_type": "test.foo" } } ] } }, "size": 1, "sort": [ { "__q_control": { "order": "desc" } } ] } Consume... Tuesday, November 5, 13
  59. PUT /exch/test.foo/UN1QU3ID?version=1&refresh=true { ... "__q_status": "unack" } Consume... Tuesday, November

    5, 13
  60. PUT /exch/test.foo/UN1QU3ID?version=2&refresh=true { ... "__q_status": "ack" } Finish up... Tuesday,

    November 5, 13
  61. Performance Tuesday, November 5, 13

  62. Conclusion Tuesday, November 5, 13