Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Consistency & Elasticsearch

Consistency & Elasticsearch

Fun with working around eventual consistency!

Elasticsearch Inc

November 04, 2013
Tweet

More Decks by Elasticsearch Inc

Other Decks in Technology

Transcript

  1. Elasticsearch
    & Data Consistency
    @drewr
    Tuesday, November 5, 13

    View full-size slide

  2. Search
    Tuesday, November 5, 13

    View full-size slide

  3. Data
    Tuesday, November 5, 13

    View full-size slide

  4. Fast
    Tuesday, November 5, 13

    View full-size slide

  5. Distributed
    Tuesday, November 5, 13

    View full-size slide

  6. Tuesday, November 5, 13

    View full-size slide

  7. Tuesday, November 5, 13

    View full-size slide

  8. Tuesday, November 5, 13

    View full-size slide

  9. Tuesday, November 5, 13

    View full-size slide

  10. Tuesday, November 5, 13

    View full-size slide

  11. Tuesday, November 5, 13

    View full-size slide

  12. Tuesday, November 5, 13

    View full-size slide

  13. Tuesday, November 5, 13

    View full-size slide

  14. Tuesday, November 5, 13

    View full-size slide

  15. Tuesday, November 5, 13

    View full-size slide

  16. Tuesday, November 5, 13

    View full-size slide

  17. Tuesday, November 5, 13

    View full-size slide

  18. Tuesday, November 5, 13

    View full-size slide

  19. Tuesday, November 5, 13

    View full-size slide

  20. Tuesday, November 5, 13

    View full-size slide

  21. P R R
    P
    Tuesday, November 5, 13

    View full-size slide

  22. Requests
    Tuesday, November 5, 13

    View full-size slide

  23. Tuesday, November 5, 13

    View full-size slide

  24. Tuesday, November 5, 13

    View full-size slide

  25. Tuesday, November 5, 13

    View full-size slide

  26. Tuesday, November 5, 13

    View full-size slide

  27. ?
    ?
    ?
    ?
    Tuesday, November 5, 13

    View full-size slide

  28. Consistency
    Tuesday, November 5, 13

    View full-size slide

  29. PUT /foo/t/one
    {
    "age": 35,
    "bio": "Some long text",
    "name": "Foo"
    }
    Tuesday, November 5, 13

    View full-size slide

  30. Update
    http://www.elasticsearch.org/guide/reference/api/update/
    Tuesday, November 5, 13

    View full-size slide

  31. POST /foo/t/one/_update
    {
    "doc": {
    "name": "Bar"
    }
    }
    Tuesday, November 5, 13

    View full-size slide

  32. POST /foo/t/one/_update
    {
    "params": {
    "ageinc": 1
    },
    "script": "ctx._source.age += ageinc"
    }
    Tuesday, November 5, 13

    View full-size slide

  33. PUT /foo/t/one
    {
    "age": 36,
    "bio": "Some long text",
    "name": "Bar"
    }
    Tuesday, November 5, 13

    View full-size slide

  34. A write is a write...
    Tuesday, November 5, 13

    View full-size slide

  35. {
    "_id": "one",
    "_index": "foo",
    "_type": "t",
    "_version": 1,
    "ok": true
    }
    Tuesday, November 5, 13

    View full-size slide

  36. {
    "_id": "one",
    "_index": "foo",
    "_type": "t",
    "_version": 1,
    "ok": true
    }
    Tuesday, November 5, 13

    View full-size slide

  37. PUT /foo/t/one
    {
    "age": 36,
    "bio": "Some long text",
    "name": "Baz"
    }
    Tuesday, November 5, 13

    View full-size slide

  38. {
    "_id": "one",
    "_index": "foo",
    "_type": "t",
    "_version": 2,
    "ok": true
    }
    Tuesday, November 5, 13

    View full-size slide

  39. PUT /foo/t/one?version=1
    {
    "age": 36,
    "bio": "Some long text",
    "name": "Baz"
    }
    Tuesday, November 5, 13

    View full-size slide

  40. {
    "error": "VersionConflict...
    current [2], provided [1]]",
    "status": 409
    }
    Tuesday, November 5, 13

    View full-size slide

  41. PUT /foo/t/one?version=2
    {
    "age": 36,
    "bio": "Some long text",
    "name": "Baz"
    }
    Tuesday, November 5, 13

    View full-size slide

  42. {
    "_id": "one",
    "_index": "foo",
    "_type": "t",
    "_version": 3,
    "ok": true
    }
    Tuesday, November 5, 13

    View full-size slide

  43. GET/Search
    Tuesday, November 5, 13

    View full-size slide

  44. PUT /foo/t/one?version=3&refresh=true
    {
    "age": 36,
    "bio": "Some long text",
    "name": "Quux"
    }
    Tuesday, November 5, 13

    View full-size slide

  45. {
    "_id": "one",
    "_index": "foo",
    "_type": "t",
    "_version": 4,
    "ok": true
    }
    Tuesday, November 5, 13

    View full-size slide

  46. ?consistency=...
    Tuesday, November 5, 13

    View full-size slide

  47. ?replication=...
    Tuesday, November 5, 13

    View full-size slide

  48. Case Study
    Tuesday, November 5, 13

    View full-size slide

  49. Tuesday, November 5, 13

    View full-size slide

  50. Tuesday, November 5, 13

    View full-size slide

  51. No New Deployment
    Tuesday, November 5, 13

    View full-size slide

  52. No New Deployment
    Consume perf loss OK
    Tuesday, November 5, 13

    View full-size slide

  53. No New Deployment
    Consume perf loss OK
    Don’t really care about AMQP
    Tuesday, November 5, 13

    View full-size slide

  54. Rabbit Elasticsearch
    Exchange Index
    Queue Type
    Routing key ?
    Tuesday, November 5, 13

    View full-size slide

  55. news.politics.uk
    news.sports
    news.politics.usa
    news.*
    trade.dow
    trade.nasdaq
    Tuesday, November 5, 13

    View full-size slide

  56. Search!
    Tuesday, November 5, 13

    View full-size slide

  57. POST /exch/test.foo
    {
    "op": "frob",
    "thing": "/over/there"
    }
    Publish...
    Tuesday, November 5, 13

    View full-size slide

  58. GET /exch/test.foo/_search
    {
    "query": {
    "bool": {
    "must": [
    {
    "constant_score": {
    "filter": {
    "missing": {
    "existence": true,
    "field": "__q_status",
    "null_value": true
    }
    }
    }
    },
    {
    "prefix": {
    "_type": "test.foo"
    }
    }
    ]
    }
    },
    "size": 1,
    "sort": [
    {
    "__q_control": {
    "order": "desc"
    }
    }
    ]
    }
    Consume...
    Tuesday, November 5, 13

    View full-size slide

  59. PUT /exch/test.foo/UN1QU3ID?version=1&refresh=true
    {
    ...
    "__q_status": "unack"
    }
    Consume...
    Tuesday, November 5, 13

    View full-size slide

  60. PUT /exch/test.foo/UN1QU3ID?version=2&refresh=true
    {
    ...
    "__q_status": "ack"
    }
    Finish up...
    Tuesday, November 5, 13

    View full-size slide

  61. Performance
    Tuesday, November 5, 13

    View full-size slide

  62. Conclusion
    Tuesday, November 5, 13

    View full-size slide