Consistency & Elasticsearch

Fun with working around eventual consistency!

Transcript

1. Elasticsearch & Data Consistency @drewr Tuesday, November 5, 13
2. Search Tuesday, November 5, 13
3. Data Tuesday, November 5, 13
4. Fast Tuesday, November 5, 13
5. Distributed Tuesday, November 5, 13
6. Tuesday, November 5, 13
7. Tuesday, November 5, 13
8. Tuesday, November 5, 13
9. Tuesday, November 5, 13
10. Tuesday, November 5, 13
11. Tuesday, November 5, 13
12. Tuesday, November 5, 13
13. Tuesday, November 5, 13
14. Tuesday, November 5, 13
15. Tuesday, November 5, 13
16. Tuesday, November 5, 13
17. Tuesday, November 5, 13
18. Tuesday, November 5, 13
19. Tuesday, November 5, 13
20. Tuesday, November 5, 13
21. P R R P Tuesday, November 5, 13
22. Requests Tuesday, November 5, 13
23. Tuesday, November 5, 13
24. Tuesday, November 5, 13
25. Tuesday, November 5, 13
26. Tuesday, November 5, 13
27. ? ? ? ? Tuesday, November 5, 13
28. Consistency Tuesday, November 5, 13
29. PUT /foo/t/one { "age": 35, "bio": "Some long text", "name": "Foo" } Tuesday, November 5, 13
30. Update http://www.elasticsearch.org/guide/reference/api/update/ Tuesday, November 5, 13
31. POST /foo/t/one/_update { "doc": { "name": "Bar" } } Tuesday, November 5, 13
32. POST /foo/t/one/_update { "params": { "ageinc": 1 }, "script": "ctx._source.age += ageinc" } Tuesday, November 5, 13
33. PUT /foo/t/one { "age": 36, "bio": "Some long text", "name": "Bar" } Tuesday, November 5, 13
34. A write is a write... Tuesday, November 5, 13
35. { "_id": "one", "_index": "foo", "_type": "t", "_version": 1, "ok": true } Tuesday, November 5, 13
36. { "_id": "one", "_index": "foo", "_type": "t", "_version": 1, "ok": true } Tuesday, November 5, 13
37. PUT /foo/t/one { "age": 36, "bio": "Some long text", "name": "Baz" } Tuesday, November 5, 13
38. { "_id": "one", "_index": "foo", "_type": "t", "_version": 2, "ok": true } Tuesday, November 5, 13
39. PUT /foo/t/one?version=1 { "age": 36, "bio": "Some long text", "name": "Baz" } Tuesday, November 5, 13
40. { "error": "VersionConflict... current [2], provided [1]]", "status": 409 } Tuesday, November 5, 13
41. PUT /foo/t/one?version=2 { "age": 36, "bio": "Some long text", "name": "Baz" } Tuesday, November 5, 13
42. { "_id": "one", "_index": "foo", "_type": "t", "_version": 3, "ok": true } Tuesday, November 5, 13
43. GET/Search Tuesday, November 5, 13
44. PUT /foo/t/one?version=3&refresh=true { "age": 36, "bio": "Some long text", "name": "Quux" } Tuesday, November 5, 13
45. { "_id": "one", "_index": "foo", "_type": "t", "_version": 4, "ok": true } Tuesday, November 5, 13
46. ?consistency=... Tuesday, November 5, 13
47. ?replication=... Tuesday, November 5, 13
48. Case Study Tuesday, November 5, 13
49. Tuesday, November 5, 13
50. Tuesday, November 5, 13
51. No New Deployment Tuesday, November 5, 13
52. No New Deployment Consume perf loss OK Tuesday, November 5, 13
53. No New Deployment Consume perf loss OK Don’t really care about AMQP Tuesday, November 5, 13
54. Rabbit Elasticsearch Exchange Index Queue Type Routing key ? Tuesday, November 5, 13
55. news.politics.uk news.sports news.politics.usa news.* trade.dow trade.nasdaq Tuesday, November 5, 13
56. Search! Tuesday, November 5, 13
57. POST /exch/test.foo { "op": "frob", "thing": "/over/there" } Publish... Tuesday, November 5, 13
58. GET /exch/test.foo/_search { "query": { "bool": { "must": [ { "constant_score": { "filter": { "missing": { "existence": true, "field": "__q_status", "null_value": true } } } }, { "prefix": { "_type": "test.foo" } } ] } }, "size": 1, "sort": [ { "__q_control": { "order": "desc" } } ] } Consume... Tuesday, November 5, 13
59. PUT /exch/test.foo/UN1QU3ID?version=1&refresh=true { ... "__q_status": "unack" } Consume... Tuesday, November 5, 13
60. PUT /exch/test.foo/UN1QU3ID?version=2&refresh=true { ... "__q_status": "ack" } Finish up... Tuesday, November 5, 13
61. Performance Tuesday, November 5, 13
62. Conclusion Tuesday, November 5, 13