Elasticsearch & Data Consistency

Washington, DC Elasticsearch Meetup

How does an update-heavy environment leverage Elasticsearch? We talk about what goes well and what doesn't, along with a useful case study.

Transcript

1. Elasticsearch & Data Consistency @drewr Thursday, September 5, 13 * Introduction * Local training Oct 21-22, early bird ends Sept 21 * Why this topic? - I come from a mostly read-only archiving use, lots of people ask about update perf...
2. Search Thursday, September 5, 13 * Document-based (JSON) * Boolean, fuzzy suggesters, phrase, custom scoring * Lucene
3. Data Thursday, September 5, 13 * Keep a copy of the document * Don’t have to go to another data store
4. Fast Thursday, September 5, 13 * Distributed and concurrent from the ground up * Only locks for some disk writes
5. Distributed Thursday, September 5, 13
6. Thursday, September 5, 13 Shard, the irreducible container Need somewhere to store it -> node
7. Thursday, September 5, 13 Shard, the irreducible container Need somewhere to store it -> node
8. Thursday, September 5, 13
9. Thursday, September 5, 13 Node with two shards
10. Thursday, September 5, 13 Four....
11. Thursday, September 5, 13 Etc.
12. Thursday, September 5, 13
13. Thursday, September 5, 13 Cluster of one node
14. Thursday, September 5, 13 Two nodes
15. Thursday, September 5, 13 Etc.
16. Thursday, September 5, 13 Thirty-five-node cluster
17. Thursday, September 5, 13 Indices are a lightweight abstraction over shards This is a single index with two shards Add a node...
18. Thursday, September 5, 13 Nothing gained by moving anything to it (caveat with perf) Tell ES you want a copy of the data for availability
19. Thursday, September 5, 13
20. Thursday, September 5, 13 Two nodes, four shards For search, completely identical
21. P R R P Thursday, September 5, 13 Two nodes, four shards For search, completely identical
22. Requests Thursday, September 5, 13
23. Thursday, September 5, 13 Search...
24. Thursday, September 5, 13 * Writes are always indexing operations * Doc comes in with ID, or we generate one, hash it (or routing value), give it a unique shard
25. Thursday, September 5, 13 * Primary then replicas * What happens when a search request comes in? * Might choose primary or replica (can tell it primary, but you may not care)
26. Thursday, September 5, 13 * Primary then replicas * What happens when a search request comes in? * Might choose primary or replica (can tell it primary, but you may not care)
27. ? ? ? ? Thursday, September 5, 13 * Might choose primary or replica (can tell it primary, but you may not care) * What if replica doesn’t have the doc yet?
28. Consistency Thursday, September 5, 13 * Eventually consistent (optimistic replication, every copy will eventually see the update) * Crazy! I need transactions!
29. PUT /foo/t/one { "age": 35, "bio": "Some long text", "name": "Foo" } Thursday, September 5, 13 Let’s index a doc
30. Update http://www.elasticsearch.org/guide/reference/api/update/ Thursday, September 5, 13
31. POST /foo/t/one/_update { "doc": { "name": "Bar" } } Thursday, September 5, 13 * We replace just that field * Save network... maybe we can sneak our update in!
32. POST /foo/t/one/_update { "params": { "ageinc": 1 }, "script": "ctx._source.age += ageinc" } Thursday, September 5, 13 * Or a script * Same effect...
33. PUT /foo/t/one { "age": 36, "bio": "Some long text", "name": "Bar" } Thursday, September 5, 13 Or we can just PUT it again
34. A write is a write... Thursday, September 5, 13 * Index, update, delete.... Same end result! * Immutability! * Optimistic concurrency control * Fast, no locks, no magic, HTTP is stateless anyway (ETags)
35. { "_id": "one", "_index": "foo", "_type": "t", "_version": 1, "ok": true } Thursday, September 5, 13 Look at response again...
36. { "_id": "one", "_index": "foo", "_type": "t", "_version": 1, "ok": true } Thursday, September 5, 13
37. PUT /foo/t/one { "age": 36, "bio": "Some long text", "name": "Baz" } Thursday, September 5, 13 PUT it again
38. { "_id": "one", "_index": "foo", "_type": "t", "_version": 2, "ok": true } Thursday, September 5, 13
39. PUT /foo/t/one?version=1 { "age": 36, "bio": "Some long text", "name": "Baz" } Thursday, September 5, 13 PUT it again
40. { "error": "VersionConflict... current [2], provided [1]]", "status": 409 } Thursday, September 5, 13
41. PUT /foo/t/one?version=2 { "age": 36, "bio": "Some long text", "name": "Baz" } Thursday, September 5, 13 Version 2 this time...
42. { "_id": "one", "_index": "foo", "_type": "t", "_version": 3, "ok": true } Thursday, September 5, 13
43. GET/Search Thursday, September 5, 13 * Realtime/NRT
44. PUT /foo/t/one?version=3&refresh=true { "age": 36, "bio": "Some long text", "name": "Quux" } Thursday, September 5, 13 Version 2 this time...
45. { "_id": "one", "_index": "foo", "_type": "t", "_version": 4, "ok": true } Thursday, September 5, 13
46. ?consistency=... Thursday, September 5, 13 How many replicas need to respond before success?
47. ?replication=... Thursday, September 5, 13 Flag to control replication, which is sync by default
48. Case Study Thursday, September 5, 13
49. Thursday, September 5, 13
50. Thursday, September 5, 13 Lightweight work queue * No new deployment (already have nice distributed data store) * Can lose some perf during consume (publish a bunch and walk away, workers busy anyway) * Just want a highly available place to store messages
51. No New Deployment Thursday, September 5, 13 Lightweight work queue * No new deployment (already have nice distributed data store) * Can lose some perf during consume (publish a bunch and walk away, workers busy anyway) * Just want a highly available place to store messages
52. No New Deployment Consume perf loss OK Thursday, September 5, 13 Lightweight work queue * No new deployment (already have nice distributed data store) * Can lose some perf during consume (publish a bunch and walk away, workers busy anyway) * Just want a highly available place to store messages
53. No New Deployment Consume perf loss OK Don’t really care about AMQP Thursday, September 5, 13 Lightweight work queue * No new deployment (already have nice distributed data store) * Can lose some perf during consume (publish a bunch and walk away, workers busy anyway) * Just want a highly available place to store messages
54. Rabbit Elasticsearch Exchange Index Queue Type Routing key ? Thursday, September 5, 13 We don’t care about routing it to any particular place.... just store it and we can find it
55. news.politics.uk news.sports news.politics.usa news.* trade.dow trade.nasdaq Thursday, September 5, 13 Routing keys are just queues. Queries can be anything, in this case prefix matching a few types.
56. Search! Thursday, September 5, 13 * Can create different exchanges with different shard configurations and risk profiles * Can look for newest message in queue, or ones from Tuesday * Can look for ones containing the job referencing cats
57. POST /exch/test.foo { "op": "frob", "thing": "/over/there" } Publish... Thursday, September 5, 13 Consuming a message is a reindex as unack’ed, maintaining concurrency control Get back _id
58. GET /exch/test.foo/_search { "query": { "bool": { "must": [ { "constant_score": { "filter": { "missing": { "existence": true, "field": "__q_status", "null_value": true } } } }, { "prefix": { "_type": "test.foo" } } ] } }, "size": 1, "sort": [ { "__q_control": { "order": "desc" } } ] } Consume... Thursday, September 5, 13 Search looks for anything that’s not unacked or acked, just return one Get _id
59. PUT /exch/test.foo/UN1QU3ID?version=1&refresh=true { ... "__q_status": "unack" } Consume... Thursday, September 5, 13 Can now mark unack’ed, maintaining concurrency control
60. PUT /exch/test.foo/UN1QU3ID?version=2&refresh=true { ... "__q_status": "ack" } Finish up... Thursday, September 5, 13 When we’re done, we ack it... Won’t show up in consumption queries *** Remember, this works because when we try to unack (start working), we will get a failure if someone else already has it. Not free to work until successful unack.
61. Performance Thursday, September 5, 13 Publishing... ES is a champ 20k msgs/sec on little 4-core MBA Consuming... constant refresh is slow... segment explosion, lots of merging, but still 50-100 msgs/sec In the typical work queue scenario, adding a few ms or secs to a job that may run for minutes is acceptable
62. Conclusion Thursday, September 5, 13 * Full consistency is possible in ES, lots of trade-offs users can decide to make * Finding data is often more important and nuanced than storing it. Search is king. * RDBMS not going anywhere, neither are graph databases. Right tool yada... * ?