Elasticsearch & Data Consistency

Slide 1

Slide 1 text

Elasticsearch & Data Consistency @drewr Thursday, September 5, 13 * Introduction * Local training Oct 21-22, early bird ends Sept 21 * Why this topic? - I come from a mostly read-only archiving use, lots of people ask about update perf...

Slide 2

Slide 2 text

Search Thursday, September 5, 13 * Document-based (JSON) * Boolean, fuzzy suggesters, phrase, custom scoring * Lucene

Slide 3

Slide 3 text

Data Thursday, September 5, 13 * Keep a copy of the document * Don’t have to go to another data store

Slide 4

Slide 4 text

Fast Thursday, September 5, 13 * Distributed and concurrent from the ground up * Only locks for some disk writes

Slide 5

Slide 5 text

Distributed Thursday, September 5, 13

Slide 6

Slide 6 text

Thursday, September 5, 13 Shard, the irreducible container Need somewhere to store it -> node

Slide 7

Slide 7 text

Thursday, September 5, 13 Shard, the irreducible container Need somewhere to store it -> node

Slide 8

Slide 8 text

Thursday, September 5, 13

Slide 9

Slide 9 text

Thursday, September 5, 13 Node with two shards

Slide 10

Slide 10 text

Thursday, September 5, 13 Four....

Slide 11

Slide 11 text

Thursday, September 5, 13 Etc.

Slide 12

Slide 12 text

Thursday, September 5, 13

Slide 13

Slide 13 text

Thursday, September 5, 13 Cluster of one node

Slide 14

Slide 14 text

Thursday, September 5, 13 Two nodes

Slide 15

Slide 15 text

Thursday, September 5, 13 Etc.

Slide 16

Slide 16 text

Thursday, September 5, 13 Thirty-ﬁve-node cluster

Slide 17

Slide 17 text

Thursday, September 5, 13 Indices are a lightweight abstraction over shards This is a single index with two shards Add a node...

Slide 18

Slide 18 text

Thursday, September 5, 13 Nothing gained by moving anything to it (caveat with perf) Tell ES you want a copy of the data for availability

Slide 19

Slide 19 text

Thursday, September 5, 13

Slide 20

Slide 20 text

Thursday, September 5, 13 Two nodes, four shards For search, completely identical

Slide 21

Slide 21 text

P R R P Thursday, September 5, 13 Two nodes, four shards For search, completely identical

Slide 22

Slide 22 text

Requests Thursday, September 5, 13

Slide 23

Slide 23 text

Thursday, September 5, 13 Search...

Slide 24

Slide 24 text

Thursday, September 5, 13 * Writes are always indexing operations * Doc comes in with ID, or we generate one, hash it (or routing value), give it a unique shard

Slide 25

Slide 25 text

Thursday, September 5, 13 * Primary then replicas * What happens when a search request comes in? * Might choose primary or replica (can tell it primary, but you may not care)

Slide 26

Slide 26 text

Thursday, September 5, 13 * Primary then replicas * What happens when a search request comes in? * Might choose primary or replica (can tell it primary, but you may not care)

Slide 27

Slide 27 text

? ? ? ? Thursday, September 5, 13 * Might choose primary or replica (can tell it primary, but you may not care) * What if replica doesn’t have the doc yet?

Slide 28

Slide 28 text

Consistency Thursday, September 5, 13 * Eventually consistent (optimistic replication, every copy will eventually see the update) * Crazy! I need transactions!

Slide 29

Slide 29 text

PUT /foo/t/one { "age": 35, "bio": "Some long text", "name": "Foo" } Thursday, September 5, 13 Let’s index a doc

Slide 30

Slide 30 text

Update http://www.elasticsearch.org/guide/reference/api/update/ Thursday, September 5, 13

Slide 31

Slide 31 text

POST /foo/t/one/_update { "doc": { "name": "Bar" } } Thursday, September 5, 13 * We replace just that ﬁeld * Save network... maybe we can sneak our update in!

Slide 32

Slide 32 text

POST /foo/t/one/_update { "params": { "ageinc": 1 }, "script": "ctx._source.age += ageinc" } Thursday, September 5, 13 * Or a script * Same effect...

Slide 33

Slide 33 text

PUT /foo/t/one { "age": 36, "bio": "Some long text", "name": "Bar" } Thursday, September 5, 13 Or we can just PUT it again

Slide 34

Slide 34 text

A write is a write... Thursday, September 5, 13 * Index, update, delete.... Same end result! * Immutability! * Optimistic concurrency control * Fast, no locks, no magic, HTTP is stateless anyway (ETags)

Slide 35

Slide 35 text

{ "_id": "one", "_index": "foo", "_type": "t", "_version": 1, "ok": true } Thursday, September 5, 13 Look at response again...

Slide 36

Slide 36 text

{ "_id": "one", "_index": "foo", "_type": "t", "_version": 1, "ok": true } Thursday, September 5, 13

Slide 37

Slide 37 text

PUT /foo/t/one { "age": 36, "bio": "Some long text", "name": "Baz" } Thursday, September 5, 13 PUT it again

Slide 38

Slide 38 text

{ "_id": "one", "_index": "foo", "_type": "t", "_version": 2, "ok": true } Thursday, September 5, 13

Slide 39

Slide 39 text

PUT /foo/t/one?version=1 { "age": 36, "bio": "Some long text", "name": "Baz" } Thursday, September 5, 13 PUT it again

Slide 40

Slide 40 text

{ "error": "VersionConflict... current [2], provided [1]]", "status": 409 } Thursday, September 5, 13

Slide 41

Slide 41 text

PUT /foo/t/one?version=2 { "age": 36, "bio": "Some long text", "name": "Baz" } Thursday, September 5, 13 Version 2 this time...

Slide 42

Slide 42 text

{ "_id": "one", "_index": "foo", "_type": "t", "_version": 3, "ok": true } Thursday, September 5, 13

Slide 43

Slide 43 text

GET/Search Thursday, September 5, 13 * Realtime/NRT

Slide 44

Slide 44 text

PUT /foo/t/one?version=3&refresh=true { "age": 36, "bio": "Some long text", "name": "Quux" } Thursday, September 5, 13 Version 2 this time...

Slide 45

Slide 45 text

{ "_id": "one", "_index": "foo", "_type": "t", "_version": 4, "ok": true } Thursday, September 5, 13

Slide 46

Slide 46 text

?consistency=... Thursday, September 5, 13 How many replicas need to respond before success?

Slide 47

Slide 47 text

?replication=... Thursday, September 5, 13 Flag to control replication, which is sync by default

Slide 48

Slide 48 text

Case Study Thursday, September 5, 13

Slide 49

Slide 49 text

Thursday, September 5, 13

Slide 50

Slide 50 text

Thursday, September 5, 13 Lightweight work queue * No new deployment (already have nice distributed data store) * Can lose some perf during consume (publish a bunch and walk away, workers busy anyway) * Just want a highly available place to store messages

Slide 51

Slide 51 text

No New Deployment Thursday, September 5, 13 Lightweight work queue * No new deployment (already have nice distributed data store) * Can lose some perf during consume (publish a bunch and walk away, workers busy anyway) * Just want a highly available place to store messages

Slide 52

Slide 52 text

No New Deployment Consume perf loss OK Thursday, September 5, 13 Lightweight work queue * No new deployment (already have nice distributed data store) * Can lose some perf during consume (publish a bunch and walk away, workers busy anyway) * Just want a highly available place to store messages

Slide 53

Slide 53 text

No New Deployment Consume perf loss OK Don’t really care about AMQP Thursday, September 5, 13 Lightweight work queue * No new deployment (already have nice distributed data store) * Can lose some perf during consume (publish a bunch and walk away, workers busy anyway) * Just want a highly available place to store messages

Slide 54

Slide 54 text

Rabbit Elasticsearch Exchange Index Queue Type Routing key ? Thursday, September 5, 13 We don’t care about routing it to any particular place.... just store it and we can ﬁnd it

Slide 55

Slide 55 text

news.politics.uk news.sports news.politics.usa news.* trade.dow trade.nasdaq Thursday, September 5, 13 Routing keys are just queues. Queries can be anything, in this case preﬁx matching a few types.

Slide 56

Slide 56 text

Search! Thursday, September 5, 13 * Can create different exchanges with different shard conﬁgurations and risk proﬁles * Can look for newest message in queue, or ones from Tuesday * Can look for ones containing the job referencing cats

Slide 57

Slide 57 text

POST /exch/test.foo { "op": "frob", "thing": "/over/there" } Publish... Thursday, September 5, 13 Consuming a message is a reindex as unack’ed, maintaining concurrency control Get back _id

Slide 58

Slide 58 text

GET /exch/test.foo/_search { "query": { "bool": { "must": [ { "constant_score": { "filter": { "missing": { "existence": true, "field": "__q_status", "null_value": true } } } }, { "prefix": { "_type": "test.foo" } } ] } }, "size": 1, "sort": [ { "__q_control": { "order": "desc" } } ] } Consume... Thursday, September 5, 13 Search looks for anything that’s not unacked or acked, just return one Get _id

Slide 59

Slide 59 text

PUT /exch/test.foo/UN1QU3ID?version=1&refresh=true { ... "__q_status": "unack" } Consume... Thursday, September 5, 13 Can now mark unack’ed, maintaining concurrency control

Slide 60

Slide 60 text

PUT /exch/test.foo/UN1QU3ID?version=2&refresh=true { ... "__q_status": "ack" } Finish up... Thursday, September 5, 13 When we’re done, we ack it... Won’t show up in consumption queries *** Remember, this works because when we try to unack (start working), we will get a failure if someone else already has it. Not free to work until successful unack.

Slide 61

Slide 61 text

Performance Thursday, September 5, 13 Publishing... ES is a champ 20k msgs/sec on little 4-core MBA Consuming... constant refresh is slow... segment explosion, lots of merging, but still 50-100 msgs/sec In the typical work queue scenario, adding a few ms or secs to a job that may run for minutes is acceptable

Slide 62

Slide 62 text

Conclusion Thursday, September 5, 13 * Full consistency is possible in ES, lots of trade-offs users can decide to make * Finding data is often more important and nuanced than storing it. Search is king. * RDBMS not going anywhere, neither are graph databases. Right tool yada... * ?