Cassandra Python Driver: Benchmarking concurrency for the NYT ⨍aбrik Platform

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

Cassandra python driver Benchmarking concurrency for nyt⨍aбrik [email protected]

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

Oregon Dublin Tokyo São Paulo

Slide 10

Slide 10 text

Oregon Dublin Tokyo São Paulo a a a a b b b b c c

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

A Global Mesh with a Memory Message-‐based: WebSocket, AMQP, SockJS If in doubt: •  Resend •  Reconnect •  Reread Idempotent: •  Replicating •  Racy •  Resolving Classes of service: •  Gold: replicate/race •  Silver: prioritize •  Bronze: queueable Millions of users

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

Message: an event with data CREATE TABLE source_data ( hash_key int, -- real ones are more complex message_id timeuuid, body blob, -- whatever metadata text, -- JSON PRIMARY KEY (hash_key, message_id) );

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

Client Rabbit Cassandra

Slide 17

Slide 17 text

Client Rabbit Cassandra

Slide 18

Slide 18 text

Client Rabbit Cassandra 1-10kb 1-10kb Ack Ack Push

Slide 19

Slide 19 text

Client Rabbit Cassandra 1kb 1kb 10-150kb 10-150kb Pull Synchronous: C* Thrift or CQL Native

Slide 20

Slide 20 text

Client Rabbit Cassandra Concurrent Degree = 3 (using the Libev event Loop) Asynchronous: CQL Native only

Slide 21

Slide 21 text

Client Rabbit Cassandra More Concurrency Can also try: •  DC Aware •  Token Aware •  Subprocessing

Slide 22

Slide 22 text

Build one def build_message(self): message = { "message_id": str(uuid.uuid1()), "hash_key": randint(0, self._hash_key_range), # int(e ** 8) "app_id": self._app_id, "timestamp": datetime.utcnow().isoformat() + 'Z', "content_type": "application/binary", "body": os.urandom(randint(1, self._body_range)) # int(e ** 9) }

Slide 23

Slide 23 text

Kick-‐off def push_message(self): if self._submitted_count.next() < self._message_count: message = self.build_message() self.submit_query(message) def push_initial_data(self): self._start_time = time() try: with self._lock: for i in range( 0, min(CONCURRENCY, self._message_count) ): self.push_message()

Slide 24

Slide 24 text

Put it in the pipeline def submit_query(self, message): body = message.pop('body') substitution_args = ( json.dumps(message, **JSON_DUMPS_ARGS), body, message['hash_key'], uuid.UUID(message['message_id']) ) future = self._cql_session.execute_async( self._query, substitution_args ) future.add_callback(self.push_or_finish) future.add_errback(self.note_error)

Slide 25

Slide 25 text

Maintain concurrency or

Slide 26

Slide 26 text

Client Rabbit Cassandra 1-10kb 1-10kb Ack Ack Push

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

No content

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

No content

Slide 39

Slide 39 text

Push some messages usage: bm_push.py [-h] [-c [CQL_HOST [CQL_HOST ...]]] [-d LOCAL_DC] [--remote-dc-hosts REMOTE_DC_HOSTS] [-p PREFETCH_COUNT] [-w WORKER_COUNT] [-a] [-t] [-n {ONE, TWO, THREE, QUORUM, ALL, LOCAL_QUORUM, EACH_QUORUM, SERIAL, LOCAL_SERIAL, LOCAL_ONE}] [-r] [-j] [-l {CRITICAL,ERROR,WARNING,INFO,DEBUG,NOTSET}] Push messages from a RabbitMQ queue into a Cassandra table.

Slide 40

Slide 40 text

Push messages many times usage: run_push.py [-h] [-c [CQL_HOST [CQL_HOST ...]]] [-i ITERATIONS] [-d LOCAL_DC] [-w [worker_count [worker_count ...]]] [-p [prefetch_count [prefetch_count ...]]] [-n [level [level ...]]] [-a] [-t] [-m MESSAGE_EXPONENT] [-b BODY_EXPONENT] [-l {CRITICAL,ERROR,WARNING,INFO,DEBUG,NOTSET}] Run multiple test cases based upon the product of worker_counts, prefetch_counts, and consistency_levels. Each test case may be run with up to 4 variations reflecting the use or not of the dc_aware and token_aware policies. The results are output to stdout as a JSON object.