A Preliminary Performance Evaluation of Cassandra Accord

Transcript

1. A Preliminary Performance Evaluation of Cassandra Accord Youki Shiraishi <youki.shiraishi@ntt.com> NTT, Inc. October 13, 2026 | Community Over Code 2026
2. Cassandra 6.0 and Accord Accord expands Cassandra’s transactional capabilities. Cassandra 6.0 introduces Accord consensus protocol, which enables strictly serializable transactions across tables and partitions. Leaderless consensus in production. Accord brings recent advances in leaderless consensus research into Cassandra: the first deployment of such a protocol in a widely used production database. Complex protocol, hard-to-predict performance. Accord combines a complex protocol with substantial optimizations: medium path, privileged coordinators, and ephemeral reads. Their interactions complicate performance prediction, motivating empirical evaluation. © 2026 NTT, Inc. 1
3. Contribution Research questions Evaluate Accord’s transaction performance under different scenarios, focusing on three key research questions. How well do Accord transactions scale? How do the transaction complexity and type affect performance? How does contention impact performance? Disclaimer This evaluation is based on Cassandra 6.0 alpha2. Behavior and performance may differ in the final release. © 2026 NTT, Inc. 2
4. Protocol Overview Consensus Phase Execution Phase Request(txn) Coordinator Reply(result) PreAccept(txn, t0) Ok(t1, deps) Accept(t*, merged deps) Ok(delta deps) Commit(t*, deps*) Ok() Stable/Read(t*, deps*) Ok([data]) Execute Apply(t*, deps*, outcome) Ok() Replica X1 Replica X2 Replica X3 Replica Y1 Replica Y2 Replica Y3 Fast Path Medium Path Slow Path Figure 1: Sequence diagram of read–write transaction across two shards. Read/Apply rounds may additionally wait for dependencies at replicas. Contention, clock skew, and message delay can cause fallback from fast to medium or slow path. © 2026 NTT, Inc. 3
5. Consensus Phase In the consensus phase, coordinators establish a total order on transactions by agreeing on execution timestamps and stabilize dependencies for safe execution. Fast Path: Coordinator sends PreAccept(txn, t0) to fast path electorates; every participating shard has a supermajority quorum (typically ⌊ 3𝑛 4 ⌋ + 1) supporting t0. Merge dependencies; skip Accept and Commit. Medium Path: Fast path failed, but every participating shard has a majority quorum (⌊ 𝑛2 ⌋ + 1) of eligible voters supporting t0. Run Accept to confirm t0 and dependencies; skip Commit. Slow Path: Otherwise, choose the maximum timestamp t1 returned from participants. Run Accept to agree on t1 and collect additional dependencies, then Commit the merged set, each with a majority quorum in every shard. © 2026 NTT, Inc. 4
6. Execution Phase In the execution phase, coordinator reads consistent data, computes the transaction’s result, and apply writes, using the agreed timestamp and dependencies. Read: Coordinator sends the agreed timestamp, dependencies, and Read requests to at least one replica in every shard. Each replica waits until preceding transactions’ writes are applied locally, then returns data. Execute: Once all required data is available, coordinator evaluates transaction logic to determine result and writes. Persist: Coordinator sends Apply to persist the writes and result at replicas; each replica applies the writes once its local dependencies are resolved. © 2026 NTT, Inc. 5
7. Optimization: Privileged Coordinator Benefit. Can reduce the required fast quorum, making fast path easier to achieve. Key idea. Coordinator votes locally before sending PreAccept to other replicas, then sends its vote and dependencies. For safety, it records the fast-path decision locally before propagation. Condition. Coordinator must be a fast-path voter for every participating shard. Local voting and execution must succeed. © 2026 NTT, Inc. 6
8. Optimization: Ephemeral Reads Benefit. Avoid consensus phase and Accept/Commit rounds, with no durable transaction record for the read. Key idea. Coordinator sends GetEphemeralReadDeps instead of PreAccept to collect dependencies from a quorum, waits for them to execute, and then reads. Other transactions do not track ephemeral reads as dependencies. Condition. Enabled for single-key or range reads. Single-key reads are strictly serializable; range reads provide per-key linearizability. © 2026 NTT, Inc. 7
9. Avoiding Excessive Backlog Replica signals soft rejection when the unapplied dependency count exceeds min_soft_reject_count=100 and at least one of these conditions holds1 : More than max_soft_reject_count=1000 unapplied dependencies. Oldest dependency’s age exceeds soft_reject_age=10s. Total dependency age exceeds soft_reject_cumulative_age=60s. Coordinator rejects the transaction if soft rejections reach hard_reject_ratio=0.5 of collected PreAccept replies. 1 All values shown are defaults. © 2026 NTT, Inc. 8
10. Implementation in Cassandra AccordService PreAccept, Accept, ... CommandStore #1 Table A: A1, A2 Other nodes Accord Node ⋮ Command Cache CFK Cache T1: Applied, ts=42, deps={} x: {T2} T2: Accepted, ts=83, deps={T1} y: {T1, T2} z: {} CommandStore #2 Table B: B1, B4 Command Cache CFK Cache CommandStore #3 Table B: B2, B3 Command Cache CFK Cache Segments Memtable Memtable SSTable User Table A Memtable SSTable User Table B ⋮ Compactor SSTable SSTable Journal (System) CFK (System) Figure 2: AccordNode coordinates and routes transactions; CommandStores manage local shards; Command cache holds transaction state; CommandsForKey cache tracks per-key conflicts and execution order; and Journal persists state for recovery. © 2026 NTT, Inc. 9
11. SimpleBank Benchmark SimpleBank is a set of banking transactions (Balance, Transfer, Payment, Audit, and Payroll) designed to benchmark Accord transaction performance. Each record stores the balance of a customer’s checking or savings account; each customer’s checking and savings accounts reside in the same partition. Like YCSB+T, it models a closed economy, i.e., all transactions should preserve a total balance across all accounts. CREATE TABLE accounts ( id bigint, kind text, -- 'CHECKING' or 'SAVINGS' balance bigint, PRIMARY KEY (id, kind) ) WITH transactional_mode = 'full'; Listing 1: SimpleBank schema. © 2026 NTT, Inc. 10
12. SimpleBank: Transaction Types Table 1: SimpleBank transactions. Transaction Type Description Balance Transfer Audit Payment Payroll 1-key read-only 1-key read-write 2-key read-only 2-key read-write 4-key read-write Read a customer’s checking and savings balances. Move funds from checking to savings. Read two customers’ checking balances. Transfer funds between two checking accounts. Transfer funds from one checking to three recipients. Customer selection. Customer IDs follow a Zipf distribution: 𝜃 = 0.0 is uniform; larger 𝜃 increases skew. Distinctness. Each transaction selects distinct customer IDs (partition keys): one for Balance/Transfer, two for Audit/Payment, and four for Payroll. © 2026 NTT, Inc. 11
13. SimpleBank: Balance & Transfer -- balance(a) BEGIN TRANSACTION LET chk = (SELECT balance FROM accounts WHERE id=<a> AND kind='CHECKING'); LET sav = (SELECT balance FROM accounts WHERE id=<a> AND kind='SAVINGS'); SELECT chk.balance, sav.balance; COMMIT TRANSACTION; -- transfer(a) BEGIN TRANSACTION UPDATE accounts SET balance -= 1 WHERE id=<a> AND kind='CHECKING'; UPDATE accounts SET balance += 1 WHERE id=<a> AND kind='SAVINGS'; COMMIT TRANSACTION; Listing 2: Balance and Transfer transactions. © 2026 NTT, Inc. 12
14. SimpleBank: Audit & Payment -- audit(a1, a2) BEGIN TRANSACTION LET a1 = (SELECT balance FROM accounts WHERE id=<a1> AND kind='CHECKING'); LET a2 = (SELECT balance FROM accounts WHERE id=<a2> AND kind='CHECKING'); SELECT a1.balance, a2.balance; COMMIT TRANSACTION; -- payment(a1, a2) BEGIN TRANSACTION UPDATE accounts SET balance -= 1 WHERE id=<a1> AND kind='CHECKING'; UPDATE accounts SET balance += 1 WHERE id=<a2> AND kind='CHECKING'; COMMIT TRANSACTION; Listing 3: Payment and Audit transactions. © 2026 NTT, Inc. 13
15. SimpleBank: Payroll -- payroll(a1, a2, a3, a4) BEGIN TRANSACTION UPDATE accounts SET balance -= 3 WHERE id=<a1> AND kind='CHECKING'; UPDATE accounts SET balance += 1 WHERE id=<a2> AND kind='CHECKING'; UPDATE accounts SET balance += 1 WHERE id=<a3> AND kind='CHECKING'; UPDATE accounts SET balance += 1 WHERE id=<a4> AND kind='CHECKING'; COMMIT TRANSACTION; Listing 4: Payroll transaction. © 2026 NTT, Inc. 14
16. Experimental Setup We used 3- or 6-node Cassandra clusters with SimpleStrategy. All cluster nodes and benchmark client ran on identically configured servers shown below and were located in the same rack. Table 2: Software configuration. OS Java Cassandra NoSQLBench © 2026 NTT, Inc. Ubuntu 26.04.1 OpenJDK 21 6.0 alpha2 5.25.17-release Table 3: Hardware configuration. CPU DRAM Storage Network AMD EPYC 9554P (64C/128T) 768 GiB DDR5-4800 NVMe SSD (R: 6800MB/s, W: 4000MB/s) 100 GbE 15
17. Preliminary Experiment: Stability Research question Do Accord transactions maintain stable performance under a heavy, long-running workload? Setup Dataset. 1M customers Workload. Transfer on uniform random keys in [0, 106 ) Duration. 30 minutes Cluster. 3-node (rf=3) Client threads. 128 (closed-loop) © 2026 NTT, Inc. 16
18. Remaining segments Throughput (tps) Preliminary Experiment: Stability 30K 24,903 tps 20K 10K 0 600 20,294 tps (-18.5%) 30 second average 5 minute average 684 All 3 nodes combined 400 200 00 5 10 15 20 Elapsed time (min) 25 30 Figure 3: Throughput and remaining journal segment count over time. © 2026 NTT, Inc. 17
19. Preliminary Experiment: Stability [deeper … [deep… [deeper callees co… [deepe… Compress… Rando… ScopedMemoryAccess… Compre… Rand… BufferMa… FileHa… ScopedMemoryAccess… Buffer… File… RandomAc… SSTabl… DirectByteBuffer.g… Random… SSTa… RandomAc… Abstra… DirectByteBuffer.g… [dee… Random… Abst… FileHandl… SSTabl… JournalKey$JournalKe… FileHan… SSTa… BtiTableReade… BtiTab… JournalKey$JournalKey… BtiTableR… BtiT… OnDiskIndex.compareWi… BtiTableReader.r… BtiTableReader.rowIte… StorageHook$1.makeRow… Comm… Tri… Appl… OnDiskIndex.binarySear… Sin… SinglePartitionReadCom… Mem… Appl… OnDiskIndex.lookUpAll SinglePartitionReadCommand.query… Colu… Appl… Segment.readAll SinglePartitionReadCommand.query… Cass… Acco… SinglePartitionReadCommand.queryM… Keys… Appl… Pre… AccordJournal.readAll SinglePartitionReadComma… Acco… SinglePartitionReadCommand.queryS… Keys… Appl… Pre… AccordJournal.readAll SinglePartitionReadComma… Acco… ReadCommand.executeLocally Muta… MapReduceCommandS… AccordJournal.loadDiffs ReadCommand.executeLocal… Acco… ReadCommand.executeLocally Muta… MapReduceCommandS… Accord… AccordJournal.load ReadCommand.executeLocal… Acco… TxnNamedRead$1.call TxnW… Acco… TxnNamedRead$1.call 0x00… Acco… AsyncCallbacks.callAndCallback Asyn… Acco… AsyncCallbacks$CallAndCallback.run Asyn… Accord… AccordExecutor$PlainChain.runInternal Appl… Journal.readAll StorageHook$1.ma… SinglePartitionR… SinglePartitionReadComma… AccordTask$ForFun… Accord… AccordTask$ForFun… Accord… AccordJournal.loadCommand AccordTask.runInternal AccordExecutor$SequentialExecu… AccordExecutor$SequentialQueue… Abst… Accord… Acco… Pre… Mut… AccordTask$For… Accor… TxnNamedRead$1.call 0x0… AccordTask$For… Accor… AsyncCallbacks.callAndCallba… Asy… AccordTask.runInternal AccordCache$CommandAdapte… AsyncCallbacks$CallAndCallba… Asy… AccordExecutor$Sequential… AccordExecutor$LoadRunnable… AccordExecutor$PlainChain.runInte… AccordExecutorLoops.lambda$wrap$0 AccordExecutorLoops.lambda$wrap$0 0x00007982f27dbac0.run 0x00007982f27dbac0.run FastThreadLocalRunnable.run FastThreadLocalRunnable.run Thread.runWith Thread.runWith Txn… MapReduceComma…Accor… AccordCommandStore.loadCo… AccordExecutorAbstractLockLoop$2.run TxnNamedRead$1.call Pre… MapReduceComma… AccordCache$CommandAdapte… AccordExecutorAbstractLockLoop$2.run Thread.run SinglePartitionReadComma… AccordExecutor$Sequential… Abs… Accor… Acc… Thread.run CassandraThread.run CassandraThread.run AccordExecutor CPU samples AccordExecutor CPU samples (a) First 60 seconds. (b) Last 60 seconds. Figure 4: Flame graphs captured during a long-running workload. In the last 60 seconds, the OnDiskIndex lookup path accounts for a larger share of CPU samples. © 2026 NTT, Inc. 18
20. Preliminary Experiment: Stability Implications Journal segment scans introduce significant overhead. Under heavy load, segment accumulation increases the cost of journal scans following cache misses, degrading performance. Default configuration, accord.journal.{compaction_period=60s, compact_max_segments=32}, may require tuning to prevent excessive segment buildup under heavy load. The 30 minute run left a total of 684 segments across three nodes. We set compaction_period=30s for subsequent experiments to avoid buildup. © 2026 NTT, Inc. 19
21. Evaluation: Scalability Research question How scalable is Accord under general workloads? Setup Dataset. 1M, 2M customers Workload. SimpleBank mixed workload (Balance 60%, Payment 25%, Transfer 10%, Audit 4%, and Payroll 1%) on random keys Duration. 300s warmup + 300s measurement Cluster. 3-node for 1M customers, 6-node for 2M customers (rf=3) Client threads. 1–1024 (closed-loop) © 2026 NTT, Inc. 20
22. Evaluation: Scalability 79,596 tps 80K N=3, SF=1M N=6, SF=2M Throughput (txn/s) 70K ~2x 60K 50K 39,489 tps 40K 30K 20K 10K 0 0 200 400 600 800 1000 Client Threads Figure 5: Scalability under SimpleBank mixed workload. © 2026 NTT, Inc. 21
23. Evaluation: Transaction Type and Complexity Research question How do the transaction type and complexity affect performance? Single-key vs multi-key? Read-only vs read-write? Setup Dataset. 100K customers Workloads. Transfer, Balance, Payment, Audit, and Payroll on random keys Duration. 300s warmup + 300s measurement Cluster. 3-node (rf=3) Input rate. 1K – 128K tps using 2048 threads (open-loop) © 2026 NTT, Inc. 22
24. Evaluation: Transaction Type and Complexity Balance Transfer Audit Payment Payroll Mean Latency (ms) 150 100 50 0 0 10K 20K 30K Throughput (tps) 64K 96K 128K Figure 6: Throughput-latency plot for five transaction types. Each point represents a measurement at a different input rate. © 2026 NTT, Inc. 23
25. Evaluation: Transaction Type and Complexity Implications Balance transactions (single-key ephemeral reads) achieve much higher throughput because they require neither consensus nor persistence. Transfer, Payment, and Payroll transactions (read-write, accessing one, two, and four keys, respectively) show that performance decreases as more keys are accessed. Audit transactions (2-key read-only) outperform Payment transactions (2-key read-write) because they have no writes. © 2026 NTT, Inc. 24
26. Evaluation: Contention Research question How does contention affect transaction performance? Setup Dataset. 100K customers Workloads. Transfer and Payment on Zipfian keys (varying exponent) Duration. 300s warmup + 300s measurement Cluster. 6-node (rf=3) Input rate. 1K – 16K tps using 2048 client threads (open-loop) © 2026 NTT, Inc. 25
27. 2,000 tps 4,000 tps 8,000 tps 12,000 tps 16,000 tps 16K 8K 0 100 50 Fast Path (%) 0 100 Failures (%) Latency Throughput (ms) (tps) 1,000 tps 0.5 95 90 85 0.25 0 0 0.25 0.5 0.75 1.0 Zipfian Exponent θ Figure 7: Transfer performance versus Zipfian exponent 𝜃 at different input rates. Throughput and latency are measured for successful transactions. No slow-path executions were observed.
28. 2,000 tps 4,000 tps 8,000 tps 12,000 tps 16,000 tps 16K 8K 0 2.0 1.0 Fast Path (%) 0.0 100 Failures (%) Latency Throughput (s) (tps) 1,000 tps 50 95 90 25 0 0 0.25 0.5 0.75 1.0 Zipfian Exponent θ Figure 8: Payment performance versus Zipfian exponent 𝜃 at different input rates. Throughput and latency are measured for successful transactions. No slow-path executions were observed.
29. Evaluation: Contention Implications Payment is more sensitive to skew than Transfer. Its cross-partition updates amplify dependencies and contention on hot records. (Also, the privileged coordinator optimization may contribute to Transfer’s resilience.) A high fast-path rate alone does not guarantee good performance. Dependency waits and resource contention may still delay execution. Higher input rates can severely degrade throughput under strong skew. Increased contention on hot records may amplify dependency waits and coordination overhead. Capacity depends on both input rate and skew. © 2026 NTT, Inc. 26
30. Conclusion Evaluation results Scalability. Doubling both cluster size and dataset size increased mixed-workload throughput by approximately 2x. Transaction type. Ephemeral reads achieved the highest throughput. Read-write transactions accessing more partitions had lower peak throughput. Contention. Strong skew substantially degraded performance despite a high fast-path rate. Future work Geo-distribution and clock skew. Evaluate how message delays and clock skew affect transaction performance. More on Scalability. Evaluate performance in larger clusters. © 2026 NTT, Inc. 27
31. Innovating a Sustainable Future for People and Planet