Scylla: Faster than a Speeding Byte

Scylla is a NoSQL database providing Apache Cassandra compatibility, distinguishing itself by supporting millions of operations per second, per node, with predictably low latency, on similar hardware. This talk will introduce Scylla, highlight some of its achievements, and lift the veil on the many design decisions, from programming model to the low-level, mechanical sympathetic details of its memory allocators.

Transcript

1. SCYLLA: Faster than a Speeding Byte! Duarte Nunes @duarte_nunes
2. ❏ Introducing Scylla ❏ System Overview ❏ Seastar ❏ Resource Management ❏ Workload Conditioning ❏ Closing AGENDA
3. Scylla • Clustered NoSQL database compatible with Apache Cassandra • ~10X performance on same hardware • Low latency, esp. higher percentiles • Self tuning • Mechanically sympathetic C++14
4. YCSB Benchmark: 3 node Scylla cluster vs 3, 9, 15, 30 Nodes Cassandra clusters 3 Scylla 30 Cassandra 3 Cassandra 3 Scylla 30 Cassandra 3 Cassandra
5. Scylla vs Cassandra - CL:LOCAL_QUORUM, Outbrain Case Study Scylla and Cassandra handling the full load (peak of ~12M RPM) 200ms 10ms 20x Lower Latency
6. Scylla benchmark by Samsung op/s Full report: http://tinyurl.com/msl-scylladb
7. Cassandra Compatibility • CQL language and protocol • Legacy Thrift protocol
8. Cassandra Compatibility • CQL language and protocol • Legacy Thrift protocol • SStable file format
9. Cassandra Compatibility • CQL language and protocol • Legacy Thrift protocol • SStable file format • Configuration file format • JMX management protocol • Management command line tool
10. Sharing the Ecosystem • Spark • Presto • JanusGraph • KairosDB
11. Monitoring
12. ❏ Introducing Scylla ❏ System Overview ❏ Seastar ❏ Resource Management ❏ Workload Conditioning ❏ Closing AGENDA
13. Dynamo-based system
14. Dynamo-based system • Masterless
15. Dynamo-based system • Masterless • Data is replicated across a set of replicas
16. Dynamo-based system • Masterless • Data is replicated across a set of replicas • Data is partitioned across all nodes
17. Dynamo-based system • Masterless • Data is replicated across a set of replicas • Data is partitioned across all nodes • An operation can specify a Consistency Level
18. CAP Theorem • Consistent under partitions ◦ i.e., Spanner, Zookeeper ◦ Unavailable ◦ Linearizability, single system image ◦ Expensive due to coordination
19. CAP Theorem • Available under partitions ◦ i.e., Scylla, Cassandra, Dynamo ◦ Local operations, asynchronous propagation ◦ Anomalies ◦ Requires repair ◦ More difficult to program ◦ Fast and highly available
20. Concurrent Updates R1 R2 1 2 2 1 set(1, ‘a’) set(1, ‘b’)
21. Concurrent Updates R1 R2 1 2 ? ? set(1, ‘a’) set(1, ‘b’) How to make concurrent updates commute?
22. Concurrent Updates R1 R2 1 2 2 2 set(1, ‘a’) set(1, ‘b’) max(ts) max(ts)
23. Diverging replicas & anti-entropy R1 R2 1 2 1 2 set(1, ‘a’) set(1, ‘b’)
24. Data model Partition Key1 Clustering Key1 Clustering Key1 Clustering Key2 Clustering Key2 ... ... ... ... ... CREATE TABLE playlists (id int, song_id int, title text, PRIMARY KEY (id, song_id)); INSERT INTO playlists (id, song_id, title) VALUES (62, 209466, 'Ænima’'); Sorted by Primary Key
25. Log-Structured Merge Tree SStable 1 Time
26. Log-Structured Merge Tree SStable 1 SStable 2 Time
27. Log-Structured Merge Tree SStable 1 SStable 2 SStable 3 Time
28. Log-Structured Merge Tree SStable 1 SStable 2 SStable 3 Time SStable 4
29. Log-Structured Merge Tree SStable 1 SStable 2 SStable 3 Time SStable 4 SStable 1+2+3
30. Log-Structured Merge Tree SStable 1 SStable 2 SStable 3 Time SStable 4 SStable 5 SStable 1+2+3
31. Log-Structured Merge Tree SStable 1 SStable 2 SStable 3 Time SStable 4 SStable 5 SStable 1+2+3 Foreground Job Background Job
32. Request path SSTable Memtable
33. Request path SSTable Memtable Reads
34. Request path SSTable Memtable Reads Commit Log Writes
35. Implementation Goals • Efficiency: ◦ Make the most out of every cycle • Utilization: ◦ Squeeze every cycle from the machine • Control ◦ Spend the cycles on what we want, when we want
36. ❏ Introducing Scylla ❏ System Overview ❏ Seastar ❏ Resource Management ❏ Workload Conditioning ❏ Closing AGENDA
37. • Thread-per-core design (shard) ◦ No blocking. Ever. Enter Seastar www.seastar-project.org
38. Enter Seastar www.seastar-project.org • Thread-per-core design (shard) ◦ No blocking. Ever. • Asynchronous networking, file I/O, multicore
40. Enter Seastar www.seastar-project.org • Thread-per-core design (shard) ◦ No blocking. Ever. • Asynchronous networking, file I/O, multicore • Future/promise based APIs
41. Enter Seastar www.seastar-project.org • Thread-per-core design (shard) ◦ No blocking. Ever. • Asynchronous networking, file I/O, multicore • Future/promise based APIs • Usermode TCP/IP stack included in the box
42. Seastar task scheduler Traditional stack Seastar stack Promise Task Promise Task Promise Task Promise Task CPU Promise Task Promise Task Promise Task Promise Task CPU Promise Task Promise Task Promise Task Promise Task CPU Promise Task Promise Task Promise Task Promise Task CPU Promise Task Promise Task Promise Task Promise Task CPU Promise is a pointer to eventually computed value Task is a pointer to a lambda function Scheduler CPU Scheduler CPU Scheduler CPU Scheduler CPU Scheduler CPU Thread Stack Thread Stack Thread Stack Thread Stack Thread Stack Thread Stack Thread Stack Thread Stack Thread is a function pointer Stack is a byte array from 64k to megabytes Context switch cost is high. Large stacks pollutes the caches No sharing, millions of parallel events
43. Seastar memcached
44. Pedis https://github.com/fastio/pedis
45. Futures future<> f = _conn->read_exactly(4).then([] (temporary_buffer<char> buf) { int id = buf_to_id(buf); unsigned core = id % smp::count; return smp::submit_to(core, [id] { return lookup(id); }).then([this] (sstring result) { return _conn->write(result); }); });
46. Futures future<> f = _conn->read_exactly(4).then([] (temporary_buffer<char> buf) { int id = buf_to_id(buf); unsigned core = id % smp::count; return smp::submit_to(core, [id] { return lookup(id); }).then([this] (sstring result) { return _conn->write(result); }); });
47. Futures future<> f = _conn->read_exactly(4).then([] (temporary_buffer<char> buf) { int id = buf_to_id(buf); unsigned core = id % smp::count; return smp::submit_to(core, [id] { return lookup(id); }).then([this] (sstring result) { return _conn->write(result); }); });
48. Futures future<> f = _conn->read_exactly(4).then([] (temporary_buffer<char> buf) { int id = buf_to_id(buf); unsigned core = id % smp::count; return smp::submit_to(core, [id] { return lookup(id); }).then([this] (sstring result) { return _conn->write(result); }); });
49. Futures future<> f = _conn->read_exactly(4).then([] (temporary_buffer<char> buf) { int id = buf_to_id(buf); unsigned core = id % smp::count; return smp::submit_to(core, [id] { return lookup(id); }).then([this] (sstring result) { return _conn->write(result); }); });
50. Futures future<> f = _conn->read_exactly(4).then([] (temporary_buffer<char> buf) { int id = buf_to_id(buf); unsigned core = id % smp::count; return smp::submit_to(core, [id] { return lookup(id); }).then([this] (sstring result) { return _conn->write(result); }); });
51. Seastar memory allocator • Non-Thread safe! ◦ Each core gets a private memory pool
52. Seastar memory allocator • Non-Thread safe! ◦ Each core gets a private memory pool • Allocation back pressure ◦ Allocator calls a callback when low on memory ◦ Scylla evicts cache in response
53. Seastar memory allocator • Non-Thread safe! ◦ Each core gets a private memory pool • Allocation back pressure ◦ Allocator calls a callback when low on memory ◦ Scylla evicts cache in response • Inter-core free() through message passing
54. ❏ Introducing Scylla ❏ System Overview ❏ Seastar ❏ Resource Management ❏ Workload Conditioning ❏ Closing AGENDA
55. Usermode I/O scheduler Query Commitlog Compaction Queue Queue Queue Userspace I/O Scheduler Disk
56. Figuring out optimal disk concurrency Max useful disk concurrency
57. Linux page cache Linux page cache SSTables • 4k granularity
58. Linux page cache Linux page cache SSTables • Parasitic rows Page (4k) Your data (300b)
59. Linux page cache Linux page cache SSTables • 4k granularity • Thread-safe
60. Linux page cache Linux page cache SSTables • 4k granularity • Thread-safe • Synchronous APIs
61. Linux page cache Linux page cache SSTables • Page faults Page fault Suspend thread Initiate I/O Context switch I/O completes Context switch Interrupt Map page Resume thread App thread Kernel SSD
62. Linux page cache Linux page cache SSTables • 4k granularity • Thread-safe • Synchronous APIs • General-purpose
63. Linux page cache Linux page cache SSTables • 4k granularity • Thread-safe • Synchronous APIs • General-purpose • Lack of control
64. Linux page cache Linux page cache SSTables • 4k granularity • Thread-safe • Synchronous APIs • General-purpose • Lack of control • Lack of control
65. Linux page cache Linux page cache SSTables • 4k granularity • Thread-safe • Synchronous APIs • General-purpose • Lack of control • Lack of control • ...on the other hand ◦ Exists ◦ Hundreds of man-years ◦ Handling lots of edge cases
66. Cassandra cache Key cache Row cache On-heap / Off-heap Linux page cache SSTables • Complex tuning
67. Scylla cache Unified cache SSTables
68. Probabilistic Cache Warmup • A replica with a cold cache should be sent less requests •
69. Yet another allocator (Problems with malloc/free) • Memory gets fragmented over time ◦ If the workload changes sizes of allocated objects ◦ Allocating a large contiguous block requires evicting most of cache
70. Memory
71. Memory
72. Memory
73. Memory
74. Memory
75. Memory
76. Memory
77. Memory
78. Memory
79. Memory
80. Memory
81. Memory
82. Memory
83. Memory
84. Memory
85. Memory
86. Memory OOM :(
87. Memory OOM :(
88. Memory OOM :(
89. Memory OOM :(
90. Memory OOM :(
91. Memory OOM :(
92. Memory OOM :(
93. Memory
94. Log-structured memory allocation • Bump-pointer allocation to current segment • Frees leave holes in segments • Compaction will try to solve this
95. Compacting LSA • Teach allocator how to move objects around ◦ Updating references • Garbage collect Compact! ◦ Starting with the most sparse segments ◦ Lock to pin objects • Used mostly for the cache ◦ Large majority of memory allocated ◦ Small subset of allocation sites
96. ❏ Introducing Scylla ❏ System Overview ❏ Seastar ❏ Resource Management ❏ Workload Conditioning ❏ Closing AGENDA
97. • Internal feedback loops to balance competing loads ◦ Consume what you export Workload Conditioning
98. Memtable Seastar Scheduler Compaction Query Repair Commitlog SSD WAN CPU Workload Conditioning
99. Memtable Seastar Scheduler Compaction Query Repair Commitlog SSD Compaction Backlog Monitor WAN CPU Workload Conditioning
100. Memtable Seastar Scheduler Compaction Query Repair Commitlog SSD Compaction Backlog Monitor WAN CPU Workload Conditioning Adjust priority
101. Memtable Seastar Scheduler Compaction Query Repair Commitlog SSD Memory Monitor WAN CPU Workload Conditioning
102. Memtable Seastar Scheduler Compaction Query Repair Commitlog SSD Memory Monitor Adjust priority WAN CPU Workload Conditioning
103. ❏ Introducing Scylla ❏ System Overview ❏ Seastar ❏ Resource Management ❏ Workload Conditioning ❏ Closing AGENDA
104. • Careful system design and control of the software stack can maximize throughput • Without sacrificing latency • Without requiring complex end-user tuning • While having a lot of fun Conclusions
105. • Download: http://www.scylladb.com • Twitter: @ScyllaDB • Source: http://github.com/scylladb/scylla • Mailing lists: scylladb-user @ groups.google.com • Slack: ScyllaDB-Users • Blog: http://www.scylladb.com/blog • Join: http://www.scylladb.com/company/careers • Me: duarte@scylladb.com How to interact
106. Questions? @duarte_nunes