Scylla: NoSQL at Ludicrous Speed

Scylla is a NoSQL database compatible with Apache Cassandra, distinguishing itself by supporting millions of operations per second, per node, with predictably low latency, on similar hardware.
Achieving such speed requires a great deal of mechanical sympathy: ScyllaDB employs a totally asynchronous, share-nothing programming model, relies on its own memory allocators, and meticulously schedules all its IO requests.
In this talk we will go over some low-level details of the techniques involved and how they fully utilize the underlying hardware resources.

Transcript

1. Scylla: NoSQL at Ludicrous Speed Duarte Nunes @duarte_nunes
2. Agenda 2 o Introducing Scylla o System Architecture o Seastar o Resource Management o Autonomous Database o Closing
3. 3 + Clustered NoSQL database compatible with Apache Cassandra + ~10X performance on same hardware + Low latency, especially higher percentiles + Autonomous + Mechanically sympathetic C++17 Scylla 10000 foot view
4. 4 YCSB Benchmark Throughput 3 Scylla 30 Cassandra 3 Cassandra 9 Cassandra 15 Cassandra
5. 5 YCSB Benchmark Latency 3 Scylla 30 Cassandra 3 Cassandra
6. 6 + CQL language and protocol + Legacy Thrift protocol + SStable file format + Configuration file format + JMX management protocol + Management command line tool Cassandra Compatibility + Sharing the ecosystem + Spark + Presto + JanusGraph + KairosDB + Kong
7. 7 Monitoring 3 Scylla 30 Cassandra 3 Cassandra
8. 8 System Architecture
9. 9 Data Model Partition Key Clustering Key Values Vladimir Nabokov 1962, Pale Fire pages: 315, genre: ‘fiction’ Vladimir Nabokov 1972, Transparent Things pages: 144, genre: ‘fiction’ Haruki Murakami 1997, Wind-Up Bird Chronicle pages: 607, genre: ‘fiction’, translator: ‘Jay Rubin’ David Foster Wallace 1996, Infinite Jest pages: 1,079, genre: ‘fiction’ Fyodor Dostoevsky 1869, The Eternal Husband pages: 130, genre: ‘fiction’, translator: ‘Constance Garnett’ Fyodor Dostoevsky 1869, The Idiot pages: 720, genre: ‘fiction’, translator: ‘Pevear and Volokhonsky’ Fyodor Dostoevsky 1880, The Brothers Karamazov pages: 825, genre: ‘fiction’, translator: ‘Pevear and Volokhonsky’
10. 10 Partitioning Fyodor Dostoevsky -2729816749760468891 David Foster Wallace 519370469886941605
11. 11 Partitioning Fyodor Dostoevsky -2729816749760468891 David Foster Wallace 519370469886941605
12. 12 Partitioning Fyodor Dostoevsky -2729816749760468891 David Foster Wallace 519370469886941605
13. 13 Replication CLIENT REQUEST COORDINATOR REPLICA 1 REPLICA 2 REPLICA 3 cql rpc rpc rpc
14. 14 Consistency Last write wins
15. 15 Consistency Consistency à la carte ✓ ✓
16. 16 Consistency Consistency à la carte ✓ ✓ ✓
17. 17 Consistency Anti-entropy ✓ ✓ ✓ + Reconciliation + Request path + Read repair + Request path, probabilistic + Repair + Background process
18. 18 + High availability + Masterless system + Latency + Guaranteed to get the lowest the system can provide at any time Eventual consistency
19. 19 Storage Log-Structured Merge Tree SStable 1 Time
20. 20 Storage Log-Structured Merge Tree SStable 1 SStable 2 Time
21. 21 Storage Log-Structured Merge Tree SStable 1 SStable 2 SStable 3 Time
22. 22 Storage Log-Structured Merge Tree SStable 1 SStable 2 SStable 3 Time SStable 4
23. 23 Storage Log-Structured Merge Tree SStable 1 SStable 2 SStable 3 Time SStable 4 SStable 1+2+3
24. 24 Storage Log-Structured Merge Tree SStable 1 SStable 2 SStable 3 Time SStable 4 SStable 5 SStable 1+2+3
25. 25 Storage Log-Structured Merge Tree SStable 1 SStable 2 SStable 3 Time SStable 4 SStable 5 SStable 1+2+3 Foreground Job Background Job
26. 26 Request Path Reads Memtable SStables
27. 27 Request Path Reads Memtable Reads SStables
28. 28 Request Path Reads Memtable Reads Commit Log Writes SStables
29. 29 + Efficiency + Make the most out of every cycle + Utilization + Squeeze every cycle from the machine + Control + Spend the cycles on what we want, when we want Implementation Goals
30. Seastar http://seastar.io 30
31. 31 + Thread-per-core design + No blocking. Ever. + A “shard” in seastar lingo. Seastar C++ framework for high-performance servers
32. 32 + Thread-per-core design + No blocking. Ever. + A “shard” in seastar lingo. + Asynchronous + Networking Seastar C++ framework for high-performance servers
33. 33 + Thread-per-core design + No blocking. Ever. + A “shard” in seastar lingo. + Asynchronous + Networking + File I/O Seastar C++ framework for high-performance servers
34. 34 + Thread-per-core design + No blocking. Ever. + A “shard” in seastar lingo. + Asynchronous + Networking + File I/O + Multicore Seastar C++ framework for high-performance servers
35. 35 + Thread-per-core design + No blocking. Ever. + A “shard” in seastar lingo. + Asynchronous + Networking + File I/O + Multicore + Future/promise based APIs Seastar C++ framework for high-performance servers
36. 36 + Thread-per-core design + No blocking. Ever. + A “shard” in seastar lingo. + Asynchronous + Networking + File I/O + Multicore + Future/promise based APIs + Optional user-mode TCP/IP stack included Seastar C++ framework for high-performance servers
37. 37 Seastar task scheduler Traditional Stack Seastar Stack Promise Task Promise Task Promise Task Promise Task CPU Promise Task Promise Task Promise Task Promise Task CPU Promise Task Promise Task Promise Task Promise Task CPU Promise Task Promise Task Promise Task Promise Task CPU Promise Task Promise Task Promise Task Promise Task CPU Promise is a pointer to eventually computed value Task is a pointer to a lambda function Scheduler CPU Scheduler CPU Scheduler CPU Scheduler CPU Scheduler CPU Thread Stack Thread Stack Thread Stack Thread Stack Thread Stack Thread Stack Thread Stack Thread Stack Thread is a function pointer Stack is a byte array from 64k to megabytes Context switch cost is high. Large stacks pollutes the caches No sharing, millions of parallel events
38. 38 future<> f = _conn->read_exactly(4).then([] (temporary_buffer<char> buf) { int id = buf_to_id(buf); unsigned core = id % smp::count; return smp::submit_to(core, [id] { return lookup(id); }).then([this] (sstring result) { return _conn->write(result); }); }); Futures
39. 39 future<> f = _conn->read_exactly(4).then([] (temporary_buffer<char> buf) { int id = buf_to_id(buf); unsigned core = id % smp::count; return smp::submit_to(core, [id] { return lookup(id); }).then([this] (sstring result) { return _conn->write(result); }); }); Futures
40. 40 future<> f = _conn->read_exactly(4).then([] (temporary_buffer<char> buf) { int id = buf_to_id(buf); unsigned core = id % smp::count; return smp::submit_to(core, [id] { return lookup(id); }).then([this] (sstring result) { return _conn->write(result); }); }); Futures
41. 41 future<> f = _conn->read_exactly(4).then([] (temporary_buffer<char> buf) { int id = buf_to_id(buf); unsigned core = id % smp::count; return smp::submit_to(core, [id] { return lookup(id); }).then([this] (sstring result) { return _conn->write(result); }); }); Futures
42. 42 future<> f = _conn->read_exactly(4).then([] (temporary_buffer<char> buf) { int id = buf_to_id(buf); unsigned core = id % smp::count; return smp::submit_to(core, [id] { return lookup(id); }).then([this] (sstring result) { return _conn->write(result); }); }); Futures
43. 43 future<> f = _conn->read_exactly(4).then([] (temporary_buffer<char> buf) { int id = buf_to_id(buf); unsigned core = id % smp::count; return smp::submit_to(core, [id] { return lookup(id); }).then([this] (sstring result) { return _conn->write(result); }); }); Futures
44. 44 Memory allocator + Non-Thread safe! + Each core gets a private memory pool
45. 45 Memory allocator + Non-Thread safe! + Each core gets a private memory pool + Allocation back pressure + Allocator calls a callback when low on memory + Scylla evicts cache in response
46. 46 Memory allocator + Non-Thread safe! + Each core gets a private memory pool + Allocation back pressure + Allocator calls a callback when low on memory + Scylla evicts cache in response + Inter-core free() through message passing
47. 47 Resource Management
48. 48 User-mode I/O Scheduler Query Commitlog Compaction Queue Queue Queue Userspace I/O Scheduler Disk
49. 49 Measuring optimal concurrency
50. 50 Measuring optimal concurrency Max useful disk concurrency
51. 51 Linux page cache Linux page cache SSTables + 4k granularity
52. 52 Linux page cache Linux page cache SSTables + Parasitic rows Page (4k) Your data (300b)
53. 53 Linux page cache Linux page cache SSTables + 4k granularity + Thread-safe
54. 54 Linux page cache Linux page cache SSTables + 4k granularity + Thread-safe + Synchronous APIs
55. 55 Linux page cache Linux page cache SSTables + 4k granularity + Thread-safe + Synchronous APIs App thread Kernel SSD Page fault Suspend thread Initiate I/O Context switch I/O completes Interrupt Context switch Map page Resume thread
56. 56 Linux page cache Linux page cache SSTables + 4k granularity + Thread-safe + Synchronous APIs + General-purpose
57. 57 Linux page cache Linux page cache SSTables + 4k granularity + Thread-safe + Synchronous APIs + General-purpose + Lack of control
58. 58 Linux page cache Linux page cache SSTables + 4k granularity + Thread-safe + Synchronous APIs + General-purpose + Lack of control + Lack of control
59. 59 Linux page cache Linux page cache SSTables + 4k granularity + Thread-safe + Synchronous APIs + General-purpose + Lack of control + Lack of control + ...on the other hand + Exists + Hundreds of man-years + Handles lots of edge cases
60. 60 Unified cache Linux page cache SSTables Key cache Row cache On-heap / Off-heap Complex Tuning Cassandra
61. 61 Unified cache Linux page cache SSTables Key cache Row cache On-heap / Off-heap Complex Tuning Scylla Unified cache SSTables Cassandra
62. Heat-Weighted Load Balancing RF=3, CL=QUORUM CLIENT REQUEST COORDINATOR RESTART REPLICA 1 REPLICA 2 REPLICA 3 cql rpc rpc
63. Total Requests Served after Node Restart Before: After:
64. Cold cache warmup A replica with cold cache is sent less requests
65. 65 + Memory gets fragmented over time + If the workload changes sizes of allocated objects + Allocating a large contiguous block requires evicting most of cache Yet another allocator Issues with malloc/free
66. Memory 66
67. Memory 67
68. Memory 68
69. Memory 69
70. Memory 70
71. Memory 71
72. Memory 72
73. Memory 73
74. Memory 74
75. Memory 75
76. Memory 76
77. Memory 77
78. Memory 78
79. Memory 79
80. Memory 80
81. Memory 81
82. Memory OOM :( 82
83. Memory OOM :( 83 OOM :(
84. Memory OOM :( 84 OOM :(
85. Memory OOM :( 85 OOM :(
86. Memory OOM :( 86 OOM :(
87. Memory OOM :( 87 OOM :(
88. Memory OOM :( 88 OOM :(
89. Memory 89
90. 90 + Bump-pointer allocation to current segment + Frees leave holes in segments + Compaction will try to solve this Log-structured memory allocation
91. 91 + Teach allocator how to move objects around + Updating back-pointers + Garbage collect Compact! + Starting with the most sparse segments + Lock to pin objects + Used mostly for the cache and memtables + Large majority of memory allocated + Small subset of allocation sites Log-structured memory allocation
92. Autonomous Database 92
93. 93 Autonomous Control theory in practice Memtable Seastar Scheduler Compaction Query Repair Commitlog SSD WAN CPU
94. 94 Autonomous Control theory in practice Memtable Seastar Scheduler Compaction Query Repair Commitlog SSD Compaction Backlog Controller WAN CPU
95. 95 Autonomous Control theory in practice Memtable Seastar Scheduler Compaction Query Repair Commitlog SSD Compaction Backlog Controller Adjust priority WAN CPU
96. 96 Autonomous Control theory in practice Memtable Seastar Scheduler Compaction Query Repair Commitlog SSD Compaction Backlog Controller Memory Controller Adjust priority WAN CPU
97. 97 Autonomous Control theory in practice Memtable Seastar Scheduler Compaction Query Repair Commitlog SSD Compaction Backlog Controller Memory Controller Adjust priority Adjust priority WAN CPU
98. 98 Steady state Compaction backlog controller in practice
99. 99 Increased ingestion rate Compaction backlog controller in practice
100. 100 New steady state Compaction backlog controller in practice
101. 101 Latencies Compaction backlog controller in practice
102. 102 Closing
103. 103 + Careful system design and control of the software stack can maximize throughput + Without sacrificing latency + Without requiring complex end-user tuning + While having a lot of fun Conclusions
104. 104 + Download: http://www.scylladb.com + Twitter: @ScyllaDB + Source: http://github.com/scylladb/scylla + Mailing lists: scylladb-user@groups.google.com + Slack: ScyllaDB-Users + Blog: http://www.scylladb.com/blog + Join: http://www.scylladb.com/company/careers How to interact A potpourri of links
105. United States 1900 Embarcadero Road Palo Alto, CA 94303 Israel 11 Galgalei Haplada Herzelia, Israel www.scylladb.com @scylladb Thank You!
106. Questions? 106 duarte@scylladb.com @duarte_nunes