ScyllaDB: NoSQL at Ludicrous Speed

ScyllaDB is a NoSQL database providing Apache Cassandra compatibility, distinguishing itself by supporting millions of operations per second, per node, with predictably low latency, on similar hardware. This talk will cover the design decisions and techniques that enable such an achievement.

Transcript

1. SCYLLA: NoSQL at Ludicrous Speed Duarte Nunes @duarte_nunes
2. ❏ Introducing ScyllaDB ❏ Seastar ❏ Resource Management ❏ Workload Conditioning ❏ Closing AGENDA
3. ScyllaDB • Clustered NoSQL database compatible with Apache Cassandra • ~10X performance on same hardware • Low latency, esp. higher percentiles • Self tuning • Mechanically sympathetic C++14
4. YCSB Benchmark: 3 node Scylla cluster vs 3, 9, 15, 30 Cassandra machines 3 Scylla 30 Cassandra 3 Cassandra 3 Scylla 30 Cassandra 3 Cassandra
5. Scylla vs Cassandra - CL:LOCAL_QUORUM, Outbrain Case Study Scylla and Cassandra handling the full load (peak of ~12M RPM) 200ms 10ms 20x Lower Latency 5
6. Scylla benchmark by Samsung op/s Full report: http://tinyurl.com/msl-scylladb
7. Dynamo-based system
8. Data model Partition Key1 Clustering Key1 Clustering Key1 Clustering Key2 Clustering Key2 ... ... ... ... ... CREATE TABLE playlists (id int, song_id int, title text, PRIMARY KEY (id, song_id )); INSERT INTO playlists (id, song_id, title) VALUES (62, 209466, 'Ænima’'); Sorted by Primary Key
9. Log-Structured Merge Tree SStable 1 Time
10. Log-Structured Merge Tree SStable 1 SStable 2 Time
11. Log-Structured Merge Tree SStable 1 SStable 2 SStable 3 Time
12. Log-Structured Merge Tree SStable 1 SStable 2 SStable 3 Time SStable 4
13. Log-Structured Merge Tree SStable 1 SStable 2 SStable 3 Time SStable 4 SStable 1+2+3
14. Log-Structured Merge Tree SStable 1 SStable 2 SStable 3 Time SStable 4 SStable 5 SStable 1+2+3
15. Log-Structured Merge Tree SStable 1 SStable 2 SStable 3 Time SStable 4 SStable 5 SStable 1+2+3 Foreground Job Background Job
16. Request path SSTable Memtable
17. Request path SSTable Memtable Reads
18. Request path SSTable Memtable Reads Commit Log Writes
19. Implementation Goals • Efficiency: ◦ Make the most out of every cycle • Utilization: ◦ Squeeze every cycle from the machine • Control ◦ Spend the cycles on what we want, when we want
20. ❏ Introducing ScyllaDB ❏ System Architecture ❏ Node Architecture ❏ Seastar ❏ Resource Management ❏ Workload Conditioning ❏ Closing AGENDA
21. • Thread-per-core design (shard) ◦ No blocking. Ever. Enter Seastar www.seastar-project.org
22. Enter Seastar www.seastar-project.org • Thread-per-core design (shard) ◦ No blocking. Ever. • Asynchronous networking, file I/O, multicore
23. Enter Seastar www.seastar-project.org • Thread-per-core design (shard) ◦ No blocking. Ever. • Asynchronous networking, file I/O, multicore • Future/promise based APIs
24. Enter Seastar www.seastar-project.org • Thread-per-core design (shard) ◦ No blocking. Ever. • Asynchronous networking, file I/O, multicore • Future/promise based APIs • Usermode TCP/IP stack included in the box
25. Seastar task scheduler Traditional stack Seastar stack Promise Task Promise Task Promise Task Promise Task CPU Promise Task Promise Task Promise Task Promise Task CPU Promise Task Promise Task Promise Task Promise Task CPU Promise Task Promise Task Promise Task Promise Task CPU Promise Task Promise Task Promise Task Promise Task CPU Promise is a pointer to eventually computed value Task is a pointer to a lambda function Scheduler CPU Scheduler CPU Scheduler CPU Scheduler CPU Scheduler CPU Thread Stack Thread Stack Thread Stack Thread Stack Thread Stack Thread Stack Thread Stack Thread Stack Thread is a function pointer Stack is a byte array from 64k to megabytes Context switch cost is high. Large stacks pollutes the caches No sharing, millions of parallel events
26. Seastar memcached
27. Pedis https://github.com/fastio/pedis
28. Futures future<> f = _conn->read_exactly(4).then([] (temporary_buffer<char> buf) { int id = buf_to_id(buf); unsigned core = id % smp::count; return smp::submit_to(core, [id] { return lookup(id); }).then([this] (sstring result) { return _conn->write(result); }); });
29. Futures future<> f = _conn->read_exactly(4).then([] (temporary_buffer<char> buf) { int id = buf_to_id(buf); unsigned core = id % smp::count; return smp::submit_to(core, [id] { return lookup(id); }).then([this] (sstring result) { return _conn->write(result); }); });
30. Futures future<> f = _conn->read_exactly(4).then([] (temporary_buffer<char> buf) { int id = buf_to_id(buf); unsigned core = id % smp::count; return smp::submit_to(core, [id] { return lookup(id); }).then([this] (sstring result) { return _conn->write(result); }); });
31. Futures future<> f = _conn->read_exactly(4).then([] (temporary_buffer<char> buf) { int id = buf_to_id(buf); unsigned core = id % smp::count; return smp::submit_to(core, [id] { return lookup(id); }).then([this] (sstring result) { return _conn->write(result); }); });
32. Futures future<> f = _conn->read_exactly(4).then([] (temporary_buffer<char> buf) { int id = buf_to_id(buf); unsigned core = id % smp::count; return smp::submit_to(core, [id] { return lookup(id); }).then([this] (sstring result) { return _conn->write(result); }); });
33. Futures future<> f = _conn->read_exactly(4).then([] (temporary_buffer<char> buf) { int id = buf_to_id(buf); unsigned core = id % smp::count; return smp::submit_to(core, [id] { return lookup(id); }).then([this] (sstring result) { return _conn->write(result); }); });
34. No escaping the monad future<> f = …; f.get(); // not allowed
35. Unless... future<> f = seastar::async([&] { future<> f = …; f.get(); });
36. Unless... future<> f = seastar::async([&] { future<> f = …; f.get(); });
37. Seastar memory allocator • Non-Thread safe! ◦ Each core gets a private memory pool
38. Seastar memory allocator • Non-Thread safe! ◦ Each core gets a private memory pool • Allocation back pressure ◦ Allocator calls a callback when low on memory ◦ Scylla evicts cache in response
39. Seastar memory allocator • Non-Thread safe! ◦ Each core gets a private memory pool • Allocation back pressure ◦ Allocator calls a callback when low on memory ◦ Scylla evicts cache in response • Inter-core free() through message passing
40. ❏ Introducing ScyllaDB ❏ System Architecture ❏ Node Architecture ❏ Seastar ❏ Resource Management ❏ Workload Conditioning ❏ Closing AGENDA
41. Usermode I/O scheduler Storage Block Layer Filesystem
42. Usermode I/O scheduler Storage Block Layer Filesystem Disk I/O Scheduler Class A Class B
43. Usermode I/O scheduler Query Commitlog Compaction Queue Queue Queue Userspace I/O Scheduler Disk
44. Figuring out optimal disk concurrency Max useful disk concurrency
45. Cassandra cache Linux page cache SSTables • 4k granularity • Thread-safe • Synchronous APIs • General-purpose • Lack of control2 • ...on the other hand ◦ Exists ◦ Hundreds of man-years ◦ Handling lots of edge cases
46. Cassandra cache Linux page cache SSTables • Parasitic rows SSTable page (4k) Your data (300b)
47. Cassandra cache Linux page cache SSTables • Page faults Page fault Suspend thread Initiate I/O Context switch I/O completes Context switch Interrupt Map page Resume thread App thread Kernel SSD
48. Cassandra cache Key cache Row cache On-heap / Off-heap Linux page cache SSTables • Complex tuning
49. Scylla cache Unified cache SSTables
50. Probabilistic Cache Warmup • A replica with a cold cache should be sent less requests •
51. Yet another allocator (Problems with malloc/free) • Memory gets fragmented over time ◦ If the workload changes sizes of allocated objects ◦ Allocating a large contiguous block requires evicting most of cache
52. Memory
53. Memory
54. Memory
55. Memory
56. Memory
57. Memory
58. Memory
59. Memory
60. Memory
61. Memory
62. Memory
63. Memory
64. Memory
65. Memory
66. Memory
67. Memory
68. Memory OOM :(
69. Memory OOM :(
70. Memory OOM :(
71. Memory OOM :(
72. Memory OOM :(
73. Memory OOM :(
74. Memory OOM :(
75. Memory
76. Log-structured memory allocation • Bump-pointer allocation to current segment • Frees leave holes in segments • Compaction will try to solve this
77. Compacting LSA • Teach allocator how to move objects around ◦ Updating references • Garbage collect Compact! ◦ Starting with the most sparse segments ◦ Lock to pin objects • Used mostly for the cache ◦ Large majority of memory allocated ◦ Small subset of allocation sites
78. ❏ Introducing ScyllaDB ❏ System Architecture ❏ Node Architecture ❏ Seastar ❏ Resource Management ❏ Workload Conditioning ❏ Closing AGENDA
79. • Internal feedback loops to balance competing loads ◦ Consume what you export Workload Conditioning
80. Memtable Seastar Scheduler Compaction Query Repair Commitlog SSD WAN CPU Workload Conditioning
81. Memtable Seastar Scheduler Compaction Query Repair Commitlog SSD Compaction Backlog Monitor WAN CPU Workload Conditioning
82. Memtable Seastar Scheduler Compaction Query Repair Commitlog SSD Compaction Backlog Monitor WAN CPU Workload Conditioning Adjust priority
83. Memtable Seastar Scheduler Compaction Query Repair Commitlog SSD Memory Monitor WAN CPU Workload Conditioning
84. Memtable Seastar Scheduler Compaction Query Repair Commitlog SSD Memory Monitor Adjust priority WAN CPU Workload Conditioning
85. ❏ Introducing ScyllaDB ❏ System Architecture ❏ Node Architecture ❏ Seastar ❏ Workload Conditioning ❏ Closing AGENDA
86. • Careful system design and control of the software stack can maximize throughput • Without sacrificing latency • Without requiring complex end-user tuning • While having a lot of fun Conclusions
87. • Download: http://www.scylladb.com • Twitter: @ScyllaDB • Source: http://github.com/scylladb/scylla • Mailing lists: scylladb-user @ groups.google.com • Slack: ScyllaDB-Users • Blog: http://www.scylladb.com/blog • Join: http://www.scylladb.com/company/careers • Me: duarte@scylladb.com How to interact
88. SCYLLA, NoSQL at Ludicrous Speed Thank you. @duarte_nunes