Kick-off Meetup: Alex P " Cassandra internals, or how we ripped off Memtables"

Transcript

6. What’s inside?
8. Data Types ByteBuffer-backed ! Easy to Serialise Easy to compose AbstractType<T> implements Comparator<ByteBuffer> {! public abstract T compose(ByteBuffer bytes); ! public abstract ByteBuffer decompose(T value);! }
10. Commit log
11. Commit Log Quite straight forward Appends writes & mutations Flushes Writes based on strategy (periodic, size, etc ) Used as a tmp storage for HA & performance
12. Memtable Replayed (to Memtable) if node crashed Never accessed directly for reads Always accessed for writes Maintained until Memtable is flushed to SSTable
13. Memtable
14. Memtable
15. Memtable Not a row cache Intermediate location for data until flushed to SSTable Uses SkipListMap for CFname / CF lookup CF extends AbstractColumnContainer
16. Memtable Container holds SortedColumns (atomic / thread safe / array / tree-map backed etc) Atomic is implemented with SnapTreeMap navigatable map (range queries) atomic, consistent iteration very fast snapshots / clones !
17. Memtable SnapTreeMap relies on Comparator internally, simple comparison due to binary data type uniformity simple range queries (head/tail/from/to/including/excluding) 100% complies to Cassandra interface / guarantees
18. SSTable
19. SSTable Sorted String Table Stored on Disk Uses Bloom-filter to reduce disk lookups Writes sequentially (performant) Uses compaction to reduce overhead (configurable)
21. SSTable Immutable Indexed (data location by key held in memory) >= 1 per CF !
22. SSTable Compaction Merges SSTables Discards Tombstones, reclaims space Refreshes location index
23. Lessons? Making an embedded in-memory store basic data types are expressible in terms of byte-buffer memory allocation / access patterns are to be addressed maintain a Commit Log for durability in concurrent environments, use concurrent data structures use data structures that give maximum available features: sorted index per column would allow searches on arbitrary column levelling / compaction allows simplifying sequencing data use approximate DSs (appx. histograms, bloom filters, sketches) to reduce overhead, get some big-O guarantees
24. @ifesdjeen