NoSQL and Cassandra

NoSQL and Cassandra introduction.

Transcript

1. NoSQL and Bin Liu 2017.7.7
2. ‣ Comment on mysqlperformanceblog.com Select queries were slow until I added an index onto the timestamp eld... Adding the index really helped our reporting, BUT now the inserts are taking forever. “ “
3. ‣ MySQL bug #9544 I'm trying to create indexes on a table with 308 million rows. It took ~20 minutes to load the table but 10 days to build indexes on it. “ “
4. Schema Scalability Performance 4
5. NoSQL KV Column-oriented Document Database Graph Database 5
6. NoSQL Redis BigTable Dynamo CouchDB Cassandra MongoDB Neo4j 6
7. CAP Consistency Availability Partition-tolerance 7
8. CP Bigtable/HBase Redis AP Dynamo/Cassandra CouchDB CA MySQL Postgre 8
9. Hbase vs Cassandra HBase Cassandra CAP cp ap sharding(Region) SPoF Y N HOP 1-3 1 Consistency Level OP W/R 9
10. Cassandra history 2007 Amazon Dynamo key/value store 2008 Facebook Dynamo ASF 10
11. Cassandra 11
12. Amazon Dynamo Consistent hashing Partitioning Replication One-hop routing Google BigTable Column Families Memtables SSTables 12
14. HBase 14
15. Cassandra 15
16. Virtual nodes: Partitioner: Replication strategy: Snitch: 16
17. Consistent hashing 17
18. Tokens row key -> partition key -> token 0 - 2127-1 18
19. HBase 19
20. Virtual nodes Vnodes 20
22. Partitioners distribute func: hash(row_key) -> token 22
23. Cassandra partitioners Murmur3Partitioner RandomPartitioner MD5 hash ByteOrderedPartitioner = 23
24. Murmur3Partitioner MurmurHash function 64-bit hash value of the partition key -263 to +263-1. RandomPartitioner 3-5 Auto loadblanace 24
25. RandomPartitioner MD5 hash value Cryptographic hash 0 to 2127 -1 Auto loadblanace 25
26. ByteOrderedPartitioner key Range scan 26
27. Snitches rack 27
28. Cassandra Replica 28
30. Cassandra Data model 30
31. Cassandra Data model keyspace vs database column family vs table super column column key 31
32. Keyspace key String value ColumnFamily[] 32
33. ColumnFamily key byte[] value Column[] 33
34. Column name(byte[]) value(byte[]) timestamp(long) 34
35. gossip P2P 35
36. SEED NODES 3 seed node 36
37. Storage engine Log-Structured Merge Tree I/O Read/Write 37
38. Data structure Memtable SSTable Bloom Filter Index Data Row Cache(LRU, CF data) Partition Key Cache(SSTable postion) Commit log(shared among tables) 38
39. RDMBS B+Tree for read O(log N) Sharding Scale out 39
40. SSTable row key bloom lter 40
42. Bloom Filter : O(k) 42
43. Bloom var: m,k: int hash_func: func(Object) -> n // 0 <= n < m f[k]: hash_func[] array_search[m], array_dict[m]:bit[] input: Object # write for i := range(k): array_dict[f[i](input)] = 1 # read for i := range(k): array_search[f[i](input)] = 1 if array_dict & array_search = array_search: // may be exists else: // not exists 43
44. Bloom m=10, k=3; f1= %10; f2= %10; f3= %10 A: (82,171,25) -> 2,1,5 B: (85,173,29) -> 5,3,9 C: (87,171,25) -> 7,1,5 D: (82,175,31) -> 2,5,1 Bit 0 1 2 3 4 5 6 7 8 9 A 0 1 1 0 0 1 0 0 0 0 B 0 0 0 1 0 1 0 0 0 1 Dict 0 1 1 1 0 1 0 0 0 1 C 0 1 0 0 0 1 0 1 0 0 D 0 1 1 0 0 1 0 0 0 0 44
46. Dictionary Encoding From: http://engineer.retty.me/entry/columnar-storage-format 46
47. Run-Length Encoding RLE From: http://engineer.retty.me/entry/columnar-storage-format 47
48. Delta Encoding From: http://engineer.retty.me/entry/columnar-storage-format 48
49. Double-delta-encoding 0: 0 [-63, 64]: 10 + D (7bits) [-255, 256]: 110 + D (9bits) [-2047, 2048]: 1110 + D (12bits) Other: 1111 + D(32bits) 49
50. XOR encoding x = value1 `XOR` value2 if x == 0: write 0 else: write 1 if this_meaningful_bits == prev_meaningful_bits: write 0 write this_meaningful_bits else: write 1 write length(prefix_zero) # 5 bits write length(this_meaningful_bits) # 6 bits write this_meaningful_bits 50
51. Gorilla timestamp and value 51
52. String CPU gzip lzo snappy 52
53. Write data 53
54. Compaction SSTable SSTable I/O 54
55. Reading Data 55
56. Consistency Eventual consistency Tunable(operation-level) consistency CP or AP 56
57. Strong consistency R + W > N R: W: N: ONE/ANY/QUORUM/ALL/... ... 57
58. Consistency (R + W > N) (R + W <= N) W=ONE,R=ALL;W=ALL,R=ONE 58
59. ? read repair hinted handoffs 59
60. SEDA(Staged Event-Driven Architecture) Thread + Event (Event Queue+Thread Pool) Thread pool by stage Event Driven between stage by message passing 60
61. Questions ? 61