Everything You Need to Know About NewSQL in 2020

The database is usually the heart of a software system. And there are many database technologies that we can pick from. In this talk, we’ll explore where RDBMS and NoSQL fall short and how NewSQL fills the gap. We’ll see what types of NewSQL databases exist and how they work. And we’ll also go over different NewSQL solutions that we can pick for our projects. By the end of the talk, we’ll have a good understanding of when and how to apply a NewSQL database in our big scale applications.

Transcript

1. Everything You Need to Know About NewSQL in 2020 Nikolay Stoitsev, Engineering Manager @ Halo DX
2. Journey
3. Relational DB NoSQL
4. database size fetched per second updated per second
5. ACID transactions
6. Atomicity
7. Consistency
8. Isolation
9. Durability
10. Scaling How to scale it???
11. Horizontal vs Vertical
15. “Bigger servers don’t exist” The SRE team
16. Horizontal Scaling 10TB
17. Horizontal Scaling 2.5TB 2.5TB 2.5TB 2.5TB
18. Partitioning & Routing
19. Replications ID 1 - 100k ID 1 - 100k ID 100k+ ID 100k+
20. Failure handling ID 1 - 100k ID 1 - 100k ID 100k+ ID 100k+ copy
21. Rebalancing 2.5TB 2.5TB 2.5TB 2.5TB 500GB 500GB 500GB 500GB
22. Horizontal scaling out of the box Meet NoSQL
23. F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A. Wallach, M. Burrows, T. Chandra, A. Fikes, and R. E. Gruber. Bigtable: A distributed storage system for structured data. ACM Trans. Comput. Syst., 26:4:1–4:26, June 2008.
24. G. DeCandia, D. Hastorun, M. Jampani, G. Kakulapati, A. Lakshman, A. Pilchin, S. Sivasubramanian, P. Vosshall, and W. Vogels. Dynamo: amazon’s highly available key-value store. SIGOPS Oper. Syst. Rev., 41:205–220, October 2007.
25. Our choice Apache Cassandra
26. Why Apache Cassandra? Sharding, Replication, Fault Tolerance Decentralized Multiple Data Centers Performance Proven in Production Gives Control
28. But no transactions :(
30. Consistent Hashing
31. Consistent Hashing Two Phase Commit
32. Consistent Hashing Two Phase Commit Three Phase Commit
33. Consistent Hashing Two Phase Commit Three Phase Commit Quorum
34. Consistent Hashing Two Phase Commit Three Phase Commit Quorum Eventual Consistency
35. Transactions + Scalability Meet NewSQL
36. https://ai.google/research/pubs/pub27898
37. Partitioning tables in tablets Tablet #1 Tablet #2 Tablet #3
38. Tablets are replicated Node Node Node Node Tablet #1 Tablet #1 Tablet #1
39. One leader for each tablet Leader Follower Follower Leader for tablet #1
40. Paxos to pick the leader Leader Follower Follower
41. Paxos to pick the leader Leader Follower Follower New Leader
42. Read queries
43. Read queries Leader Follower Follower
44. Read queries Leader Follower Follower Can I read? Here is the data I have. Is it latest?
45. Read queries Leader Follower Follower Can I read? Yes
46. Read queries Leader Follower Follower Can I read? Yes
47. Write queries
48. Write query Leader Follower Follower
49. Write query Leader Follower Follower
50. Write query Leader Follower Follower Write Write
51. Write query Leader Follower Follower Write Write Done Done
52. Write query Leader Follower Follower Write Write Done Done Done
53. Distributed transactions
54. Distributed transaction Leader Follower Follower Leader Follower Follower
55. Distributed transaction Leader Follower Follower Leader Follower Follower Paxos Magic
56. Distributed transaction Leader Follower Follower Leader Follower Follower Transaction Leader
57. Distributed transaction Leader Follower Follower Leader Follower Follower Transaction Leader
58. Distributed transaction Leader Follower Follower Leader Follower Follower Transaction Leader Write
59. Distributed transaction Leader Follower Follower Leader Follower Follower Transaction Leader Write Done
60. Are transactions ACID?
61. Atomicity
62. Durability
63. Consistency
64. Isolation :(
65. True Time API Atomic Clock GPS Clock
66. TrueTime API time
67. Types of NewSQL databases • • •
68. Databases with novel architecture CockroachDB • Open Source - https://github.com/cockroachdb/cockroach • Easy to setup • Horizontal scalability and high availability • Geo-partitioning and distribution of data • ACID transactions
69. How it works?
70. Calvin and FaunaDB • Scalable • ACID Transactions • One global consensus protocol • GraphQL • User defined functions • Commercial
71. Databases with novel architecture NuoDB • Horizontal scalability • High availability • ACID transactions • Separate Transaction Management and Storage Management • Used in banking • https://nuodb.com VoltDB • Horizontal scalability with Geo Replication • High availability • ACID transactions (Serialized) • In-memory database • Low latency • Used in many domains • https://www.voltdb.com/
72. Middlewares Apache Ignite • Peer-to-peer mesh network • Distributed queries • Distributed caching • Storage and processing framework • Horizontally scalable • Atomic types • Replicated data structures • https://ignite.apache.org Apache Trafodion • SQL query language on Apache HBase • Big data workloads • ACID transactions • https://trafodion.apache.org
73. Managed Cloud Databases Cloud Spanner • Horizontal scalability • High availability • ACID transactions • Planet scale • Widely used • https://cloud.google.com/s panner Azure Cosmos DB • Horizontal scalability • High availability • ACID transactions • Planet scale • Widely used • Multi-Model • https://azure.microsoft.com /en-us/services/cosmos-db
74. When to use NewSQL • • •
75. Summary RDBMS Easy to run Has transactions Can’t scale NoSQL Can scale Provides availability No transactions NewSQL Can Scale Has transactions Lack maturity
76. Nikolay Stoitsev Engineering Manager at Halo DX