Everything You Need to Know About NewSQL in 2020

by Nikolay Stoitsev

Slide 1

Slide 1 text

Everything You Need to Know About NewSQL in 2020 Nikolay Stoitsev, Engineering Manager @ Halo DX

Slide 2

Slide 2 text

Journey

Slide 3

Slide 3 text

Relational DB NoSQL

Slide 4

Slide 4 text

database size fetched per second updated per second

Slide 5

Slide 5 text

ACID transactions

Slide 6

Slide 6 text

Atomicity

Slide 7

Slide 7 text

Consistency

Slide 8

Slide 8 text

Isolation

Slide 9

Slide 9 text

Durability

Slide 10

Slide 10 text

Scaling How to scale it???

Slide 11

Slide 11 text

Horizontal vs Vertical

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

“Bigger servers don’t exist” The SRE team

Slide 16

Slide 16 text

Horizontal Scaling 10TB

Slide 17

Slide 17 text

Horizontal Scaling 2.5TB 2.5TB 2.5TB 2.5TB

Slide 18

Slide 18 text

Partitioning & Routing

Slide 19

Slide 19 text

Replications ID 1 - 100k ID 1 - 100k ID 100k+ ID 100k+

Slide 20

Slide 20 text

Failure handling ID 1 - 100k ID 1 - 100k ID 100k+ ID 100k+ copy

Slide 21

Slide 21 text

Rebalancing 2.5TB 2.5TB 2.5TB 2.5TB 500GB 500GB 500GB 500GB

Slide 22

Slide 22 text

Horizontal scaling out of the box Meet NoSQL

Slide 23

Slide 23 text

F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A. Wallach, M. Burrows, T. Chandra, A. Fikes, and R. E. Gruber. Bigtable: A distributed storage system for structured data. ACM Trans. Comput. Syst., 26:4:1–4:26, June 2008.

Slide 24

Slide 24 text

G. DeCandia, D. Hastorun, M. Jampani, G. Kakulapati, A. Lakshman, A. Pilchin, S. Sivasubramanian, P. Vosshall, and W. Vogels. Dynamo: amazon’s highly available key-value store. SIGOPS Oper. Syst. Rev., 41:205–220, October 2007.

Slide 25

Slide 25 text

Our choice Apache Cassandra

Slide 26

Slide 26 text

Why Apache Cassandra? Sharding, Replication, Fault Tolerance Decentralized Multiple Data Centers Performance Proven in Production Gives Control

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

But no transactions :(

Slide 29

Slide 29 text

No content

Slide 30

Slide 30 text

Consistent Hashing

Slide 31

Slide 31 text

Consistent Hashing Two Phase Commit

Slide 32

Slide 32 text

Consistent Hashing Two Phase Commit Three Phase Commit

Slide 33

Slide 33 text

Consistent Hashing Two Phase Commit Three Phase Commit Quorum

Slide 34

Slide 34 text

Consistent Hashing Two Phase Commit Three Phase Commit Quorum Eventual Consistency

Slide 35

Slide 35 text

Transactions + Scalability Meet NewSQL

Slide 36

Slide 36 text

https://ai.google/research/pubs/pub27898

Slide 37

Slide 37 text

Partitioning tables in tablets Tablet #1 Tablet #2 Tablet #3

Slide 38

Slide 38 text

Tablets are replicated Node Node Node Node Tablet #1 Tablet #1 Tablet #1

Slide 39

Slide 39 text

One leader for each tablet Leader Follower Follower Leader for tablet #1

Slide 40

Slide 40 text

Paxos to pick the leader Leader Follower Follower

Slide 41

Slide 41 text

Paxos to pick the leader Leader Follower Follower New Leader

Slide 42

Slide 42 text

Read queries

Slide 43

Slide 43 text

Read queries Leader Follower Follower

Slide 44

Slide 44 text

Read queries Leader Follower Follower Can I read? Here is the data I have. Is it latest?

Slide 45

Slide 45 text

Read queries Leader Follower Follower Can I read? Yes

Slide 46

Slide 46 text

Read queries Leader Follower Follower Can I read? Yes

Slide 47

Slide 47 text

Write queries

Slide 48

Slide 48 text

Write query Leader Follower Follower

Slide 49

Slide 49 text

Write query Leader Follower Follower

Slide 50

Slide 50 text

Write query Leader Follower Follower Write Write

Slide 51

Slide 51 text

Write query Leader Follower Follower Write Write Done Done

Slide 52

Slide 52 text

Write query Leader Follower Follower Write Write Done Done Done

Slide 53

Slide 53 text

Distributed transactions

Slide 54

Slide 54 text

Distributed transaction Leader Follower Follower Leader Follower Follower

Slide 55

Slide 55 text

Distributed transaction Leader Follower Follower Leader Follower Follower Paxos Magic

Slide 56

Slide 56 text

Distributed transaction Leader Follower Follower Leader Follower Follower Transaction Leader

Slide 57

Slide 57 text

Distributed transaction Leader Follower Follower Leader Follower Follower Transaction Leader

Slide 58

Slide 58 text

Distributed transaction Leader Follower Follower Leader Follower Follower Transaction Leader Write

Slide 59

Slide 59 text

Distributed transaction Leader Follower Follower Leader Follower Follower Transaction Leader Write Done

Slide 60

Slide 60 text

Are transactions ACID?

Slide 61

Slide 61 text

Atomicity

Slide 62

Slide 62 text

Durability

Slide 63

Slide 63 text

Consistency

Slide 64

Slide 64 text

Isolation :(

Slide 65

Slide 65 text

True Time API Atomic Clock GPS Clock

Slide 66

Slide 66 text

TrueTime API time

Slide 67

Slide 67 text

Types of NewSQL databases ● ● ●

Slide 68

Slide 68 text

Databases with novel architecture CockroachDB ● Open Source - https://github.com/cockroachdb/cockroach ● Easy to setup ● Horizontal scalability and high availability ● Geo-partitioning and distribution of data ● ACID transactions

Slide 69

Slide 69 text

How it works?

Slide 70

Slide 70 text

Calvin and FaunaDB ● Scalable ● ACID Transactions ● One global consensus protocol ● GraphQL ● User deﬁned functions ● Commercial

Slide 71

Slide 71 text

Databases with novel architecture NuoDB ● Horizontal scalability ● High availability ● ACID transactions ● Separate Transaction Management and Storage Management ● Used in banking ● https://nuodb.com VoltDB ● Horizontal scalability with Geo Replication ● High availability ● ACID transactions (Serialized) ● In-memory database ● Low latency ● Used in many domains ● https://www.voltdb.com/

Slide 72

Slide 72 text

Middlewares Apache Ignite ● Peer-to-peer mesh network ● Distributed queries ● Distributed caching ● Storage and processing framework ● Horizontally scalable ● Atomic types ● Replicated data structures ● https://ignite.apache.org Apache Trafodion ● SQL query language on Apache HBase ● Big data workloads ● ACID transactions ● https://trafodion.apache.org

Slide 73

Slide 73 text

Managed Cloud Databases Cloud Spanner ● Horizontal scalability ● High availability ● ACID transactions ● Planet scale ● Widely used ● https://cloud.google.com/s panner Azure Cosmos DB ● Horizontal scalability ● High availability ● ACID transactions ● Planet scale ● Widely used ● Multi-Model ● https://azure.microsoft.com /en-us/services/cosmos-db

Slide 74

Slide 74 text

When to use NewSQL ● ● ●

Slide 75

Slide 75 text

Summary RDBMS Easy to run Has transactions Can’t scale NoSQL Can scale Provides availability No transactions NewSQL Can Scale Has transactions Lack maturity

Slide 76

Slide 76 text

Nikolay Stoitsev Engineering Manager at Halo DX