Scalar DL Technical Overview

Scalar DL is a Byzantine fault detection middleware for transactional database systems.

- https://github.com/scalar-labs/scalardl

Transcript

1. ScalarDL Technical Overview 1 Oct, 2022 1
2. © 2022 Scalar, inc. Confidential ScalarDL: Byzantine Fault Detection Middleware 2 Scalar DB Storages / Databases Applications OS Scalar DL Ledger Auditor Client SDK Contract Function Scalable and Practical Byzantine Fault Detection Middleware for Transactional Database Systems
3. © 2022 Scalar, inc. Confidential Key Characteristics in ScalarDL 3 Tamper-evident Scalable Correct Database-agnostic Cloud-agnostic Byzantine faults can be detected as long as N > f Performance and availability can be linearly scaled No intermediate states are visible, and data is always up-to-date No dependency on particular database products No dependency on particular public clouds N: # of administrative domains in a database system, f: # of faulty domains Cassandra, Amazon DynamoDB, Azure Cosmos DB, and JDBC-supported major RDBMSs (MySQL, PostgreSQL, Oracle Database) are supported. AWS, Azure, EKS, and AKS are supported. ACID-compliant and strong consistent database system (i.e., strict serializable) DVi = DV / N・RF (RF: Replication Factor << N)
4. © 2022 Scalar, inc. Confidential ScalarDL System Architecture 4 Pattern 1 Pattern 1: Ledger only. ScalarDL provides data integrity in a database system. But, it does not guarantee to detect Byzantine faults. Pattern 2: Ledger+Auditor. ScalarDL guarantees to detect Byzantine faults. (as long as either Ledger or Auditor is correct.) Primary ScalarDL Ledger Primary Databases Administrative domain 1 Database System Scalar DL Client SDK Application programs Primary Secondary ScalarDL Ledger Primary Databases Administrative domain 1 Database System ScalarDL Auditor Secondary Databases Administrative domain 2
5. © 2022 Scalar, inc. Confidential ScalarDL: Ledger 5 User (Client) Java Contract Signature Private Key ScalarDL Ledger function invoke() { if (accounts[0].data.balance < args.val) { throw new Error(“not enough balance”); } accounts[0].data.balance -= args.val; accounts[1].data.balance += args.val; results = { … }; } Payment Contract Request : (contract, args, sig) Asset ID Age Data (before) Data (after) Sigs Func (ref) Args Hash A 1 { } { balance = 100, …} charge { val = 100 } B 1 { } { balance = 200, …} charge { val = 200 } A 2 { A: {balance = 100}, B: {balance = 200} …} { balance = 90, …} payment { val = 10 } SN = Func (SN-1 , Args) Deterministic & TE TE If S0 is TE ⇒ SN is TE Public Key B 2 { A: {balance = 100}, B: {balance = 200} …} { balance = 210, …} payment { val = 10 } H(A1) H(B1) State hash chain Data integrity of ledger entries: Arguments { accounts = [“A”, “B”], val = 10, …} * includes other accounts data Argument
6. © 2022 Scalar, inc. Confidential ScalarDL: Auditor 6 managed by a different AD Execute Asset Proof Request Proof Asset Proof T Compare to detect Byzantine faults Auditor manages the proofs of execution to recompute assets without trusting the integrity of data given from Ledger (1 patented, 1 filed, accepted by PVLDB) User (Client) ScalarDL Ledger ScalarDL Auditor See our PVLDB paper for more details
7. © 2022 Scalar, inc. Confidential Key Difference between Public Blockchains and Scalar DL 7 Public Blockchains (Ethereum …) ScalarDL • History diverges in normal cases – It diverges even if no malicious activities ⇒ Impossible to guarantee finality • History diverges only in abnormal cases – If it diverges, it is caused by malicious activities ⇒ Possible to guarantee finality IEEE ICDCS Workshop’16
8. © 2022 Scalar, inc. Confidential Key Difference between BFT/BFD SMR (Private Blockchains*) and Scalar DL 8 BFT/BFD SMR, Private Blockchains ScalarDL Data is totally ordered ⇒ Hard to parallelize/scale Data is partially ordered ⇒ Easy to parallelize/scale Sequential processing is required Parallelizable * Private blockchains are thought to be variants of techniques called Byzantine-fault tolerance state machine replication (BFT SMR) See our PVLDB paper for more details
9. © 2022 Scalar, inc. Confidential Unique Features 1 : ACID Nested Contract Execution • Multiple contracts can be executed atomically 9 Contract 1 Transaction Tamper-evident ledger Contract 2 Contract 3 Atomic execution
10. © 2022 Scalar, inc. Confidential Unique Features 2 : User-defined Function (patented) • Function is a business logic for creating deletable and easy to search records • Remedy the downsides of ledger structure and tamper-evidence property 10 Mutable database Function Contract Transaction Tamper-evident ledger Atomic execution Business logic Evidence that business logic is executed
11. © 2022 Scalar, inc. Confidential Benefits of User-defined Function 11 Not deletable Not searchable flexibly Not tamper-evident Inconsistent Tamper-evident Deletable Searchable flexibly Consistent Applications Applications BFT/BFD (e.g., Blockchain) DB Applications DB BFT/BFD (e.g., Blockchain) Applications log log Mutable records Tamper-evident records ScalarDL Ledger Scalar DL Auditor Replicated in a decentralized way
12. © 2022 Scalar, inc. Confidential Benchmark Results (YCSB、TPC-C) • Benchmarked Systems: ScalarDL、BFD-SMR (PeerReviewTx) [SOSPʼ07] • Databases: PostgreSQL (1 node / AD）, Cassandra （3 nodes / AD） • YCSB-F: 100M records, 1RMW (1tx: 1read 1write) • TPC-C: 100 warehouses, NewOrder-Payment (NP) • Environment: AWS. 2 ADs. c5d.4xlarge for each database instance. 12 YCBS-F (read-modify-write) TPC-C (NP) ScalarDL scaled its throughput as client threads increased. See our PVLDB paper for more details
13. © 2022 Scalar, inc. Confidential Scalability (TPC-C) 13 ScalarDL achieved near-linear scalability. See our PVLDB paper for more details
14. © 2022 Scalar, inc. Confidential Use Cases : Where to use ScalarDL 14 • Data integrity is required – Data can not be altered or deleted maliciously • Scalability is required – Many requests from many end-users or devices • There is a main organization – And there are a few auditors • Byzantine fault detection is acceptable Digital evidence / Database forensics Our focuses:
15. © 2022 Scalar, inc. Confidential For More Details • ScalarDL internal mechanism, fundumental differences between ScalarDL and other technologies/producuts. – https://dl.acm.org/doi/abs/10.14778/3523210.3523212 – PVDLB is a top-tier international conference in database and data engineering. • GitHub (Usage) – https://github.com/scalar-labs/scalardl • Engineering Blog – https://medium.com/scalar-engineering 15