Scalar DL Technical Overview

Scalar DL is a Byzantine fault detection middleware for transactional database systems.

- https://github.com/scalar-labs/scalardl

Transcript

1. ScalarDL Technical Overview
   1 Oct, 2022
   1
   

   View Slide

2. © 2022 Scalar, inc.
   Confidential
   ScalarDL: Byzantine Fault Detection Middleware
   2
   Scalar DB
   Storages / Databases
   Applications
   OS
   Scalar DL
   Ledger
   Auditor
   Client SDK
   Contract Function
   Scalable and Practical Byzantine Fault Detection Middleware
   for Transactional Database Systems
   

   View Slide

3. © 2022 Scalar, inc.
   Confidential
   Key Characteristics in ScalarDL
   3
   Tamper-evident Scalable Correct
   Database-agnostic Cloud-agnostic
   Byzantine faults can be detected
   as long as N > f
   Performance and availability
   can be linearly scaled
   No intermediate states are visible,
   and data is always up-to-date
   No dependency on
   particular database products
   No dependency on
   particular public clouds
   N: # of administrative domains in a
   database system, f: # of faulty domains
   Cassandra, Amazon DynamoDB, Azure
   Cosmos DB, and JDBC-supported major
   RDBMSs (MySQL, PostgreSQL, Oracle
   Database) are supported.
   AWS, Azure, EKS, and AKS are
   supported.
   ACID-compliant and strong consistent
   database system (i.e., strict serializable)
   DVi
   = DV / N・RF
   (RF: Replication Factor << N)
   

   View Slide

4. © 2022 Scalar, inc.
   Confidential
   ScalarDL System Architecture
   4
   Pattern 1
   Pattern 1: Ledger only.
   ScalarDL provides data integrity in a database
   system. But, it does not guarantee to detect
   Byzantine faults.
   Pattern 2: Ledger+Auditor.
   ScalarDL guarantees to detect Byzantine faults.
   (as long as either Ledger or Auditor is correct.)
   Primary
   ScalarDL Ledger
   Primary Databases
   Administrative domain 1
   Database System
   Scalar DL Client SDK
   Application programs
   Primary Secondary
   ScalarDL Ledger
   Primary Databases
   Administrative domain 1
   Database System
   ScalarDL Auditor
   Secondary Databases
   Administrative domain 2
   

   View Slide

5. © 2022 Scalar, inc.
   Confidential
   ScalarDL: Ledger
   5
   User (Client)
   Java
   Contract
   Signature
   Private Key
   ScalarDL Ledger
   function invoke() {
   if (accounts[0].data.balance < args.val) {
   throw new Error(“not enough balance”);
   }
   accounts[0].data.balance -= args.val;
   accounts[1].data.balance += args.val;
   results = { … };
   }
   Payment Contract
   Request :
   (contract, args, sig)
   Asset ID Age Data (before) Data (after) Sigs Func (ref) Args Hash
   A 1 { } { balance = 100,
   …}
   charge { val = 100 }
   B 1 { } { balance = 200,
   …}
   charge { val = 200 }
   A 2 { A: {balance = 100},
   B: {balance = 200} …}
   { balance = 90,
   …}
   payment { val = 10 }
   SN
   = Func (SN-1
   , Args)
   Deterministic & TE TE
   If S0
   is TE ⇒ SN
   is TE
   Public Key
   B 2 { A: {balance = 100},
   B: {balance = 200} …}
   { balance = 210,
   …}
   payment { val = 10 }
   H(A1)
   H(B1)
   State
   hash
   chain
   Data integrity of ledger entries:
   Arguments
   { accounts = [“A”, “B”], val = 10, …}
   * includes other accounts data
   Argument
   

   View Slide

6. © 2022 Scalar, inc.
   Confidential
   ScalarDL: Auditor
   6
   managed by
   a different AD
   Execute
   Asset Proof
   Request Proof
   Asset Proof
   T
   Compare to
   detect Byzantine
   faults
   Auditor manages the proofs of execution to recompute assets without trusting
   the integrity of data given from Ledger (1 patented, 1 filed, accepted by PVLDB)
   User (Client)
   ScalarDL Ledger
   ScalarDL Auditor
   See our PVLDB
   paper for more
   details
   

   View Slide

7. © 2022 Scalar, inc.
   Confidential
   Key Difference between Public Blockchains and Scalar DL
   7
   Public Blockchains
   (Ethereum …)
   ScalarDL
   • History diverges in normal cases
   – It diverges even if no malicious
   activities ⇒ Impossible to
   guarantee finality
   • History diverges only in abnormal cases
   – If it diverges, it is caused by
   malicious activities ⇒ Possible to
   guarantee finality
   IEEE ICDCS Workshop’16
   

   View Slide

8. © 2022 Scalar, inc.
   Confidential
   Key Difference between BFT/BFD SMR (Private Blockchains*) and Scalar DL
   8
   BFT/BFD SMR, Private
   Blockchains
   ScalarDL
   Data is totally ordered
   ⇒ Hard to parallelize/scale
   Data is partially ordered
   ⇒ Easy to parallelize/scale
   Sequential processing is required
   Parallelizable
   * Private blockchains are thought to be variants of
   techniques called Byzantine-fault tolerance state
   machine replication (BFT SMR)
   See our PVLDB
   paper for more
   details
   

   View Slide

9. © 2022 Scalar, inc.
   Confidential
   Unique Features 1 : ACID Nested Contract Execution
   • Multiple contracts can be executed atomically
   9
   Contract 1
   Transaction
   Tamper-evident ledger
   Contract 2
   Contract 3
   Atomic
   execution
   

   View Slide

10. © 2022 Scalar, inc.
    Confidential
    Unique Features 2 : User-defined Function (patented)
    • Function is a business logic for creating deletable and easy to search
    records
    • Remedy the downsides of ledger structure and tamper-evidence property
    10
    Mutable database
    Function
    Contract
    Transaction
    Tamper-evident ledger
    Atomic
    execution
    Business logic
    Evidence that business
    logic is executed
    

    View Slide

11. © 2022 Scalar, inc.
    Confidential
    Benefits of User-defined Function
    11
    Not deletable
    Not searchable flexibly
    Not tamper-evident
    Inconsistent
    Tamper-evident
    Deletable
    Searchable flexibly
    Consistent
    Applications
    Applications
    BFT/BFD (e.g., Blockchain)
    DB
    Applications DB BFT/BFD (e.g., Blockchain)
    Applications
    log
    log
    Mutable records
    Tamper-evident records
    ScalarDL Ledger Scalar DL Auditor
    Replicated in a
    decentralized way
    

    View Slide

12. © 2022 Scalar, inc.
    Confidential
    Benchmark Results (YCSB、TPC-C)
    • Benchmarked Systems: ScalarDL、BFD-SMR (PeerReviewTx) [SOSPʼ07]
    • Databases: PostgreSQL (1 node / AD）, Cassandra （3 nodes / AD）
    • YCSB-F: 100M records, 1RMW (1tx: 1read 1write)
    • TPC-C: 100 warehouses, NewOrder-Payment (NP)
    • Environment: AWS. 2 ADs. c5d.4xlarge for each database instance.
    12
    YCBS-F (read-modify-write) TPC-C (NP)
    ScalarDL scaled its throughput as client threads increased.
    See our PVLDB
    paper for more
    details
    

    View Slide

13. © 2022 Scalar, inc.
    Confidential
    Scalability (TPC-C)
    13
    ScalarDL achieved near-linear scalability.
    See our PVLDB
    paper for more
    details
    

    View Slide

14. © 2022 Scalar, inc.
    Confidential
    Use Cases : Where to use ScalarDL
    14
    • Data integrity is required
    – Data can not be altered or deleted maliciously
    • Scalability is required
    – Many requests from many end-users or devices
    • There is a main organization
    – And there are a few auditors
    • Byzantine fault detection is acceptable
    Digital evidence / Database forensics
    Our focuses:
    

    View Slide

15. © 2022 Scalar, inc.
    Confidential
    For More Details
    • ScalarDL internal mechanism, fundumental differences
    between ScalarDL and other technologies/producuts.
    – https://dl.acm.org/doi/abs/10.14778/3523210.3523212
    – PVDLB is a top-tier international conference in database
    and data engineering.
    • GitHub (Usage)
    – https://github.com/scalar-labs/scalardl
    • Engineering Blog
    – https://medium.com/scalar-engineering
    15
    

    View Slide