$30 off During Our Annual Pro Sale. View Details »

Scalar DL Technical Overview

Scalar DL Technical Overview

Scalar DL is a Byzantine fault detection middleware for transactional database systems.

- https://github.com/scalar-labs/scalardl

Scalar, Inc.

October 01, 2022
Tweet

More Decks by Scalar, Inc.

Other Decks in Technology

Transcript

  1. ScalarDL Technical Overview
    1 Oct, 2022
    1

    View Slide

  2. © 2022 Scalar, inc.
    Confidential
    ScalarDL: Byzantine Fault Detection Middleware
    2
    Scalar DB
    Storages / Databases
    Applications
    OS
    Scalar DL
    Ledger
    Auditor
    Client SDK
    Contract Function
    Scalable and Practical Byzantine Fault Detection Middleware
    for Transactional Database Systems

    View Slide

  3. © 2022 Scalar, inc.
    Confidential
    Key Characteristics in ScalarDL
    3
    Tamper-evident Scalable Correct
    Database-agnostic Cloud-agnostic
    Byzantine faults can be detected
    as long as N > f
    Performance and availability
    can be linearly scaled
    No intermediate states are visible,
    and data is always up-to-date
    No dependency on
    particular database products
    No dependency on
    particular public clouds
    N: # of administrative domains in a
    database system, f: # of faulty domains
    Cassandra, Amazon DynamoDB, Azure
    Cosmos DB, and JDBC-supported major
    RDBMSs (MySQL, PostgreSQL, Oracle
    Database) are supported.
    AWS, Azure, EKS, and AKS are
    supported.
    ACID-compliant and strong consistent
    database system (i.e., strict serializable)
    DVi
    = DV / N・RF
    (RF: Replication Factor << N)

    View Slide

  4. © 2022 Scalar, inc.
    Confidential
    ScalarDL System Architecture
    4
    Pattern 1
    Pattern 1: Ledger only.
    ScalarDL provides data integrity in a database
    system. But, it does not guarantee to detect
    Byzantine faults.
    Pattern 2: Ledger+Auditor.
    ScalarDL guarantees to detect Byzantine faults.
    (as long as either Ledger or Auditor is correct.)
    Primary
    ScalarDL Ledger
    Primary Databases
    Administrative domain 1
    Database System
    Scalar DL Client SDK
    Application programs
    Primary Secondary
    ScalarDL Ledger
    Primary Databases
    Administrative domain 1
    Database System
    ScalarDL Auditor
    Secondary Databases
    Administrative domain 2

    View Slide

  5. © 2022 Scalar, inc.
    Confidential
    ScalarDL: Ledger
    5
    User (Client)
    Java
    Contract
    Signature
    Private Key
    ScalarDL Ledger
    function invoke() {
    if (accounts[0].data.balance < args.val) {
    throw new Error(“not enough balance”);
    }
    accounts[0].data.balance -= args.val;
    accounts[1].data.balance += args.val;
    results = { … };
    }
    Payment Contract
    Request :
    (contract, args, sig)
    Asset ID Age Data (before) Data (after) Sigs Func (ref) Args Hash
    A 1 { } { balance = 100,
    …}
    charge { val = 100 }
    B 1 { } { balance = 200,
    …}
    charge { val = 200 }
    A 2 { A: {balance = 100},
    B: {balance = 200} …}
    { balance = 90,
    …}
    payment { val = 10 }
    SN
    = Func (SN-1
    , Args)
    Deterministic & TE TE
    If S0
    is TE ⇒ SN
    is TE
    Public Key
    B 2 { A: {balance = 100},
    B: {balance = 200} …}
    { balance = 210,
    …}
    payment { val = 10 }
    H(A1)
    H(B1)
    State
    hash
    chain
    Data integrity of ledger entries:
    Arguments
    { accounts = [“A”, “B”], val = 10, …}
    * includes other accounts data
    Argument

    View Slide

  6. © 2022 Scalar, inc.
    Confidential
    ScalarDL: Auditor
    6
    managed by
    a different AD
    Execute
    Asset Proof
    Request Proof
    Asset Proof
    T
    Compare to
    detect Byzantine
    faults
    Auditor manages the proofs of execution to recompute assets without trusting
    the integrity of data given from Ledger (1 patented, 1 filed, accepted by PVLDB)
    User (Client)
    ScalarDL Ledger
    ScalarDL Auditor
    See our PVLDB
    paper for more
    details

    View Slide

  7. © 2022 Scalar, inc.
    Confidential
    Key Difference between Public Blockchains and Scalar DL
    7
    Public Blockchains
    (Ethereum …)
    ScalarDL
    • History diverges in normal cases
    – It diverges even if no malicious
    activities ⇒ Impossible to
    guarantee finality
    • History diverges only in abnormal cases
    – If it diverges, it is caused by
    malicious activities ⇒ Possible to
    guarantee finality
    IEEE ICDCS Workshop’16

    View Slide

  8. © 2022 Scalar, inc.
    Confidential
    Key Difference between BFT/BFD SMR (Private Blockchains*) and Scalar DL
    8
    BFT/BFD SMR, Private
    Blockchains
    ScalarDL
    Data is totally ordered
    ⇒ Hard to parallelize/scale
    Data is partially ordered
    ⇒ Easy to parallelize/scale
    Sequential processing is required
    Parallelizable
    * Private blockchains are thought to be variants of
    techniques called Byzantine-fault tolerance state
    machine replication (BFT SMR)
    See our PVLDB
    paper for more
    details

    View Slide

  9. © 2022 Scalar, inc.
    Confidential
    Unique Features 1 : ACID Nested Contract Execution
    • Multiple contracts can be executed atomically
    9
    Contract 1
    Transaction
    Tamper-evident ledger
    Contract 2
    Contract 3
    Atomic
    execution

    View Slide

  10. © 2022 Scalar, inc.
    Confidential
    Unique Features 2 : User-defined Function (patented)
    • Function is a business logic for creating deletable and easy to search
    records
    • Remedy the downsides of ledger structure and tamper-evidence property
    10
    Mutable database
    Function
    Contract
    Transaction
    Tamper-evident ledger
    Atomic
    execution
    Business logic
    Evidence that business
    logic is executed

    View Slide

  11. © 2022 Scalar, inc.
    Confidential
    Benefits of User-defined Function
    11
    Not deletable
    Not searchable flexibly
    Not tamper-evident
    Inconsistent
    Tamper-evident
    Deletable
    Searchable flexibly
    Consistent
    Applications
    Applications
    BFT/BFD (e.g., Blockchain)
    DB
    Applications DB BFT/BFD (e.g., Blockchain)
    Applications
    log
    log
    Mutable records
    Tamper-evident records
    ScalarDL Ledger Scalar DL Auditor
    Replicated in a
    decentralized way

    View Slide

  12. © 2022 Scalar, inc.
    Confidential
    Benchmark Results (YCSB、TPC-C)
    • Benchmarked Systems: ScalarDL、BFD-SMR (PeerReviewTx) [SOSPʼ07]
    • Databases: PostgreSQL (1 node / AD), Cassandra (3 nodes / AD)
    • YCSB-F: 100M records, 1RMW (1tx: 1read 1write)
    • TPC-C: 100 warehouses, NewOrder-Payment (NP)
    • Environment: AWS. 2 ADs. c5d.4xlarge for each database instance.
    12
    YCBS-F (read-modify-write) TPC-C (NP)
    ScalarDL scaled its throughput as client threads increased.
    See our PVLDB
    paper for more
    details

    View Slide

  13. © 2022 Scalar, inc.
    Confidential
    Scalability (TPC-C)
    13
    ScalarDL achieved near-linear scalability.
    See our PVLDB
    paper for more
    details

    View Slide

  14. © 2022 Scalar, inc.
    Confidential
    Use Cases : Where to use ScalarDL
    14
    • Data integrity is required
    – Data can not be altered or deleted maliciously
    • Scalability is required
    – Many requests from many end-users or devices
    • There is a main organization
    – And there are a few auditors
    • Byzantine fault detection is acceptable
    Digital evidence / Database forensics
    Our focuses:

    View Slide

  15. © 2022 Scalar, inc.
    Confidential
    For More Details
    • ScalarDL internal mechanism, fundumental differences
    between ScalarDL and other technologies/producuts.
    – https://dl.acm.org/doi/abs/10.14778/3523210.3523212
    – PVDLB is a top-tier international conference in database
    and data engineering.
    • GitHub (Usage)
    – https://github.com/scalar-labs/scalardl
    • Engineering Blog
    – https://medium.com/scalar-engineering
    15

    View Slide