Scalar DL is a Byzantine fault detection middleware for transactional database systems.
- https://github.com/scalar-labs/scalardl
ScalarDL Technical Overview1 Oct, 20221
View Slide
© 2022 Scalar, inc.ConfidentialScalarDL: Byzantine Fault Detection Middleware2Scalar DBStorages / DatabasesApplicationsOSScalar DLLedgerAuditorClient SDKContract FunctionScalable and Practical Byzantine Fault Detection Middlewarefor Transactional Database Systems
© 2022 Scalar, inc.ConfidentialKey Characteristics in ScalarDL3Tamper-evident Scalable CorrectDatabase-agnostic Cloud-agnosticByzantine faults can be detectedas long as N > fPerformance and availabilitycan be linearly scaledNo intermediate states are visible,and data is always up-to-dateNo dependency onparticular database productsNo dependency onparticular public cloudsN: # of administrative domains in adatabase system, f: # of faulty domainsCassandra, Amazon DynamoDB, AzureCosmos DB, and JDBC-supported majorRDBMSs (MySQL, PostgreSQL, OracleDatabase) are supported.AWS, Azure, EKS, and AKS aresupported.ACID-compliant and strong consistentdatabase system (i.e., strict serializable)DVi= DV / N・RF(RF: Replication Factor << N)
© 2022 Scalar, inc.ConfidentialScalarDL System Architecture4Pattern 1Pattern 1: Ledger only.ScalarDL provides data integrity in a databasesystem. But, it does not guarantee to detectByzantine faults.Pattern 2: Ledger+Auditor.ScalarDL guarantees to detect Byzantine faults.(as long as either Ledger or Auditor is correct.)PrimaryScalarDL LedgerPrimary DatabasesAdministrative domain 1Database SystemScalar DL Client SDKApplication programsPrimary SecondaryScalarDL LedgerPrimary DatabasesAdministrative domain 1Database SystemScalarDL AuditorSecondary DatabasesAdministrative domain 2
© 2022 Scalar, inc.ConfidentialScalarDL: Ledger5User (Client)JavaContractSignaturePrivate KeyScalarDL Ledgerfunction invoke() {if (accounts[0].data.balance < args.val) {throw new Error(“not enough balance”);}accounts[0].data.balance -= args.val;accounts[1].data.balance += args.val;results = { … };}Payment ContractRequest :(contract, args, sig)Asset ID Age Data (before) Data (after) Sigs Func (ref) Args HashA 1 { } { balance = 100,…}charge { val = 100 }B 1 { } { balance = 200,…}charge { val = 200 }A 2 { A: {balance = 100},B: {balance = 200} …}{ balance = 90,…}payment { val = 10 }SN= Func (SN-1, Args)Deterministic & TE TEIf S0is TE ⇒ SNis TEPublic KeyB 2 { A: {balance = 100},B: {balance = 200} …}{ balance = 210,…}payment { val = 10 }H(A1)H(B1)StatehashchainData integrity of ledger entries:Arguments{ accounts = [“A”, “B”], val = 10, …}* includes other accounts dataArgument
© 2022 Scalar, inc.ConfidentialScalarDL: Auditor6managed bya different ADExecuteAsset ProofRequest ProofAsset ProofTCompare todetect ByzantinefaultsAuditor manages the proofs of execution to recompute assets without trustingthe integrity of data given from Ledger (1 patented, 1 filed, accepted by PVLDB)User (Client)ScalarDL LedgerScalarDL AuditorSee our PVLDBpaper for moredetails
© 2022 Scalar, inc.ConfidentialKey Difference between Public Blockchains and Scalar DL7Public Blockchains(Ethereum …)ScalarDL• History diverges in normal cases– It diverges even if no maliciousactivities ⇒ Impossible toguarantee finality• History diverges only in abnormal cases– If it diverges, it is caused bymalicious activities ⇒ Possible toguarantee finalityIEEE ICDCS Workshop’16
© 2022 Scalar, inc.ConfidentialKey Difference between BFT/BFD SMR (Private Blockchains*) and Scalar DL8BFT/BFD SMR, PrivateBlockchainsScalarDLData is totally ordered⇒ Hard to parallelize/scaleData is partially ordered⇒ Easy to parallelize/scaleSequential processing is requiredParallelizable* Private blockchains are thought to be variants oftechniques called Byzantine-fault tolerance statemachine replication (BFT SMR)See our PVLDBpaper for moredetails
© 2022 Scalar, inc.ConfidentialUnique Features 1 : ACID Nested Contract Execution• Multiple contracts can be executed atomically9Contract 1TransactionTamper-evident ledgerContract 2Contract 3Atomicexecution
© 2022 Scalar, inc.ConfidentialUnique Features 2 : User-defined Function (patented)• Function is a business logic for creating deletable and easy to searchrecords• Remedy the downsides of ledger structure and tamper-evidence property10Mutable databaseFunctionContractTransactionTamper-evident ledgerAtomicexecutionBusiness logicEvidence that businesslogic is executed
© 2022 Scalar, inc.ConfidentialBenefits of User-defined Function11Not deletableNot searchable flexiblyNot tamper-evidentInconsistentTamper-evidentDeletableSearchable flexiblyConsistentApplicationsApplicationsBFT/BFD (e.g., Blockchain)DBApplications DB BFT/BFD (e.g., Blockchain)ApplicationsloglogMutable recordsTamper-evident recordsScalarDL Ledger Scalar DL AuditorReplicated in adecentralized way
© 2022 Scalar, inc.ConfidentialBenchmark Results (YCSB、TPC-C)• Benchmarked Systems: ScalarDL、BFD-SMR (PeerReviewTx) [SOSPʼ07]• Databases: PostgreSQL (1 node / AD), Cassandra (3 nodes / AD)• YCSB-F: 100M records, 1RMW (1tx: 1read 1write)• TPC-C: 100 warehouses, NewOrder-Payment (NP)• Environment: AWS. 2 ADs. c5d.4xlarge for each database instance.12YCBS-F (read-modify-write) TPC-C (NP)ScalarDL scaled its throughput as client threads increased.See our PVLDBpaper for moredetails
© 2022 Scalar, inc.ConfidentialScalability (TPC-C)13ScalarDL achieved near-linear scalability.See our PVLDBpaper for moredetails
© 2022 Scalar, inc.ConfidentialUse Cases : Where to use ScalarDL14• Data integrity is required– Data can not be altered or deleted maliciously• Scalability is required– Many requests from many end-users or devices• There is a main organization– And there are a few auditors• Byzantine fault detection is acceptableDigital evidence / Database forensicsOur focuses:
© 2022 Scalar, inc.ConfidentialFor More Details• ScalarDL internal mechanism, fundumental differencesbetween ScalarDL and other technologies/producuts.– https://dl.acm.org/doi/abs/10.14778/3523210.3523212– PVDLB is a top-tier international conference in databaseand data engineering.• GitHub (Usage)– https://github.com/scalar-labs/scalardl• Engineering Blog– https://medium.com/scalar-engineering15