The Path TowardsSimplifyingConsistency inDistributed Systems
View Slide
Caitie McCaffreyDistributed Systems Engineercaitiem.com@caitie
Distributed Systems EngineerFeral ConcurrencyControl ProgrammerCaitie McCaffrey
2015
2015“Application-levelMechanisms for maintainingdatabase integrity”
Halo 4 Statistics ServiceHalo 4 Statistics ServiceRecord & Aggregate StatisticsFrom Every Multiplayer HaloGame a User Plays
Halo 4 Statistics Service
Observability at TwitterMake Dashboards & AlertsAlways Available
Cmd Line Tool Viz / Dashboad Alerting SvcCuckoo-ReadCuckoo-WriteIndexing SvcRelay SvcTwitter Front EndTwitterSvcTwitterStatsiteTwitterSvcTwitterSvcScribeCollectionAgentHDFSManhattan DatabasePublic CloudObservability at Twitter
Notify Users WhenWe Receive AbuseReports
Notify Users If AnAction is Taken
ReportEvents StreamModificationEvents StreamEnrollmentEvents StreamListener ServiceDaemons ServiceExecutor ServiceNotification EventsStreamNotificationServiceManhattanMacaw
Feral ConcurrencyControl
Service ServiceServiceMonoliths
The Rise of NoSQL
Microservices
State of Affairs
LinearizableSequentialCausalPipelined RandomAccess MemoryRead YourWriteMonotonicReadMonotonicWriteWrite FromReadConsistencyModels
Can We DoBetter?
Finance & Accounting
VisiCalc1979
1983 Lotus 123
1987Excel
&StandardizationComputer AidedComputation
Standardizationfor Consistency in Distributed Systems
1988
1988“We believe the form ofcommunication that is needed isremote procedure call with at-most-once semantics”
1988“An adequate language mustprovide a modular, reasonablyautomatic method for achievingconsistency ”
Branch BAccount 789: $250Branch AAccount 123 : $100Account 456: $50SubAction: Deposit (123, 50)Transfer ActionSubAction: Withdraw(789, 50)TransferDeposit Withdrawenter topactioncoenteractionbranchA.Deposit(123, 50)actionbranchB.Withdraw(789, 50)endend
What aboutSpanner?
2015“The biggest barrier…is thatconsistency mechanisms mustintegrate across manystateful services”
CRDTs* Stolen from Chris Meiklejohnin practice
There Is No OneSize Fits AllConsistency Model
Mixed Consistency Data StoresManhattan
Computer AidedComputationfor Consistency in Distributed Systems
2016
Bank ApplicationBank Account must be > 0Deposit MoneyWithdrawal Money
‘Cause I’m Strong Enough: Reasoning About Consistency Choices in Distributed Systems
Define Invariants& ActionsDeposit MoneyWithdrawal MoneyDetermineConsistencyRequiredBank Account > 0Eventual ConsistencyStrong ConsistencyCodeGeneration?
Thank you@caitie