Exploring the Coordination Lattice: Selecting Concurrency Control for Distributed Applications Sreeja S. Nair and Marc Shapiro 08/10/2021 

Centralized Application 2 

Distributed Application 3 

Distributed Auction Sreeja Marc Gustavo 4 

Distributed Auction Sreeja Marc Gustavo 4 

100 Distributed Auction Sreeja Marc Gustavo 4 100 100 

100 Distributed Auction Sreeja Marc Gustavo 4 100 100 

100 Distributed Auction Sreeja Marc Gustavo 4 100 100 105 105 

100 Distributed Auction Sreeja Marc Gustavo 4 100 100 105 105 

100 Distributed Auction Sreeja Marc Gustavo 4 100 100 105 105 

100 Distributed Auction Sreeja Marc Gustavo 4 100 100 105 105 

Auction : Static Analysis Con fl icts detected 5 • Place bid - close auction auction • Place bid - unregister buyer buyer • Close auction - remove bid auction 

Auction : Static Analysis Con fl icts detected 5 • Place bid - close auction auction • Place bid - unregister buyer buyer • Close auction - remove bid auction Using Distributed Locks 

Constraints imposed by the system and application • Coordination to ensure safety • Skewed workload in locality and frequency • Network latency • Cost of locking Variables that can be controlled • Granularity • Type of lock - Mutual exclusion - Shared/exclusive lock • Placement of lock - Distance between client and lock 6 Selecting Ef fi cient Concurrency Control for Safety 

Coordination needed to ensure safety using locks 7 Safety constraint Place bid Close auction Unregister buyer Remove bid auction auction buyer 

Skewed in frequency per operation 8 Workload constraint Place bid Close auction Unregiste r buyer Remove bid 90% 1% 7% 2% Frequency of Place bid operation for a single auction Paris Houston Singapore 95% 5% 0% Skewed in frequency per replica 

Network latency 9 Network constraint Paris Houston Singapore 75 ms 55 ms 125 ms 

10 Lock cost constraint Function of network latency and computation Di ff erent for di ff erent placement; same for both lock modes 

Constraints Safety constraint • Locks to ensure safety Workload constraint • Skewed workload in locality and frequency Network constraint • Latency between replicas Lock cost constraint • Cost of locking as a function of computation and network latency 11 

12 Granularity variable Place bid Close auction Unregister buyer Remove bid auction buyer Place bid Close auction Unregister buyer Remove bid auction auction buyer Place bid Close auction Unregister buyer Remove bid auction buyer + auction Place bid Close auction Unregister buyer Remove bid buyer + auction Coarsening Coarsening Coarsening Coarsening 

Effect of granularity Coarsening Lock acquisition time Contention Place bid Close auction Unregister buyer Remove bid auction auction buyer Place bid Close auction Unregister buyer Remove bid auction buyer + auction Place bid Close auction Unregister buyer Remove bid buyer + auction 2 locks 1 lock 1 lock 13 

14 Mode of locking variable Mutual exclusion Shared/exclusive lock Place bid Close auction Remove bid auction Mutex Mutex Mutex Exclusive mode introduces contention Shared mode allows parallel execution Place bid Close auction Remove bid auction Shared Shared Exclusive Place bid Close auction Remove bid auction Exclusive Exclusive Shared 

14 Mode of locking variable Mutual exclusion Shared/exclusive lock Place bid Close auction Remove bid auction Mutex Mutex Mutex Exclusive mode introduces contention Shared mode allows parallel execution Place bid Close auction Remove bid auction Shared Shared Exclusive Place bid Close auction Remove bid auction Exclusive Exclusive Shared 

15 Lock placement variable Distance between client and lock affects lock acquisition time Paris Houston Singapore 95% 5% 0% Frequency of Place bid operation for a single auction 

Selecting Coordination Con fi guration Placement - Minimize acquisition cost Mode - Maximize parallelism allowed Granularity - Minimize execution time 16 

The Coordination Lattice Place bid Close auction Unregister buyer Remove bid a1 a2 b Coordination Element 17 b a1 a2 b a1a2 ba1 a2 ba1a2 

Coordination Element 18 b a1a2 Mode b unregister buyer place bid MX EX EX SX1 EX SH SX2 SH EX mode placement a1a2 MX SX1 SX2 H ouston Paris Singapore Mode a1a2 place bid close auction remove bid MX EX EX EX SX1 EX SH EX SX2 SH EX SH Coordination con fi guration mode placement b MX SX1 SX2 H ouston Paris Singapore 

Navigating the coordination lattice CcRepExecTime = ∑ r∈R (CcOpSerial(r) + CcRepSerial(r) + CcOpParallel(r)) |R| Minimize 19 CcOpSerial(r) = Impact of serialization due to exclusive mode of locking CcRepSerial(r) = Impact of serialization due to serialization inside a replica CcOpParallel(r) = Parallelism allowed by shared mode of locking Cost of locking is impacted by lock placement, a component inside all the metrics 

Two operation con fl ict x y a Placement 1 Houston 2 Paris 3 Singapore Mode x y 1 EX EX 2 EX SH 3 SH EX granularity-mode-placement Execution time (ms) Worklo ad H P S x 0 500 0 y 250 0 250 20 

Two operation con fl ict x y a Placement 1 Houston 2 Paris 3 Singapore Mode x y 1 EX EX 2 EX SH 3 SH EX granularity-mode-placement Execution time (ms) Worklo ad H P S x 0 500 0 y 250 0 250 20 1. Colocate lock with the workload 2. Choose shared mode for the most frequently distributed operation 

Three operations con fl ict x y z a1 a2 a1 , a2 a1 a2 Placement a1 a2 1 H H 2 P H 3 S H 4 H P 5 P P 6 S P 7 H S 8 P S 9 S S Mode a1 a2 x y y z 1 X X X X 2 X S X X 3 S X X X 4 X X X S 5 X S X S 6 S X X S 7 X X S X 8 X S S X 9 S X S X Placement abc 1 H 2 P 3 S Mode abc x y z 1 X X X 2 X S X 3 S X S Workload H P S x 0 1 0 y 100 100 100 z 0 1 0 21 

22 Execution time (ms) granularity-mode-placement Three operations con fl ict Workl oad H P S x 0 1 0 y 100 100 100 z 0 1 0 Placement ab bc 1 H H 2 P H 3 S H 4 H P 5 P P 6 S P 7 H S 8 P S 9 S S Mode ab bc place close close remove 1 X X X X 2 X S X X 3 S X X X 4 X X X S 5 X S X S 6 S X X S 7 X X S X 8 X S S X 9 S X S X Mode abc place close remove 1 X X X 2 X S X 3 S X S Placement abc 1 H 2 P 3 S x y z a1 a2 

23 Execution time (ms) granularity-mode-placement Three operations con fl ict Workl oad H P S x 100 100 0 y 0 0 0 z 0 0 50 Placement ab bc 1 H H 2 P H 3 S H 4 H P 5 P P 6 S P 7 H S 8 P S 9 S S Mode ab bc place close close remove 1 X X X X 2 X S X X 3 S X X X 4 X X X S 5 X S X S 6 S X X S 7 X X S X 8 X S S X 9 S X S X Mode abc place close remove 1 X X X 2 X S X 3 S X S Placement abc 1 H 2 P 3 S x y z a1 a2 

23 Execution time (ms) granularity-mode-placement Three operations con fl ict Workl oad H P S x 100 100 0 y 0 0 0 z 0 0 50 Placement ab bc 1 H H 2 P H 3 S H 4 H P 5 P P 6 S P 7 H S 8 P S 9 S S Mode ab bc place close close remove 1 X X X X 2 X S X X 3 S X X X 4 X X X S 5 X S X S 6 S X X S 7 X X S X 8 X S S X 9 S X S X Mode abc place close remove 1 X X X 2 X S X 3 S X S Placement abc 1 H 2 P 3 S Coarsening is a trade-off between contention and acquisition cost x y z a1 a2 

Future Work Probabilistic constraint model Support dynamic recon fi guration of coordination con fi guration More concurrency control mechanisms 24 

Conclusion Coordination lattice is a fi rst step to systematically explore performance implications of concurrency control A step towards engineering Just-Right-Consistent distributed applications 25 Selecting optimal distributed lock con fi guration • Coordination lattice • Metrics 

Back up 

Auction : Static Analysis Con fl icts detected • Add product - unregister seller • Create auction - unregister seller • Add to lot - add to lot • Add to lot - start auction • Add to lot - remove auction • Add to lot - remove product • Start auction - remove from lot • Place bid - close auction • Place bid - unregister buyer • Close auction - remove bid 27 

Auction : Synchronisation Condition Generation 28 • Add product - unregister seller seller • Create auction - unregister seller seller • Add to lot - add to lot seller, (product, seller) • Add to lot - start auction auction • Add to lot - remove auction auction • Add to lot - remove product (product, seller) • Start auction - remove from lot auction • Place bid - close auction auction • Place bid - unregister buyer buyer • Close auction - remove bid auction 

Auction : Grouping 29 Add product Unregister seller Create auction Place bid Close auction Unregister buyer Remove bid Register seller Register buyer Add to lot Start auction Remove from lot Remove auction Remove product 

Auction : Filtering 30 Add product Unregister seller Create auction Place bid Close auction Unregister buyer Remove bid Register seller Register buyer Add to lot Start auction Remove from lot Remove auction Remove product 

Auction : Filtering 30 Add product Unregister seller Create auction Place bid Close auction Unregister buyer Remove bid Add to lot Start auction Remove from lot Remove auction Remove product 

Placement Centralised (P) vs Clustered(P, C, N) vs Distributed 

Cost of locking 

Auction Place bid Close auction auction a b lab−auction Placement 1 Houston 2 Paris 3 Singapore Mode place bid close auction 1 X X 2 X S 3 S X granularity-mode-placement Execution time (ms) Work load H P S place 0 1000 0 close 0 0 0 33 

Auction Place bid Close auction auction a b lab−auction Placement 1 Houston 2 Paris 3 Singapore Mode place bid close auction 1 X X 2 X S 3 S X granularity-mode-placement Execution time (ms) Work load H P S place 333 333 333 close 0 0 0 34 

Auction place bid close auction remove bid a1 a2 a1 , a2 a1 a2 Placement ab bc 1 H H 2 P H 3 S H 4 H P 5 P P 6 S P 7 H S 8 P S 9 S S Mode ab bc place close close remove 1 X X X X 2 X S X X 3 S X X X 4 X X X S 5 X S X S 6 S X X S 7 X X S X 8 X S S X 9 S X S X Placement abc 1 H 2 P 3 S Mode abc place close remove 1 X X X 2 X S X 3 S X S Workload H P S place 455 455 0 close 37 37 0 remove 8 8 0 35 

Auction Placement ab bc 1 H H 2 P H 3 S H 4 H P 5 P P 6 S P 7 H S 8 P S 9 S S Mode ab bc place close close remove 1 X X X X 2 X S X X 3 S X X X 4 X X X S 5 X S X S 6 S X X S 7 X X S X 8 X S S X 9 S X S X g-m-p Execution time (ms) Placement abc 1 H 2 P 3 S Mode abc place close remove 1 X X X 2 X S X 3 S X S Workload H P S place 455 455 0 close 37 37 0 remove 8 8 0 36 

Auction Placement ab bc 1 H H 2 P H 3 S H 4 H P 5 P P 6 S P 7 H S 8 P S 9 S S Mode ab bc place close close remove 1 X X X X 2 X S X X 3 S X X X 4 X X X S 5 X S X S 6 S X X S 7 X X S X 8 X S S X 9 S X S X g-m-p Execution time (ms) Placement abc 1 H 2 P 3 S Mode abc place close remove 1 X X X 2 X S X 3 S X S Workload H P S place 455 455 0 close 37 37 0 remove 8 8 0 36 1. Colocate lock with the workload 2. Choose shared mode for the most frequently distributed operation 3. Coarsening is a trade-off between contention and acquisition cost 

Auction Placement ab bc 1 H H 2 P H 3 S H 4 H P 5 P P 6 S P 7 H S 8 P S 9 S S Mode ab bc place close close remove 1 X X X X 2 X S X X 3 S X X X 4 X X X S 5 X S X S 6 S X X S 7 X X S X 8 X S S X 9 S X S X g-m-p Execution time (ms) Placement abc 1 H 2 P 3 S Mode abc place close remove 1 X X X 2 X S X 3 S X S Workload H P S place 0 909 0 close 0 75 0 remove 0 15 0 37 Granularity 1 - 2 locks Granularity 2 - single lock 

Auction Placement ab bc 1 H H 2 P H 3 S H 4 H P 5 P P 6 S P 7 H S 8 P S 9 S S Mode ab bc place close close remove 1 X X X X 2 X S X X 3 S X X X 4 X X X S 5 X S X S 6 S X X S 7 X X S X 8 X S S X 9 S X S X g-m-p Execution time (ms) Placement abc 1 H 2 P 3 S Mode abc place close remove 1 X X X 2 X S X 3 S X S Workload H P S place 303 303 303 close 25 25 25 remove 5 5 5 38 Granularity 1 - 2 locks Granularity 2 - single lock