Baquero, Associate professor, Universidade do Minho Reviewer Béatrice Bérard, Professor, Sorbonne Université Examiner Carla Ferreira, Associate professor, Universidade Nova de Lisboa Examiner Éric Gressier-Soudan, Professor, Conservatoire National des Arts et Métiers Reviewer Bradley King, Co-founder & Field CTO, Scality Examiner Martin Kleppmann, Senior Research Associate, University of Cambridge Examiner Gustavo Petri, Researcher, Arm Cambridge Examiner Marc Shapiro, Distinguished Research Scholar, Sorbonne Université-Inria Advisor Thesis defended on 1 July 2021 before a defense committee composed of:
policies Synthesize concurrency control Speci fi cation and invariants Workload characteristics Con fl icts? Can the application afford anomalies? 2. CRDT Tree case study 3. Coordination lattice + metrics 6 Safe Y Y N N 1. New Proof rule + Tool
true since merge can happen at any time • Can be also called Concurrency Invariant The weakest precondition to be upheld for the resulting state of merge to uphold the data invariant Ensures all concurrent operations are still safe Invglobal = Invdata ∧ Invconc 12
invariant for the class of distributed applications that propagate operations • Proof rule for distributed applications regardless of the update propagation mechanism • Improving usability of the tool Proof rule and tool for verifying safety of highly-available distributed objects ‣ Modular ‣ Automated Veri fi cation 15
con fl ict resolution policies Synthesize concurrency control Ex: Tree CRDT 16 Speci fi cation and invariants Workload characteristics Con fl icts? Can the application afford anomalies? Safe Y Y N N
con fl ict resolution policies Synthesize concurrency control Ex: Tree CRDT 16 Speci fi cation and invariants Workload characteristics Con fl icts? Can the application afford anomalies? Safe Y Y N N
con fl ict resolution policies Synthesize concurrency control Ex: Tree CRDT 16 Speci fi cation and invariants Workload characteristics Con fl icts? Can the application afford anomalies? Safe Y Y N N
con fl ict resolution policies Synthesize concurrency control Ex: Tree CRDT 16 Speci fi cation and invariants Workload characteristics Con fl icts? Can the application afford anomalies? Safe Y Y N N
a under b / b a / a b move b under a / a b 18 move b under a / b a Con fl ict resolution • Skip one move among the pair of con fl icts deterministically
a b / b a move b under / 1 move a under b 2 19 Skipping 1 leads to cycle! / a b move a under b 2 / a b Independence Analysis • The conditions under which a historical move might impact the current move - Historical Enabler Move
tree with the same con fl ict resolution semantics • Prove the soundness of independence analysis for safe distributed applications that have tentative e ff ectors • Optimize metadata required for con fl ict resolution policies Coordination-free safe and highly-available replicated tree • Trade-o ff : lost move 22
Design con fl ict resolution policies Synthesize concurrency control Ex: Tree CRDT Speci fi cation and invariants Workload characteristics Con fl icts? Can the application afford anomalies? Safe Y Y N N
Design con fl ict resolution policies Synthesize concurrency control Ex: Tree CRDT Speci fi cation and invariants Workload characteristics Con fl icts? Can the application afford anomalies? Safe Y Y N N
Design con fl ict resolution policies Synthesize concurrency control Ex: Tree CRDT Speci fi cation and invariants Workload characteristics Con fl icts? Can the application afford anomalies? Safe Y Y N N
ensure safety • Skewed workload in locality and frequency • Network latency • Cost of locking Variables that can be controlled • Granularity • Type of lock - Mutual exclusion - Shared/exclusive lock • Placement of lock - Distance between client and lock 25 Selecting Ef fi cient Concurrency Control for Safety
Close auction Unregiste r buyer Remove bid 90% 1% 7% 2% Frequency of Place bid operation for a single auction Paris Houston Singapore 95% 5% 0% Skewed in frequency per replica
• Skewed workload in locality and frequency Network constraint • Latency between replicas Lock cost constraint • Cost of locking as a function of computation and network latency 30
bid MX EX EX SX1 EX SH SX2 SH EX mode placement a1a2 MX SX1 SX2 H ouston Paris Singapore Mode a1a2 place bid close auction remove bid MX EX EX EX SX1 EX SH EX SX2 SH EX SH Coordination con fi guration mode placement b MX SX1 SX2 H ouston Paris Singapore
CcRepSerial(r) + CcOpParallel(r)) |R| Minimize 38 CcOpSerial(r) = Impact of serialization due to exclusive mode of locking CcRepSerial(r) = Impact of serialization due to serialization inside a replica CcOpParallel(r) = Parallelism allowed by shared mode of locking Cost of locking is impacted by lock placement, a component inside all the metrics
Houston 2 Paris 3 Singapore Mode x y 1 EX EX 2 EX SH 3 SH EX granularity-mode-placement Execution time (ms) Worklo ad H P S x 0 500 0 y 250 0 250 39 1. Colocate lock with the workload 2. Choose shared mode for the most frequently distributed operation
a1 , a2 a1 a2 Placement a1 a2 1 H H 2 P H 3 S H 4 H P 5 P P 6 S P 7 H S 8 P S 9 S S Mode a1 a2 x y y z 1 X X X X 2 X S X X 3 S X X X 4 X X X S 5 X S X S 6 S X X S 7 X X S X 8 X S S X 9 S X S X Placement abc 1 H 2 P 3 S Mode abc x y z 1 X X X 2 X S X 3 S X S Workload H P S x 0 1 0 y 100 100 100 z 0 1 0 40
Workl oad H P S x 0 1 0 y 100 100 100 z 0 1 0 Placement ab bc 1 H H 2 P H 3 S H 4 H P 5 P P 6 S P 7 H S 8 P S 9 S S Mode ab bc place close close remove 1 X X X X 2 X S X X 3 S X X X 4 X X X S 5 X S X S 6 S X X S 7 X X S X 8 X S S X 9 S X S X Mode abc place close remove 1 X X X 2 X S X 3 S X S Placement abc 1 H 2 P 3 S x y z a1 a2
Workl oad H P S x 100 100 0 y 0 0 0 z 0 0 50 Placement ab bc 1 H H 2 P H 3 S H 4 H P 5 P P 6 S P 7 H S 8 P S 9 S S Mode ab bc place close close remove 1 X X X X 2 X S X X 3 S X X X 4 X X X S 5 X S X S 6 S X X S 7 X X S X 8 X S S X 9 S X S X Mode abc place close remove 1 X X X 2 X S X 3 S X S Placement abc 1 H 2 P 3 S x y z a1 a2
Workl oad H P S x 100 100 0 y 0 0 0 z 0 0 50 Placement ab bc 1 H H 2 P H 3 S H 4 H P 5 P P 6 S P 7 H S 8 P S 9 S S Mode ab bc place close close remove 1 X X X X 2 X S X X 3 S X X X 4 X X X S 5 X S X S 6 S X X S 7 X X S X 8 X S S X 9 S X S X Mode abc place close remove 1 X X X 2 X S X 3 S X S Placement abc 1 H 2 P 3 S Coarsening is a trade-off between contention and acquisition cost x y z a1 a2
Support dynamic recon fi guration of coordination con fi guration • More concurrency control mechanisms Selecting optimal distributed lock con fi guration • Coordination lattice • Metrics 43
safety of distributed applications • Replicated tree of Part II illustrates con fl ict resolution policy design • Coordination lattice of Part III is a fi rst step to systematically explore performance implications of concurrency control • A step towards engineering Just-Right-Consistent distributed applications 44
placed only when the status is active • When auction is closed, there is a winner • Winner is the highest bid Concurrency Invariant • Winner in either state is the highest bid in both states 48
placed only when the status is active • When auction is closed, there is a winner • Winner is the highest bid Concurrency Invariant • Winner in either state is the highest bid in both states 48
can be placed only when the status is active • When auction is closed, there is a winner • Winner is the highest bid Concurrency Invariant • Winner in either state is the highest bid in both states 48
can be placed only when the status is active • When auction is closed, there is a winner • Winner is the highest bid Concurrency Invariant • Winner in either state is the highest bid in both states 48
can be placed only when the status is active • When auction is closed, there is a winner • Winner is the highest bid Concurrency Invariant • Winner in either state is the highest bid in both states Place bid • If auction concurrently closed in other replica, precondition of merge violated! 48
can be placed only when the status is active • When auction is closed, there is a winner • Winner is the highest bid Concurrency Invariant • Winner in either state is the highest bid in both states Place bid • If auction concurrently closed in other replica, precondition of merge violated! 48
can be placed only when the status is active • When auction is closed, there is a winner • Winner is the highest bid Concurrency Invariant • Winner in either state is the highest bid in both states Place bid • If auction concurrently closed in other replica, precondition of merge violated! Close auction • Similar to place bid 48
can be placed only when the status is active • When auction is closed, there is a winner • Winner is the highest bid Concurrency Invariant • Winner in either state is the highest bid in both states Place bid • If auction concurrently closed in other replica, precondition of merge violated! Close auction • Similar to place bid 48
add/remove operations using set CRDTs [0] Move operations are known to be unsafe in concurrent execution [1] A fi x for concurrent move operations is to form a total order for operations [2], but it is expensive Another solution is to convert a move operation into a copy- delete pair [3], but ending up in multiple copies of the node [0] Stéphane Martin, Mehdi Ahmed-Nacer, Pascal Urso. Abstract unordered and ordered trees CRDT. [1] M. Najafzadeh, M. Shapiro, P. Eugster : “Co-Design and Veri fi cation of an Available File System”, VMCAI 2018 [2] Kleppmann, M., Gomes, V. B., Mulligan, D. P., & Beresford, A. R. OpSets: Sequential Speci fi cations for Replicated Datatypes. [3] Vinh Tao, Marc Shapiro, and Vianney Rancurel. Merging semantics for con fl ict updates in geo-distributed fi le systems. 53
of nodes • Node - (id, parent) Operations • add_node(id, parent) • remove_node(id, parent) • move(from_parent, id, to_parent) Invariant • Maintain tree structure : all nodes reachable from root (This implies no cycles since every node has a single parent) 54
parent-child relations, and no cycles • Special node called Root has no parent, created at tree initialisation • All non-root nodes have exactly one parent and must be reachable from root • Operations supported : add, remove, move • Used to store File system structure, xml, json etc. 55 Root
→* p // RP ∧ Root →* p' // RP' ∧ p → c // PC ∧ ¬ (c →* p') // NOT_UNDER_SELF 62 Stable (No recursive remove) Unstable against remove(p’) Fixed with tombstones
→* p // RP ∧ Root →* p' // RP' ∧ p → c // PC ∧ ¬ (c →* p') // NOT_UNDER_SELF 62 Stable (No recursive remove) Unstable against remove(p’) Unstable against move(p,c,p’’) Fixed with tombstones
∧ Root →* p' // RP' ∧ p → c // PC ∧ ¬ (c →* p') // NOT_UNDER_SELF Move in concurrent execution 64 Stable (No recursive remove) Unstable against remove(p’)
∧ Root →* p' // RP' ∧ p → c // PC ∧ ¬ (c →* p') // NOT_UNDER_SELF Move in concurrent execution 64 Stable (No recursive remove) Unstable against remove(p’) Fixed with tombstones
∧ Root →* p' // RP' ∧ p → c // PC ∧ ¬ (c →* p') // NOT_UNDER_SELF Move in concurrent execution 64 Stable (No recursive remove) Unstable against remove(p’) Unstable against move(p,c,p’’) Fixed with tombstones
∧ Root →* p' // RP' ∧ p → c // PC ∧ ¬ (c →* p') // NOT_UNDER_SELF Move in concurrent execution 64 Stable (No recursive remove) Unstable against remove(p’) Unstable against move(p,c,p’’) Fixed with tombstones Con fl ict resolution applied
∧ Root →* p' // RP' ∧ p → c // PC ∧ ¬ (c →* p') // NOT_UNDER_SELF Move in concurrent execution 64 Stable (No recursive remove) Unstable against remove(p’) Unstable against move(p,c,p’’) Unstable against concurrent move Fixed with tombstones Con fl ict resolution applied
execution NOT_UNDER_SELF forbids to move p’ under c However, a concurrent move of p’ under c would cause a cycle NOT_UNDER_SELF is not stable! Generalises to moving any node p’’ ∈ [p’, LCA) ∀n . n → * p′  ∧ n ↛ * c ∀n . c → * n Critical ancestors Critical descendants LCA p c p’
move(p, c, p’) move(p1, c1, p1’) and move(p2, c2, p2’) con fl ict when a node c1 that is being moved, is also a critical ancestor of another move: c1 ∈ [ p2’, LCA (c2, p2’) ) p p’ c a move(p, c, p’)
move(p, c, p’) move(p1, c1, p1’) and move(p2, c2, p2’) con fl ict when a node c1 that is being moved, is also a critical ancestor of another move: c1 ∈ [ p2’, LCA (c2, p2’) ) p p’ c a move(p, c, p’)
move(p, c, p’) Critical ancestors of move(a, p’, c) move(p1, c1, p1’) and move(p2, c2, p2’) con fl ict when a node c1 that is being moved, is also a critical ancestor of another move: c1 ∈ [ p2’, LCA (c2, p2’) ) p p’ c a move(p, c, p’) move(a, p’, c)
move(p, c, p’) Critical ancestors of move(a, p’, c) move(p1, c1, p1’) and move(p2, c2, p2’) con fl ict when a node c1 that is being moved, is also a critical ancestor of another move: c1 ∈ [ p2’, LCA (c2, p2’) ) p p’ c a move(p, c, p’) move(a, p’, c)
move(p, c, p’) Critical ancestors of move(a, p’, c) move(p1, c1, p1’) and move(p2, c2, p2’) con fl ict when a node c1 that is being moved, is also a critical ancestor of another move: c1 ∈ [ p2’, LCA (c2, p2’) ) p p’ c a move(p, c, p’) move(a, p’, c) Move of critical ancestor causes a cycle!
if rank(c) > rank(p’) NOT_UNDER_SELF is unstable if any critical ancestors move under critical descendants ∀ d ∈ critical descendants, a ∈ critical ancestors, rank(d) > rank(a) This implies that only a down-move can cause c →* p’ p p’ c
the same distance Down-move - moving away from the root 69 Concurrent moves on critical ancestors Up-move Down-move Up-move No Con fl ict Con fl ict Down-move Con fl ict Con fl ict
d / a c b d Lisbon Paris move b under d down-move move d under b down-move / a c b d / a c b d move a under b down-move / a c b d 1 2 3 1;2 concurrent with 3 71
d / a c b d Lisbon Paris move b under d down-move move d under b down-move / a c b d / a c b d move a under b down-move / a c b d 1 2 3 1;2 concurrent with 3 1 con fl icts with 3, 2 doesn’t con fl ict with 3 71
d / a c b d Lisbon Paris move b under d down-move move d under b down-move / a c b d / a c b d move a under b down-move / a c b d 1 2 3 1;2 concurrent with 3 1 con fl icts with 3, 2 doesn’t con fl ict with 3 3 has higher priority than 1 1 skips 71
d / a c b d Lisbon Paris move b under d down-move move d under b down-move / a c b d / a c b d move a under b down-move / a c b d 1 2 3 1;2 concurrent with 3 1 con fl icts with 3, 2 doesn’t con fl ict with 3 Both replicas apply 3, then 2 3 has higher priority than 1 1 skips 71
d Paris move d under b down-move / a c b d move a under b down-move 2 3 Both replicas apply 3, then 2 / a c b d CYCLE!! Reason: 2 is dependent on the skipped 1 72
d Paris move d under b down-move / a c b d move a under b down-move 2 3 Both replicas apply 3, then 2 / a c b d Independence Analysis CYCLE!! Reason: 2 is dependent on the skipped 1 72
concurrent moves that will result in a cycle (critical ancestors) To determine the type of move To determine skipped historical enabler moves (critical descendants) 74
product - unregister seller • Create auction - unregister seller • Add to lot - add to lot • Add to lot - start auction • Add to lot - remove auction • Add to lot - remove product • Start auction - remove from lot • Place bid - close auction • Place bid - unregister buyer • Close auction - remove bid 78
unregister seller seller • Create auction - unregister seller seller • Add to lot - add to lot seller, (product, seller) • Add to lot - start auction auction • Add to lot - remove auction auction • Add to lot - remove product (product, seller) • Start auction - remove from lot auction • Place bid - close auction auction • Place bid - unregister buyer buyer • Close auction - remove bid auction
Place bid Close auction Unregister buyer Remove bid Register seller Register buyer Add to lot Start auction Remove from lot Remove auction Remove product
Place bid Close auction Unregister buyer Remove bid Register seller Register buyer Add to lot Start auction Remove from lot Remove auction Remove product
1 Houston 2 Paris 3 Singapore Mode place bid close auction 1 X X 2 X S 3 S X granularity-mode-placement Execution time (ms) Work load H P S place 0 1000 0 close 0 0 0 84
1 Houston 2 Paris 3 Singapore Mode place bid close auction 1 X X 2 X S 3 S X granularity-mode-placement Execution time (ms) Work load H P S place 333 333 333 close 0 0 0 85
, a2 a1 a2 Placement ab bc 1 H H 2 P H 3 S H 4 H P 5 P P 6 S P 7 H S 8 P S 9 S S Mode ab bc place close close remove 1 X X X X 2 X S X X 3 S X X X 4 X X X S 5 X S X S 6 S X X S 7 X X S X 8 X S S X 9 S X S X Placement abc 1 H 2 P 3 S Mode abc place close remove 1 X X X 2 X S X 3 S X S Workload H P S place 455 455 0 close 37 37 0 remove 8 8 0 86
3 S H 4 H P 5 P P 6 S P 7 H S 8 P S 9 S S Mode ab bc place close close remove 1 X X X X 2 X S X X 3 S X X X 4 X X X S 5 X S X S 6 S X X S 7 X X S X 8 X S S X 9 S X S X g-m-p Execution time (ms) Placement abc 1 H 2 P 3 S Mode abc place close remove 1 X X X 2 X S X 3 S X S Workload H P S place 455 455 0 close 37 37 0 remove 8 8 0 87
3 S H 4 H P 5 P P 6 S P 7 H S 8 P S 9 S S Mode ab bc place close close remove 1 X X X X 2 X S X X 3 S X X X 4 X X X S 5 X S X S 6 S X X S 7 X X S X 8 X S S X 9 S X S X g-m-p Execution time (ms) Placement abc 1 H 2 P 3 S Mode abc place close remove 1 X X X 2 X S X 3 S X S Workload H P S place 455 455 0 close 37 37 0 remove 8 8 0 87 1. Colocate lock with the workload 2. Choose shared mode for the most frequently distributed operation 3. Coarsening is a trade-off between contention and acquisition cost
3 S H 4 H P 5 P P 6 S P 7 H S 8 P S 9 S S Mode ab bc place close close remove 1 X X X X 2 X S X X 3 S X X X 4 X X X S 5 X S X S 6 S X X S 7 X X S X 8 X S S X 9 S X S X g-m-p Execution time (ms) Placement abc 1 H 2 P 3 S Mode abc place close remove 1 X X X 2 X S X 3 S X S Workload H P S place 0 909 0 close 0 75 0 remove 0 15 0 88 Granularity 1 - 2 locks Granularity 2 - single lock
3 S H 4 H P 5 P P 6 S P 7 H S 8 P S 9 S S Mode ab bc place close close remove 1 X X X X 2 X S X X 3 S X X X 4 X X X S 5 X S X S 6 S X X S 7 X X S X 8 X S S X 9 S X S X g-m-p Execution time (ms) Placement abc 1 H 2 P 3 S Mode abc place close remove 1 X X X 2 X S X 3 S X S Workload H P S place 303 303 303 close 25 25 25 remove 5 5 5 89 Granularity 1 - 2 locks Granularity 2 - single lock
sreeja
aarron
tammyeverts
oripsolob
productmarketing
baasie
speakerdeck
lara
axbom
nmsamuel
lfama
kimpetersen
tessaabrams
