PWLSF#6 => Peter Alvaro: Using Reasoning about Knowledge to Analyze Distributed Systems

Peter Alvaro from UC Berkeley will present the paper "Using Reasoning about Knowledge to Analyze Distributed Systems" by Joseph Halpern.

If you have any questions, thoughts, or related information, please visit our *github-thread* on the matter: https://github.com/papers-we-love/papers-we-love/issues/147

Peter has kindly provided some references to help you get started

• Prior Halpern work on knowledge in DS:
https://www.cs.cornell.edu/home/halpern/papers/common_knowledge.pdf

• Some applications: http://dl.acm.org/citation.cfm?id=28672

• Interlandi's work on knowledge-enriched dedalus:
http://link.springer.com/chapter/10.1007%2F978-3-642-34002-4_47
http://www.dbgroup.unimo.it/~interlandi/vldb_12.pdf (appears to be an earlier draft)

• Dedalus (background for knowlog, and in case using logic to program distributed systems sounds interesting): db.cs.berkeley.edu/papers/datalog2011-dedalus.pdf

• Modal logics: http://plato.stanford.edu/entries/logic-modal/

Peter's Bio
Peter is a PhD candidate at the University of California Berkeley, where he is advised by Joseph M. Hellerstein. He holds a Master's degree in Computer Science from UC Berkeley and a Bachelor of Arts in English Literature from Middlebury College. Prior to attending UC Berkeley, Peter worked as a Senior Software Engineer in the data analytics team at Ask.com.

Peter's principal research interests are databases, distributed systems and programming languages. More details at http://www.cs.berkeley.edu/~palvaro/.

Transcript

1. Ineluctable modality of the distributed On Joseph Halpern’s work on knowledge in distributed systems Peter Alvaro UC Berkeley
2. choose-your-own-adventure talk
3. Last time at PWL… •  The agreement problem(s) •  Impossibility results •  A “weakest” failure detector Today: knowledge  
4. It’s not just for byzantine stuff I'm not a great fool, so I can clearly not choose the wine in front of you. But you must have known I was not a great fool; you would have counted on it, so I can clearly not choose the wine in front of me.
5. Why you should care A correct distributed program achieves (nontrivial) distributed property X. Some tricky questions before we start coding: 1.  Is X even attainable? 2.  Cheapest protocol that gets me X? 3.  How should I implement it?
6. A strong claim about distributed correctness properties   Uncertainty is what makes reasoning about distributed systems difficult. Uncertainty is the abundance of possibilities. Knowledge is the dual of possibility
7. A strong statement about protocols How: Protocols just describe what actions to take based on local knowledge. Why: Protocols are just mechanisms to ensure that a group has shared knowledge of a fact.
8. A good paper about bridging the gap between properties and protocols
9. For example •  Commit protocols – each agent knows the commit/abort decision AND knows that all agents know the decision •  Distributed garbage collection – an agent knows that no remote references exist to a particular object, and that all other agents know
10. For example •  When the leader has received phase 2b messages for value v and ballot bal from a majority of the acceptors, it knows that the value v has been chosen. [paxos] •  a process takes a checkpoint when it knows that all processes on which it computationally depends took their checkpoints [An Efficient Protocol for Checkpointing Recovery in Distributed Systems, Kim and Park] •  and therefore a cohort with a later viewstamp for some view knows everything known to a cohort with an earlier viewstamp for that view. [viewstamped replication] •  Since each member of Si serves as an arbitrator, the requesting node knows that it is the only node that has been granted mutual exclusion [A sqrt(N) Algorithm for Mutual Exclusion in Decentralized Systems, Maekawa]
11. Warmup: RPC protocols Hi! Alice Bob Issue: uncertainty! Uncertain environment è Uncertain outcomes
12. Warmup: RPC protocols Hi! Retry   Alice Bob
13. Warmup: RPC protocols Hi! Retry   Alice Bob
14. Warmup: RPC protocols Hi! Retry   Alice Bob
15. Warmup: RPC protocols Hi! Retry   Alice Bob
16. Warmup: RPC protocols Hi! Issues: infinite (sender) behavior & state, at-least-once delivery Retry   Alice Bob
17. Warmup: RPC protocols Hi! Hi yourself Retry with ACKS Hi! Issues: at-least once delivery Hi! Alice Bob
18. a  good  paper  about  principled  distributed  GC  
19. Warmup: RPC protocols Hi! Issues: infinite receiver state Hi! Receiver buffers, dedups Alice Bob
20. Warmup: RPC protocols Hi! Hi yourself ACK-ACKing Hi! Issue: uncertainty Hi! Alice Bob Ahoy  
21. Warmup: RPC protocols Issues: infinite hot potato Alice Bob
22. Warmup: RPC protocols Issues: infinite hot potato Alice Bob
23. what does this remind me of? Refresher: the two generals problem
24. Logic time
25. (propositional) logic ϕ ϕ if ϕ is atomic ϕ ∧ ψ true if both ϕ and ψ are true ¬ϕ true if ϕ is false Sweet duality: ϕ ∨ ψ = ¬(¬ϕ ∧ ¬ψ) ϕ ⇒ ψ= ¬(ϕ ∧ ¬ψ) q ⇒ p p = “the write is stable” q = “the write is acknowledged”
26. modality, duality ∃xϕ === ¬∀x ¬ϕ ¯ϕ === ¬£¬ϕ Symbol   Temporal   Deon/c   Epistemic   ¯   Some8mes   Is  permi:ed   Is  possible   £   Always   Is  obligatory   Is  known   Knowledge is the dual of possibility
27. Epistemic modal logic ϕ = “the write is stable” Kalice ϕ = “alice knows ϕ” Kalice Kbob ϕ = “alice knows bob knows ϕ” Kalice Kbob Kcarol ϕ = “alice knows bob knows carol knows ϕ” […]
28. Epistemic modal logic ϕ = “the write is stable” Eϕ = “everyone* knows ϕ” EEϕ = “everyone knows everyone knows ϕ” […] A driver will not feel safe going when he sees a green light unless he knows that everyone else knows and follows the rules.
29. Common knowledge ϕ = “the write is stable” Eϕ = “everyone* knows ϕ” EEϕ = “everyone knows everyone knows ϕ” […] Eiϕ = “(everyone knows * i) ϕ” Cϕ = E∞ϕ = “it is common knowledge that ϕ”
30. Distributed knowledge ϕ = “the write is stable” Dϕ = “ϕ is implicitly known by the group” Sϕ = “someone knows ϕ”
31. Protocols  climb  the  hierarchy   Cϕ […] Ek+1ϕ […] Eϕ Sϕ Dϕ ϕ  
32. Protocols  climb  the  hierarchy   Cϕ […] Ek+1ϕ […] Eϕ Sϕ Dϕ ϕ   Deadlock detection ϕ is distributed knowledge   Someone knows ϕ
33. Protocols  climb  the  hierarchy   Cϕ […] Ek+1ϕ […] Eϕ Sϕ Dϕ ϕ   Reliable broadcast Someone knows ϕ ϕ is distributed knowledge   Everyone knows ϕ
34. Protocols  climb  the  hierarchy   Cϕ […] E3ϕ E2ϕ Eϕ Sϕ Dϕ ϕ   Uniform Reliable broadcast Someone knows ϕ ϕ is distributed knowledge   Everyone knows ϕ Everyone knows everyone knows ϕ
35. Protocols  climb  the  hierarchy   Cϕ […] E3ϕ E2ϕ Eϕ Sϕ Dϕ ϕ   Someone knows ϕ ϕ is distributed knowledge   Everyone knows ϕ Everyone knows everyone knows ϕ Some crazy BFT protocol (Everyone knows)k ϕ
36. Protocols  climb  the  hierarchy   Cϕ […] E3ϕ E2ϕ Eϕ Sϕ Dϕ ϕ   Knowledge  Highway   E10ϕ          10   E100ϕ                    100   Cϕ  ∞  
37. Applications of knowledge A correct distributed program achieves (nontrivial) distributed property X. Some tricky questions before we start coding: 1.  Is X even attainable? 2.  Cheapest protocol that gets me X? 3.  How should I implement it?
38. Applications: impossibility “in a system in which communication is not guaranteed, common knowledge of initially-undetermined facts is not attainable in any run of any protocol.” Corollary: the 2 generals problem is unsolvable
39. Let’s use knowledge to prove it! But first… lots of formalism to get through L
40. Road map for the proof: 1.  Semantics of modal logic 2.  Distributed system model 3.  A quick and easy lemma 4.  Big theorem: Common knowledge is not attainable via protocol 5.  Lemma 2: if the generals attack, they have common knowledge of the attack. 6.  Corollary: 2 generals is unsolvable
41. Semantics
42. Semantics: structures Formulae are well-formed, meaningless strings of symbols Structures give meaning to formulae (in the very narrow sense of making them all either true or false) S |= ϕ
43. Semantics: propositional structures Propositional formula: S |= p ∧ q Need: 1.  a map S from variable names to T/F 2.  rules; e.g. S |= ϕ ∧ ψ iff S |= ϕ and S |= ψ
44. Semantics: first-order structures First-order formula: S |= ∀x, dog(x) ⇒ big(x) ∧ likes(x, me) Need: 1.  S assigns “records” to dog, big and likes. 2.  Rules; e.g. S |= ∀xφ iff for all d ∈  |S|,  S[x  :=  d]  |=  φ  
45. Semantics: first-order structures •  First-order logic: S |= ∀x, dog(x) ⇒ big(x) ∧ likes(x, me) dog   Rex   Fido   Rover   big   Rex   Fido   me   likes   Rex   me   Fido   me   Rover   me   me   me  
46. couple good papers about using FO logic to program distributed systems
47. Semantics – modal logic S |= (£¬p) ∧ (q ⇒ ¯r) Need: a structure that can interpret the propositional formulae under different modalities Kripke structure: (W, π, R) •  W is a set of worlds •  For each element of W, π is a propositional structure •  R is an accessibility relation among elements of W S1   S3  
48. Semantics – modal logic Temporal logic S |= (£¬p) ∧ (q ⇒ ¯r)  q      r   r   q   S1   S3   S2   Kripke structure: (W, π, R)  
49. Semantics – modal logic Epistemic logic S |= r ∧ ¬Ki r ∧ Ki (Kj r or Kj ¬r) ∧ Kj r ∧ ¬Kj ¬Ki r  q      r   r   q   S1   S3   S2   i   j   Kripke structure: (W, π, Ri )  
50. a model of distributed systems (r,t) p 1 p 2 p 3 p 4 Idealized time }h(p 4 ,r,t) A run r ∈ R
51. Knowledge-based interpretations Knowledge interpretation: I = (R, π, {v1 ,v2 ,[..]}) Knowledge point: (I, r, t) R – a set of runs π – assigns a truth assignment to propositions for each point in R vi – A view function for R for some agent i (determined by h) Kripke structure: (W, π, R)  
52. Truth in a knowledge interpretation (I,r,t) |= φ iff π(r,t)(φ) = true (If φ is a ground formula) (I,r,t) |= ¬φ iff (I,r,t) |= φ (I,r,t) |= φ ∧ ψ iff (I,r,t) |= φ and (I,r,t) |= ψ (I,r,t) |= Ki φ iff (I,r’,t’) |= φ for all (r’,t’) in R satisfying v(pi ,r,t) = v(pi ,r’,t’)   (I,r,t) |= Eφ iff (I,r’,t’) |= Ki φ for all pi (I,r,t) |= Cφ iff (I,r’,t’) |= Ekφ for all k
53. choose-your-own-adventure •  If you’d like to gloss over the proof and skip to other applications of knowledge, turn to page 62 •  If you’d like to dive into the weeds, turn to page 54.
54. Truth in a knowledge interpretation (I,r,t) |= Cφ iff (I,r’,t’) |= Ekφ for all k Fixed point axiom: Cφ = E(φ ∧ Cφ) Induction rule: From φ ⇒ E(φ ∧ ψ) infer φ ⇒ Cψ
55. communication is not guaranteed   NG1: For all runs r and times t, there exists a run r’ extending (r,t) such that […] no messages are received in r’ at or after time t. NG2: If in run r processor pi does not receive any messages in the interval (t’,t), then there is a run r’ extending (r,t’) such that […] h(pi ,r,t’’) = h(pi ,r’,t’’) for all t’’ < t, and no processor pj != pi receives a message in r’ in the interval (t’,t).  
56. Lemma 1 If, in two different runs (r and r’) of the same protocol, some h(p, r, t) = h(p, r’, t), then (I, r, t) |= Cφ iff (I, r’, t) |= Cφ Sorry, no proof today!
57. Common knowledge is not attainable in a system in which communication is not guaranteed Take runs r and r- in R, with the same initial configuration, s.t. no messages are received in r- up till time t. Then (I,r,t) |= Cφ iff (I,r-,t) |= Cφ. Proof (by induction on d(r)*):   •  Base case: d(r)=0. h(p1 ,r,t) = h(p1 ,r-,t). By Lemma 1, (I,r,t) |= Cφ iff (I,r-,t) |= Cφ. *  d(r)  is  the  number  of  messages  received  in  run  r.  
58. Common knowledge is not attainable in a system in which communication is not guaranteed Inductive case: d(r) = k+1. Let:   •  t’ < t -- the latest time a message is received in r before t. •  pj -- a processor that received a message at t’ •  pi –a processor (!= pj ) By NG2, there is a run r’ extending (r,t’) s.t. h(pi ,r,t’’)=h(pi ,r’,t’’) for all t’’ <= t, and all processors (besides pi ) receive no messages in the interval (t’, t). By construction, d(r’) <= k, so by the IH (I,r’,t) |= Cφ iff (I,r-,t) |= Cφ. But since h(pi ,r,t) = h(pi ,r’,t), by Lemma 1 (I,r’,t) |= Cφ iff (I,r,t) |= Cφ. So (I,r,t) |= Cφ iff (I,r-,t) |= Cφ. QED
59. Common knowledge is not attainable in a system in which communication is not guaranteed Review: we showed that common knowledge cannot be gained (or lost) by exchanging messages. Corollary: the 2 generals will never attack. But we still need to prove one more lemma: Any correct protocol for coordinated attack has the property that whenever the generals attack, it is common knowledge that they are attacking.
60. Lemma 2: coordinated attack requires common knowledge Let ψ = the generals are attacking Assume the generals (A and B) attack at (r*, t*) – we show that (I,r*,t*) |= Cψ. Pick an arbitrary point (r,t). We show ψ ⇒ Eψ is valid in R. •  If (I,r,t) |= ψ, then the generals attack at (r,t). Consider (r’,t’), in which A has the same history at (r,t). Since the protocol is deterministic (assumption), A must also attack in (r’,t’); since the protocol is correct, B does also, and so (I,r’,t’) |= ψ. It follows that (I,r,t) |= Eψ, so ψ ⇒ Eψ is valid in R. •  If (I,r,t) |= ¬ψ, then trivially ψ ⇒ Eψ is valid in R. By the induction rule, ψ ⇒ Cψ is valid in R
61. Coup de grace ψ = the generals are attacking 1.  By assumption, Cψ does not hold if no messages are exchanged. 2.  By theorem 1, Cψ will never hold. 3.  By lemma 2, the generals cannot attack unless Cψ.  
62. Phew. but…? Common knowledge is a prerequisite for agreement. Common knowledge is not attainable via protocol.
63. Halpern: These results may seem paradoxical.
64. Reality check Fragile assumptions on which the proofs rest: •  Deterministic protocol •  Simultaneous agreement is necessary •  “Communication not guaranteed” •  Lack of useful a priori common knowledge
65. Bootstrapping common knowledge •  The ``weakest failure detector’’ •  Spanner’s global clock •  Sequence wraparound
66. Applications of knowledge A correct distributed program achieves (nontrivial) distributed property X. Some tricky questions before we start coding: 1.  Is X even attainable? 2.  Cheapest protocol that gets me X? 3.  How should I implement it?
67. lower bounds for protocols [Hadzilacos, PODS’87]: A knowledge-theoretic analysis of atomic commitment protocols 1.  All of the variants of 2pc ((de-)centralized, linear/nested, etc) are identical from a knowledge perspective 2.  All 2PC variants attain the minimum level of knowledge needed to commit 3.  3PC attains the minimum needed to commit without blocking 4.  Lower bound for messages: nested 2PC.
68. A good paper about automatically choosing cheap coordination mechanisms
69. Applications of knowledge A correct distributed program achieves (nontrivial) distributed property X. Some tricky questions before we start coding: 1.  Is X even attainable? 2.  Cheapest protocol that gets me X? 3.  How should I implement it?
70. protocol implementation / synthesis •  Halpern and Fagin: knowledge-based programming [PODC’95]   case  of      K(Msg)  and  (KE(AckedMsg))  do  deliver(Msg)    K(Msg)  and  !KE(AckedMsg)  do  relay(Msg)       end   •  Matteo interlandi [Datalog2.0’11]: Knowlog: knowledge-enriched Dedalus  log(Tx_id,"abort")@next  :-­‐  Dvote(Vote,Tx_id),Vote=="no",                    par8cipants(X),transac8on(Tx_id,State),State=="vote-­‐req".    
71. A good paper about Dedalus
72. Monotonicity and knowledge Monotonic: the more you know, the more you know. Cϕ […] E3 ϕ E2 ϕ Eϕ Sϕ Dϕ ϕ  
73. A good paper about monotonicity and distributed consistency
74. Remember •  Knowledge is the dual of possibility •  Local knowledge dictates protocol behavior •  The purpose of protocols is obtaining a particular level of distributed knowledge •  Deep connections between semantic structures and system behavior •  Common knowledge is unattainable via protocol (but there is still hope)
75. Protocols  climb  the  hierarchy   Cϕ […] E3 ϕ E2 ϕ Eϕ Sϕ Dϕ ϕ   Knowledge  Highway   E10ϕ          10   E100ϕ                    100   Cϕ  ∞  
76. EOP   Queries?
77. Impact?    Sure!   •  Godel  prize  [1997]   – “for  defining  a  formal  no8on  of  ‘knowledge’  in   distributed  environments”   •  2009  Edsger  W.  Dijkstra  Prize  in  Distributed   Compu8ng  
78. The parable of the department store “When a man loses his wife in a department store without any prior understanding on where to meet if they get separated, the chances are good that they will find each other. It is likely that each will think of some obvious place to meet, so obvious that each will be sure that it is “obvious” to both of them. One does not simply predict where the other will go, which is wherever the first predicts the second to predict the first to go, and so ad infinitum. Not “What would I do if I were she?” but “What would I do if I were she wondering what she would do if she were wondering what I would do if I were she … ?”