Keep CALM and Query On

Conventional distributed systems wisdom dictates that perfect consistency is too expensive to guarantee in general, and consistency mechanisms—if you use them at all—should be reserved for infrequent, small-scale, mission-critical tasks. Like most design maxims, these ideas are not so easy to translate into practice; all kinds of unavoidable tactical questions pop up, e.g.:

* Exactly where in my multifaceted system is loose consistency “good enough” to meet application needs?
* How do I know that my “mission-critical” software isn’t tainted by my “best effort” components?
* How do I ensure that my design maxims are maintained as software and developer teams evolve?

Until recently, answers to these questions have been more a matter of folklore than mathematics. (One way to tell the difference: a good answer is enforceable by a compiler.)

In this talk, I will describe the CALM Theorem that links Consistency And Logical Monotonicity, and discuss how it can inform distributed software development. I'll also give a taste of Bloom, a "disorderly" distributed programming language whose compiler can automatically answer questions like the ones above. Along the way, I'll try to shed light on side questions like "Should Paxos exist?" and "Causality: What is it good for?"

Transcript

1. JOE HELLERSTEIN BERKELEY TRIFACTA PROGRAMMING PRINCIPLES FOR A DISTRIBUTED ERA
2. ± OUTLINE BY ANALOGY
3. THE GLORY OF HISTORY http://i.dailymail.co.uk/i/pix/2008/09/05/ article-1052881-028A5C6000000578-592_468x564.jpg
4. ELEGANTL Y TENACIOUS http://livingincheapluxury.files.wordpress.com/2012/06/ voc_pool_mona_1_pic_elizabeth_2_closed_eyes.jpg
5. FRUSTRATINGL Y PERSISTENT h"p://www.dailymail.co.uk/news/ar6cle-­‐1085234/The-­‐Prince-­‐Angst-­‐Troubled-­‐ Charles-­‐turns-­‐60-­‐fearing-­‐public-­‐accepted-­‐Camilla-­‐-­‐worrying-­‐hell-­‐king.html  
6. ± KEEP CALM AND CARRY ON h"p://www.theurbangent.com/2011/04/style-­‐of-­‐prince-­‐william-­‐of-­‐ wales-­‐duke-­‐of-­‐cambridge.html  
7. HISTORY TENACITY FRUSTRATION CALM ± OUTLINE
8. ± CALM VON NEUMANN TRANSACTIONS CAP CALM ± OUTLINE
9. HISTORY TENACITY FRUSTRATION CALM ± OUTLINE
10. ± CALM VON NEUMANN TRANSACTIONS CAP CALM ± OUTLINE
11. THE von NEUMANN MACHINE •  ORDER –  LIST of Instructions –  ARRAY of Memory •  THE STATE –  Mutation in time h"p://en.wikipedia.org/wiki/File:JohnvonNeumann-­‐LosAlamos.gif  
12. THE von NEUMANN MACHINE •  ORDER –  LIST of Instructions –  ARRAY of Memory •  THE STATE –  Mutation in time
13. http://www.flickr.com/photos/scobleizer/4870003098/sizes/l/in/ photostream/ ORDER AND THE STATE
14. HISTORY TENACITY FRUSTRATION CALM ± OUTLINE
15. ± CALM VON NEUMANN TRANSACTIONS CAP CALM ± OUTLINE
16. RACE DISORDER AND THE STATE h"p://www.flickr.com/photos/60057912@N00/4845067066/  
17. THE TRANSACTION CONCEPT •  ORDER –  Disorder across transactions –  Illusion of order within transactions •  THE STATE –  Registers, Memory •  Isolation –  Mutation in time •  Atomicity h"p://www.ny6mes.com/2009/12/15/science/15books.html  
18. THE TRANSACTION CONCEPT •  ELEGANT THEORY –  Serializability •  PRACTICAL ENGINEERING –  A transparent illusion •  Easy to ensure correct applications •  Tricky to scale infrastructure h"p://www.ny6mes.com/2009/12/15/science/15books.html  
19. SUMMARY ± TRIUMPH OF ORDER ± TRIUMPH OF THE STATE ± ELEGANT ILLUSION OF ORDER AND STATE –  FORMAL THEORY –  NATURAL API –  EFFICIENT IMPLEMENTATION
20. HISTORY TENACITY FRUSTRATION CALM ± OUTLINE
21. ± CALM VON NEUMANN TRANSACTIONS CAP CALM ± OUTLINE
22. ORDER, THE STATE AND GLOBALIZATION h"p://www.flickr.com/photos/29487767@N02/3574392846/  
23. http://www.flickr.com/photos/scobleizer/4870003098/sizes/l/in/ photostream/ DISTRIBUTED COMPUTING IS THE NEW NORMAL •  ORDER IS TOO COSTL Y –  Synchronization –  Coordination •  THE STATE IS HEARSAY –  Delay –  Failure –  Partition
24. THE CAP THEOREM
25. THE CAP THEOREM MIND THE CAP
26. A NEGATIVE RESULT FOR A TIME OF DISILLUSIONMENT MIND THE CAP
27. COPING WITH DISORDER •  DESIGN MAXIMS –  Commutative methods –  Inverse methods –  Free coupons •  PRACTICAL ENGINEERING –  Pragmatic Systems •  Easy to scale infrastructure •  Tricky to ensure correct applications
28. COPING WITH DISORDER •  DESIGN MAXIMS –  Commutative methods –  Inverse methods –  Apologies •  PRACTICAL ENGINEERING –  Pragmatic Systems •  Easy to scale infrastructure •  Tricky to ensure correct applications THE TRANSACTION CONCEPT •  ELEGANT THEORY –  Serializability •  PRACTICAL ENGINEERING –  A transparent illusion •  Easy to ensure correct applications •  Tricky to scale infrastructure
29. SUMMARY ELEGANCE & ORDER EXPENSIVE ILLUSIONS MAXIMS & DISORDER FRAGILE APPLICATIONS MIND THE CAP
30. HISTORY TENACITY FRUSTRATION CALM ± OUTLINE
31. ± CALM VON NEUMANN TRANSACTIONS CAP CALM ± OUTLINE
32. POSITIVE THINKING FOR THE CLOUDY FUTURE http://www.flickr.com/photos/scobleizer/4870003098/sizes/l/in/ photostream/
33. COPING WITH DISORDER •  DESIGN MAXIMS –  Commutative methods –  Inverse methods –  Apologies •  PRACTICAL ENGINEERING –  Pragmatic Systems •  Easy to scale infrastructure •  Tricky to ensure correct applications THE TRANSACTION CONCEPT •  ELEGANT THEORY –  Serializability •  PRACTICAL ENGINEERING –  A transparent illusion •  Easy to ensure correct applications •  Tricky to scale infrastructure
34. THEORY FOR APPLICATIONS COMPILERS TRUMP INFRASTRUCTURE ± CALM
35. ELEGANCE AND DISORDER •  ELEGANT THEORY –  Maxims ⇒ Theorems •  Lattices •  ± CALM Theorem •  PRACTICAL ENGINEERING –  Theorems ⇒ Compilers •  <~ bloom •  CALM Analysis http://www.flickr.com/photos/scobleizer/4870003098/sizes/l/in/ photostream/ ± CALM
36. h"p://www.flickr.com/photos/25579597@N00/40955407/  
37. Associative –  (X ◦ Y) ◦ Z = X ◦ (Y ◦ Z) –  batch-insensitive Commutative –  X ◦Y = Y ◦ X –  order-insensitive Idempotent –  X ◦ X = X –  resend-insensitive Distributed –  acronym-insensitive h"p://www.flickr.com/photos/25579597@N00/40955407/  
38. Associative –  (X ◦ Y) ◦ Z = X ◦ (Y ◦ Z) –  batch-insensitive Commutative –  X ◦Y = Y ◦ X –  order-insensitive Idempotent –  X ◦ X = X –  resend-insensitive Distributed –  acronym-insensitive
39. Storing an Integer VON NEUMANN int ctr; operator:= (x) { // assign ctr = x; } ACID 2.0 int ctr; operator<= (x) { // merge ctr = MAX(ctr, x); } DISORDERL Y INPUT STREAMS: 2, 5, 6, 7, 11, 22, 44, 91 5, 7, 2, 11, 44, 6, 22, 91, 5
40. Storing an Integer VON NEUMANN ACID 2.0 DISORDERED INPUT STREAMS: 2, 5, 6, 7, 11, 22, 44, 91 5, 7, 2, 11, 44, 6, 22, 91, 5 0   10   20   30   40   50   60   70   80   90   100   1   2   3   4   5   6   7   8   9   0   10   20   30   40   50   60   70   80   90   100   1   2   3   4   5   6   7   8   9  
41. PROGRESS •  Lemma: –  ACID 2.0 ⇒ monotonic •  Lemma: –  ACID 2.0 ⇒ confluent •  Corollary: –  ACID 2.0 ⇒ convergent –  a.k.a. “Eventually Consistent” •  No coordination! http://www.flickr.com/photos/scobleizer/4870003098/sizes/l/in/ photostream/
42. CRDTs convergent replicated data types [Shapiro, et al. 2011] •  Semilattice objects –  A class –  merge() is ACID 2.0 •  Many examples: –  int w/ Max –  set w/ Union –  map w/Insert –  …
43. CRDTs convergent replicated data types [Shapiro, et al. 2011]
44. CRDTs convergent replicated data types [Shapiro, et al. 2011] SCOPE DILEMMA –  SINGLE-OBJECT PROGRAMS? –  PROVE ACID 2.0 •  formalism? •  unit testing?
45. DESIRE: COMPOSITION •  PIECEWISE ANAL YSIS –  Multiple simple CRDTs –  Each easy to test –  Rules for composition •  SET LATTICES KNOWN –  Database query languages •  select/project/join rules •  even with recursion! –  Distributed Datalog •  see P2, etc. •  CONSISTENCY? … http://www.flickr.com/photos/44606255@N00/370973576/
46. CONSISTENCY AS LOGICAL MONOTONICITY ± THE CALM THEOREM
47. ± THE CALM THEOREM 1.  MONOTONICITY ⇒ EVENTUAL CONSISTENCY 2.  WHEN TO COORDINATE? NON-MONOTONE OPERATORS
48. THEORY ⇒ COMPILER COMPILERS TRUMP INFRASTRUCTURE ± KEEPING CALM
49. ANACHRONISM A THING BELONGING OR APPROPRIATE TO A PERIOD OTHER THAN THAT IN WHICH IT EXISTS http://www.flickr.com/photos/scobleizer/4870003098/sizes/l/in/ photostream/ "anachronism". Oxford Dictionaries. April 2010. Oxford Dictionaries. April 2010. Oxford University Press. 07 October 2012 <http://oxforddictionaries.com/definition/ english/anachronism>.
50. <~ bloom! A disorderly language of lattices and mappings. Encourages monotonicity. Highlights non- monotonicity. Designed for distribution.
51. <~ bloom operational model! •  really  a  metaphor  for  a  logic  called  dedalus   •  each  node  runs  independently   –  local  clock,  local  data,  local  execu6on   •  6mestepped  execu6on  loop  at  each  node   local%updates bloom%rules atomic,%local system%events network } { network now next
52. Hello World in <~ bloom # a chat server bloom do nodelist <= connect.map {|c| c.val} mcast <~ (mcast*nodelist).pairs { |m,n| [n.key, m.val] } end
53. Hello World in <~ bloom # a chat server bloom do nodelist <= connect.map {|c| c.val} mcast <~ (mcast*nodelist).pairs { |m,n| [n.key, m.val] } end sets of key/value pairs
54. Hello World in <~ bloom # a chat server bloom do nodelist <= connect.map {|c| c.val} mcast <~ (mcast*nodelist).pairs { |m,n| [n.key, m.val] } end instantaneous merge (union) async merge (union)
55. Hello World in <~ bloom # a chat server bloom do nodelist <= connect.map {|c| c.val} mcast <~ (mcast*nodelist).pairs { |m,n| [n.key, m.val] } end monotone functions
56. Monotone Function f monotone if x ≤ y ⇒ f(x) ≤ f(y)
57. Hello World in <~ bloom # a chat server bloom do nodelist <= connect.map {|c| c.val} mcast <~ (mcast*nodelist).pairs { |m,n| [n.key, m.val] } end monotone functions
58. state do table :nodelist channel :connect channel :mcast end # a chat server bloom do nodelist <= connect.map {|c| c.val} mcast <~ (mcast*nodelist).pairs { |m,n| [n.key, m.val] } end Tables and Channels See Getting Started docs on github
59. Hello World in <~ bloom state do table :nodelist channel :connect channel :mcast end # a chat server bloom do nodelist <= connect.map {|c| c.val} mcast <~ (mcast*nodelist).pairs { |m,n| [n.key, m.val] } end MONOTONIC PROGRAM hence EVENTUALLY CONSISTENT Lattice merge + Monotone Functions ——————
60. state do table :nodelist channel :connect channel :mcast lmax :cnt lbool :crowded end # a chat server bloom do nodelist <= connect.map {|c| c.val} mcast <~ (mcast*nodelist).pairs { |m,n| [n.key, m.val] } cnt <= nodelist.group([], count()) crowded <= cnt.gt(100) end More Lattices
61. state do table :nodelist channel :connect channel :mcast lmax :cnt # integer with MAX lbool :crowded # bool with OR end # a chat server bloom do nodelist <= connect.map {|c| c.val} mcast <~ (mcast*nodelist).pairs { |m,n| [n.key, m.val] } cnt <= nodelist.group([], count()) crowded <= cnt.gt(100) end More Lattices
62. state do table :nodelist channel :connect channel :mcast lmax :cnt # integer with MAX lbool :crowded # bool with OR end # a chat server bloom do nodelist <= connect.map {|c| c.val} mcast <~ (mcast*nodelist).pairs { |m,n| [n.key, m.val] } cnt <= nodelist.group([], count()) crowded <= cnt.gt(100) end More Lattices monotone functions across lattice types
63. Monotone Functions Across Lattice Types group/   count   gt(1)  
64. state do table :nodelist channel :connect channel :mcast channel :disconnect end # a chat server bloom do nodelist <= connect.map {|c| c.val} mcast <~ (mcast*nodelist).pairs { |m,n| [n.key, m.val] } nodelist <- disconnect.map {|c| c.val} end Non-Monotonicity Downstream of Asynchrony Non-monotone function async collection
65. CALM Analysis! 1.  For any path through a Bloom module, label: –  Asynchrony –  Non-Monotonicity –  Inconsistency 2.  Compute labels transitively across modules 3.  Identify code that needs coordination 4.  Assess comm pattern, suggest coordination –  1-1, 1-many : ordered delivery –  many-many : order proxy, Paxos, etc.
66. Alvaro Diagrams! Basic  KVS   LaZce  KVS  
67. Vector Clocks in Bloom! module VectorClock state do lmap :my_vc lmap :next_vc scratch :in_msg, [:addr, :payload] => [:clock] scratch :out_msg, [:addr, :payload] scratch :out_msg_vc, [:addr, :payload] => [:clock] end bootstrap do my_vc <= {ip_port => Bud::MaxLattice.new(0)} end bloom do next_vc <= out_msg { {ip_port => my_vc.at(ip_port) + 1} } out_msg_vc <= out_msg {|m| [m.addr, m.payload, next_vc]} next_vc <= in_msg { {ip_port => my_vc.at(ip_port) + 1} } next_vc <= my_vc next_vc <= in_msg {|m| m.clock} my_vc <+ next_vc end end
68. •  Ini6ally  all  clocks  are  zero.   •  Each  6me  a  process  experiences  an   internal  event,  it  increments  its  own   logical  clock  in  the  vector  by  one.   •  Each  6me  a  process  prepares  to  send  a   message,  it  increments  its  own   logical  clock  in  the  vector  by  one  and   then  sends  its  en6re  vector  along  with   the  message  being  sent.   •  Each  6me  a  process  receives  a  message,   it  increments  its  own   logical  clock  in  the  vector  by  one  and   updates  each  element  in  its  vector  by   taking  the  maximum  of  the  value  in  its   own  vector  clock  and  the  value  in  the   vector  in  the  received  message  (for   every  element).   Vector Clocks
 bloom v. wikipedia! bootstrap do
 my_vc <= 
 {ip_port => Bud::MaxLattice.new(0)} 
 end bloom do
 next_vc <= out_msg 
 { {ip_port => my_vc.at(ip_port) + 1} } out_msg_vc <= out_msg 
 {|m| [m.addr, m.payload, next_vc]} 
 next_vc <= in_msg 
 { {ip_port => my_vc.at(ip_port) + 1} } 
 next_vc <= my_vc
 next_vc <= in_msg {|m| m.clock}
 my_vc <+ next_vc
 end
69. ± THE CALM THEOREM COROLLARIES: WHY COORDINATE? CRON CAUSALITY REQUIRED ONL Y FOR NON-MONOTONICITY COORDINATION COMPLEXITY HOW MUCH COORDINATION IS TRUL Y NEEDED FOR YOUR ALGORITHM? FATEFUL TIME THE ONL Y USE FOR “TIME” IS TO “SEAL FATE”.
70. SUMMARY: UNITY MAXIMIZE DISORDER AND UNDERSTAND ORDER’S ROLE DISORDERL Y CODE AND WHOLE-PROGRAM ANAL YSIS MIND THE CAP ± CALM <~ bloom!
71. BOOM TEAM joe  hellerstein   ras  bodik   peter  alvaro   neil  conway   bill  marczak   haryadi  gunawi   david  maier   peter  bailis   sriram  srinivasan   emily  andrews   andy  hutchinson   alan  fekete  
72. bloom-lang.org boom.cs.berkeley.edu •  Papers –  CALM/Bloom, CIDR ‘11 –  Bloom+Lattices, SOCC ‘12 –  BloomUnit, DBTest ’12 •  Videos –  Declarative Imperative, PODS ’10 –  Bloom, Lang.Next ’12 –  Bloom+Lattices, Basho Meetup ‘12 ± MORE CALM &&! <~ bloom!
73. ACM SOCC NEXT TUES 10/16 10:45AM SAN JOSE MARRIOT DATA CONSISTENCY SESSION •  NEIL CONWAY ON LATTICE SUPPORT IN BLOOM •  PETER BAILIS ON POTENTIAL DANGERS OF CAUSAL CONSISTENCY http://www.socc2012.org ± MORE CALM &&! <~ bloom!
74. POSITIVE THINKING FOR THE CLOUDY FUTURE bloom-lang.org