Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Declarative Imperative

The Declarative Imperative

Keynote talk, PODS 2010. Resulting paper appeared in SIGMOD Record http://db.cs.berkeley.edu/papers/sigrec10-declimperative.pdf. Paper abstract below.

The rise of multicore processors and cloud computing is putting
enormous pressure on the software community to find solutions to the difficulty of parallel and distributed programming. At the same time, there is more—and more varied—interest in data-centric programming languages than at any time in computing history, in part because these languages parallelize naturally. This juxtaposition raises the possibility that the theory of declarative database query languages can provide a foundation for the next generation of parallel and distributed programming languages.

In this paper I reflect on my group’s experience over seven years using Datalog extensions to build networking protocols and distributed systems. Based on that experience, I present a number of theoretical conjectures that may both interest the
database community, and clarify important practical issues in distributed computing. Most importantly, I make a case for database researchers to take a leadership role in addressing the impending programming crisis.

Joe Hellerstein

June 07, 2010
Tweet

More Decks by Joe Hellerstein

Other Decks in Technology

Transcript

  1. D E C L A R A T I V E
    I M P E R A T I V E
    XXXxXXXXXX
    JOSEPH M HELLERSTEIN BERKELEY
    THE
    EXPERIENCES AND CONJECTURES
    I N D I S T R I B U T E D L O G I C

    View full-size slide

  2. two unfinished stories
    urgency & resurgency
    a dedalus primer
    experience
    implications and conjecture
    TODAY

    View full-size slide

  3. two unfinished stories
    urgency & resurgency
    a dedalus primer
    experience
    implications and conjecture
    TODAY

    View full-size slide

  4. STORY #1: URGENCY
    A.K.A.
    The Programming Crisis

    View full-size slide

  5. Once upon a time there was a little
    chicken called Chicken Licken. One
    day, processor clock speeds stopped
    following Moore’s Law. Instead,
    hardware vendors started making
    multicore chips — one of which
    dropped on Chicken Licken’s head.
    DOOM AND GLOOM

    View full-size slide

  6. URGENCY
    “The sky is falling! The sky is
    falling! Computers won’t get any
    faster unless programmers learn to
    write parallel code!” squawked
    Chicken Licken.
    Henny Penny clucked in agreement:
    “Worse, there is Cloud Computing on
    t h e h o r i z o n , a n d i t r e q u i r e s
    programmers to write parallel AND
    distributed code!”

    View full-size slide

  7. URGENCY
    “I would be panicked if I were in
    industry!” said John Hennessy, then
    President of Stanford University.
    Many of his friends agreed, and
    together they set off to tell the
    funding agencies.

    View full-size slide

  8. STORY #2: RESURGENCY
    A.K.A.
    Springtime for Datalog

    View full-size slide

  9. URGENCY
    In a faraway land, database
    theoreticians had reason for cheer.
    Datalog variants, like crocuses in
    the snow, were cropping up in
    fields well outside the walled
    garden of PODS where they were
    first sown.
    SPRINGTIME FOR DATALOG
    http://www.flickr.com/photos/47262904@N00/107270153/
    http://www.flickr.com/photos/14293046@N00/3451413312/

    View full-size slide

  10. URGENCY
    Many examples of Datalog were
    blossoming:
    - security protocols
    - compiler analysis
    - natural language processing
    - probabilistic inference
    - modular robotics
    - multiplayer games
    And, in a patch of applied ground
    in Berkeley, a small group was
    p l a y i n g w i t h D a t a l o g f o r
    networking and distributed systems.
    Spring, John Collier

    View full-size slide

  11. URGENCY
    The Berkeley folk named their
    project BOOM, short for the
    Berkeley Orders Of Magnitude
    project. The name commemorated
    Jim Gray’s twelfth grand challenge,
    to make it Orders Of Magnitude
    easier to write software.
    They also chose a name for the
    language in the BOOM project:
    Bloom.

    View full-size slide

  12. THE END OF THE STORY?
    Doom and Gloom?
    BOOM and Bloom!

    View full-size slide

  13. THE END OF THE STORY?
    Doom and Gloom?
    BOOM and Bloom!
    be not chicken licken!

    View full-size slide

  14. THE END OF THE STORY?
    Doom and Gloom?
    BOOM and Bloom!
    be not chicken licken!
    give in to spring fever

    View full-size slide

  15. THE DECLARATIVE
    IMPERATIVE
    a dark period for programming, yes.
    but we have seen the light ... long ago!
    1980’s:
    parallel SQL
    computationally complete extensions to query langauges
    a way forward: extend languages that parallelize easily
    be not “embarrassed” by your parallelism
    spread the news: spring is dawning!
    crisis is opportunity
    go forth from the walled garden
    be fruitful and multiply
    http://www.flickr.com/photos/60145846@N00/258950784/

    View full-size slide

  16. ALONG THE WAY:
    TASTY PODS STUFF
    parallel complexity models for the cloud
    expressivity of logics w.r.t such models
    uncovering parallelism via LP properties
    semantics of distributed consistency
    time, time travel and fate
    "Concepts are delicious snacks with
    which we try to alleviate our amazement"
    — A. J. Heschel
    http://www.flickr.com/photos/megpi/861969/

    View full-size slide

  17. two unfinished stories
    a dedalus primer
    experience
    implications and conjecture
    TODAY

    View full-size slide

  18. two unfinished stories
    a dedalus primer
    experience
    implications and conjecture
    TODAY

    View full-size slide

  19. A BRIEF INTRODUCTION TO
    DEDALUS
    Stephen Dedalus
    http://ulyssesseen.com/landing/2009/04/stephen-dedalus/

    View full-size slide

  20. A BRIEF INTRODUCTION TO
    DEDALUS
    Stephen Dedalus Daedalus (and Icarus)
    http://ulyssesseen.com/landing/2009/04/stephen-dedalus/

    View full-size slide

  21. DEDALUS IS DATALOG
    + stratified negation/aggregation
    + a successor relation
    + a common final attribute in every predicate
    + unification on that last attribute

    View full-size slide

  22. BASIC DEDALUS

    View full-size slide

  23. BASIC DEDALUS
    deductive rules
    p(X, T) :- q(X, T).
    (i.e. “plain old datalog”, timestamps required)

    View full-size slide

  24. BASIC DEDALUS
    deductive rules
    p(X, T) :- q(X, T).
    (i.e. “plain old datalog”, timestamps required)
    inductive rules
    p(X, U) :- q(X, T), successor(T, U).
    (i.e. induction in time)

    View full-size slide

  25. BASIC DEDALUS
    deductive rules
    p(X, T) :- q(X, T).
    (i.e. “plain old datalog”, timestamps required)
    inductive rules
    p(X, U) :- q(X, T), successor(T, U).
    (i.e. induction in time)
    asynchronous rules
    p(X, Z) :- q(X, T), choice({X, T}, {Z}).
    (i.e. Z chosen non-deterministically
    per binding in the body [GZ98])

    View full-size slide

  26. SUGARED DEDALUS
    deductive rules
    p(X, T) :- q(X, T).
    inductive rules
    p(X, U) :- q(X, T), successor(T, U).
    asynchronous rules
    p(X, Z) :- q(X, T), choice({X, T}, {Z}).

    View full-size slide

  27. SUGARED DEDALUS
    deductive rules
    p(X) :- q(X).
    inductive rules
    p(X)@next :- q(X).
    asynchronous rules
    p(X)@async :- q(X).

    View full-size slide

  28. SUGARED DEDALUS
    deductive rules
    p(X) :- q(X).
    (omit ubiquitous timestamp attributes)
    inductive rules
    p(X)@next :- q(X).
    (sugar for induction in time)
    asynchronous rules
    p(X)@async :- q(X).
    (sugar for non-determinism in time)

    View full-size slide

  29. A LITTLE PROGRAM

    View full-size slide

  30. A LITTLE PROGRAM
    state(‘flip’)@1.

    View full-size slide

  31. A LITTLE PROGRAM
    state(‘flip’)@1.
    toggle(‘flop’) :- state(‘flip’).

    View full-size slide

  32. A LITTLE PROGRAM
    state(‘flip’)@1.
    toggle(‘flop’) :- state(‘flip’).
    toggle(‘flip’) :- state(‘flop’).

    View full-size slide

  33. A LITTLE PROGRAM
    state(‘flip’)@1.
    toggle(‘flop’) :- state(‘flip’).
    toggle(‘flip’) :- state(‘flop’).
    state(X)@next :- toggle(X).

    View full-size slide

  34. A LITTLE PROGRAM
    state(‘flip’)@1.
    toggle(‘flop’) :- state(‘flip’).
    toggle(‘flip’) :- state(‘flop’).
    state(X)@next :- toggle(X).
    announcement(X)@async :- toggle(X).

    View full-size slide

  35. PERSISTENCE:
    BE PERSISTENT

    View full-size slide

  36. PERSISTENCE:
    BE PERSISTENT
    “Accumulate-only” storage:
    pods(X)@next :- pods(X).
    pods(‘Ullman’)@1982.

    View full-size slide

  37. PERSISTENCE:
    BE PERSISTENT
    “Accumulate-only” storage:
    pods(X)@next :- pods(X).
    pods(‘Ullman’)@1982.
    Updatable storage:
    pods(X)@next :- pods(X), !del_pods(X).
    pods(‘Libkin’)@1996.
    del_pods(‘Libkin’)@2009.

    View full-size slide

  38. PERSISTENCE:
    BE PERSISTENT
    “Accumulate-only” storage:
    pods(X)@next :- pods(X).
    pods(‘Ullman’)@1982.
    Updatable storage:
    pods(X)@next :- pods(X), !del_pods(X).
    pods(‘Libkin’)@1996.
    del_pods(‘Libkin’)@2009. note: deletion via breaking induction
    Libkin did publish in PODS ’09

    View full-size slide

  39. ATOMICITY
    & VISIBILITY

    View full-size slide

  40. ATOMICITY
    & VISIBILITY
    Example: priority queue

    View full-size slide

  41. ATOMICITY
    & VISIBILITY
    Example: priority queue
    pq(V, P)@next :- pq(V, P), !del_pq(V, P).

    View full-size slide

  42. ATOMICITY
    & VISIBILITY
    Example: priority queue
    pq(V, P)@next :- pq(V, P), !del_pq(V, P).
    qmin(min) :- pq(V, P).

    View full-size slide

  43. ATOMICITY
    & VISIBILITY
    Example: priority queue
    pq(V, P)@next :- pq(V, P), !del_pq(V, P).
    qmin(min) :- pq(V, P). qmin “sees” only the current timestamp

    View full-size slide

  44. ATOMICITY
    & VISIBILITY
    Example: priority queue
    pq(V, P)@next :- pq(V, P), !del_pq(V, P).
    qmin(min) :- pq(V, P).
    del_pq(V,P) :- pq(V,P), qmin(P).
    out(V,P)@next :- pq(V,P), qmin(P).
    qmin “sees” only the current timestamp

    View full-size slide

  45. ATOMICITY
    & VISIBILITY
    Example: priority queue
    pq(V, P)@next :- pq(V, P), !del_pq(V, P).
    qmin(min) :- pq(V, P).
    del_pq(V,P) :- pq(V,P), qmin(P).
    out(V,P)@next :- pq(V,P), qmin(P).
    removes min from pq, adds to out.
    atomically visible at “next” time
    qmin “sees” only the current timestamp

    View full-size slide

  46. ATOMICITY
    & VISIBILITY
    Example: priority queue
    pq(V, P)@next :- pq(V, P), !del_pq(V, P).
    qmin(min) :- pq(V, P).
    del_pq(V,P) :- pq(V,P), qmin(P).
    out(V,P)@next :- pq(V,P), qmin(P).
    Two Dedalus features working together:
    timestamp unification controls visibility
    temporal induction “synchronizes” timestamp assignment
    removes min from pq, adds to out.
    atomically visible at “next” time
    qmin “sees” only the current timestamp

    View full-size slide

  47. two unfinished stories
    a dedalus primer
    experience
    implications and conjecture
    TODAY

    View full-size slide

  48. TODAY
    two unfinished stories
    a dedalus primer
    experience
    implications and conjecture

    View full-size slide

  49. TODAY
    two unfinished stories
    a dedalus primer
    experience
    implications and conjecture

    View full-size slide

  50. BUT FIRST, A GAME

    View full-size slide

  51. EXPERIENCE
    No practical applications of recursive query
    theory ... have been found to date.
    ...
    I find it sad that the theory community is so
    disconnected from reality that they don’t
    even know why their ideas are irrelevant.

    View full-size slide

  52. EXPERIENCE
    No practical applications of recursive query
    theory ... have been found to date.
    ...
    I find it sad that the theory community is so
    disconnected from reality that they don’t
    even know why their ideas are irrelevant.
    Hellerstein and Stonebraker,
    Readings in Database Systems
    3rd edition (1998)

    View full-size slide

  53. MORE EXPERIENCE

    View full-size slide

  54. MORE EXPERIENCE
    In the last 7 years we have built

    View full-size slide

  55. MORE EXPERIENCE
    In the last 7 years we have built
    distributed crawlers [Coo04,Loo04]

    View full-size slide

  56. MORE EXPERIENCE
    In the last 7 years we have built
    distributed crawlers [Coo04,Loo04]
    network routing protocols [Loo05a,Loo06b]

    View full-size slide

  57. MORE EXPERIENCE
    In the last 7 years we have built
    distributed crawlers [Coo04,Loo04]
    network routing protocols [Loo05a,Loo06b]
    overlay networks (e.g. Chord) [Loo06a]

    View full-size slide

  58. MORE EXPERIENCE
    In the last 7 years we have built
    distributed crawlers [Coo04,Loo04]
    network routing protocols [Loo05a,Loo06b]
    overlay networks (e.g. Chord) [Loo06a]
    a full-service embedded sensornet stack [Chu07]

    View full-size slide

  59. MORE EXPERIENCE
    In the last 7 years we have built
    distributed crawlers [Coo04,Loo04]
    network routing protocols [Loo05a,Loo06b]
    overlay networks (e.g. Chord) [Loo06a]
    a full-service embedded sensornet stack [Chu07]
    network caching/proxying [Chu09]

    View full-size slide

  60. MORE EXPERIENCE
    In the last 7 years we have built
    distributed crawlers [Coo04,Loo04]
    network routing protocols [Loo05a,Loo06b]
    overlay networks (e.g. Chord) [Loo06a]
    a full-service embedded sensornet stack [Chu07]
    network caching/proxying [Chu09]
    relational query optimizers (System R, Cascades, Magic Sets) [Con08]

    View full-size slide

  61. MORE EXPERIENCE
    In the last 7 years we have built
    distributed crawlers [Coo04,Loo04]
    network routing protocols [Loo05a,Loo06b]
    overlay networks (e.g. Chord) [Loo06a]
    a full-service embedded sensornet stack [Chu07]
    network caching/proxying [Chu09]
    relational query optimizers (System R, Cascades, Magic Sets) [Con08]
    distributed Bayesian inference (e.g. junction trees) [Atul09]

    View full-size slide

  62. MORE EXPERIENCE
    In the last 7 years we have built
    distributed crawlers [Coo04,Loo04]
    network routing protocols [Loo05a,Loo06b]
    overlay networks (e.g. Chord) [Loo06a]
    a full-service embedded sensornet stack [Chu07]
    network caching/proxying [Chu09]
    relational query optimizers (System R, Cascades, Magic Sets) [Con08]
    distributed Bayesian inference (e.g. junction trees) [Atul09]
    distributed consensus and commit (Paxos, 2PC) [Alv09]

    View full-size slide

  63. MORE EXPERIENCE
    In the last 7 years we have built
    distributed crawlers [Coo04,Loo04]
    network routing protocols [Loo05a,Loo06b]
    overlay networks (e.g. Chord) [Loo06a]
    a full-service embedded sensornet stack [Chu07]
    network caching/proxying [Chu09]
    relational query optimizers (System R, Cascades, Magic Sets) [Con08]
    distributed Bayesian inference (e.g. junction trees) [Atul09]
    distributed consensus and commit (Paxos, 2PC) [Alv09]
    a distributed file system (HDFS) [Alv10]

    View full-size slide

  64. MORE EXPERIENCE
    In the last 7 years we have built
    distributed crawlers [Coo04,Loo04]
    network routing protocols [Loo05a,Loo06b]
    overlay networks (e.g. Chord) [Loo06a]
    a full-service embedded sensornet stack [Chu07]
    network caching/proxying [Chu09]
    relational query optimizers (System R, Cascades, Magic Sets) [Con08]
    distributed Bayesian inference (e.g. junction trees) [Atul09]
    distributed consensus and commit (Paxos, 2PC) [Alv09]
    a distributed file system (HDFS) [Alv10]
    map-reduce job scheduler [Alv10]

    View full-size slide

  65. MORE EXPERIENCE
    In the last 7 years we have built
    distributed crawlers [Coo04,Loo04]
    network routing protocols [Loo05a,Loo06b]
    overlay networks (e.g. Chord) [Loo06a]
    a full-service embedded sensornet stack [Chu07]
    network caching/proxying [Chu09]
    relational query optimizers (System R, Cascades, Magic Sets) [Con08]
    distributed Bayesian inference (e.g. junction trees) [Atul09]
    distributed consensus and commit (Paxos, 2PC) [Alv09]
    a distributed file system (HDFS) [Alv10]
    map-reduce job scheduler [Alv10]
    + OOM smaller code
    + data independence (optimization)
    − 90% declarative Datalog variants:
    Overlog, NDLog, SNLog, ...

    View full-size slide

  66. DESIGN PATTERNS

    View full-size slide

  67. DESIGN PATTERNS
    despite flaws in our languages, patterns emerged
    three main categories today

    View full-size slide

  68. DESIGN PATTERNS
    despite flaws in our languages, patterns emerged
    three main categories today
    1. recursion (“rewriting the classics”)

    View full-size slide

  69. DESIGN PATTERNS
    despite flaws in our languages, patterns emerged
    three main categories today
    1. recursion (“rewriting the classics”)
    2. communication across space-time

    View full-size slide

  70. DESIGN PATTERNS
    despite flaws in our languages, patterns emerged
    three main categories today
    1. recursion (“rewriting the classics”)
    2. communication across space-time
    3. engine architecture: threads/events

    View full-size slide

  71. 1. RECURSION
    (REWRITING THE CLASSICS)

    View full-size slide

  72. 1. RECURSION
    (REWRITING THE CLASSICS)
    finding closure without the Ancs*

    View full-size slide

  73. 1. RECURSION
    (REWRITING THE CLASSICS)
    finding closure without the Ancs*
    * SIGMOD people can EMP-athize!

    View full-size slide

  74. 1. RECURSION
    (REWRITING THE CLASSICS)
    finding closure without the Ancs*
    the web is a graph.
    * SIGMOD people can EMP-athize!

    View full-size slide

  75. 1. RECURSION
    (REWRITING THE CLASSICS)
    finding closure without the Ancs*
    the web is a graph.
    e.g. crawlers = simple monotonic reachability
    * SIGMOD people can EMP-athize!

    View full-size slide

  76. 1. RECURSION
    (REWRITING THE CLASSICS)
    finding closure without the Ancs*
    the web is a graph.
    e.g. crawlers = simple monotonic reachability
    the internet is a graph.
    * SIGMOD people can EMP-athize!

    View full-size slide

  77. 1. RECURSION
    (REWRITING THE CLASSICS)
    finding closure without the Ancs*
    the web is a graph.
    e.g. crawlers = simple monotonic reachability
    the internet is a graph.
    e.g. routing protocols, overlay nets
    * SIGMOD people can EMP-athize!

    View full-size slide

  78. 1. RECURSION
    (REWRITING THE CLASSICS)
    finding closure without the Ancs*
    the web is a graph.
    e.g. crawlers = simple monotonic reachability
    the internet is a graph.
    e.g. routing protocols, overlay nets
    recursive queries matter!
    [Coo04,Loo04,Loo05,Loo06a,Loo06b]
    * SIGMOD people can EMP-athize!

    View full-size slide

  79. 1. RECURSION
    (REWRITING THE CLASSICS)
    finding closure without the Ancs*
    the web is a graph.
    e.g. crawlers = simple monotonic reachability
    the internet is a graph.
    e.g. routing protocols, overlay nets
    recursive queries matter!
    [Coo04,Loo04,Loo05,Loo06a,Loo06b]
    challenges:
    * SIGMOD people can EMP-athize!

    View full-size slide

  80. 1. RECURSION
    (REWRITING THE CLASSICS)
    finding closure without the Ancs*
    the web is a graph.
    e.g. crawlers = simple monotonic reachability
    the internet is a graph.
    e.g. routing protocols, overlay nets
    recursive queries matter!
    [Coo04,Loo04,Loo05,Loo06a,Loo06b]
    challenges:
    distributed join semantics
    * SIGMOD people can EMP-athize!

    View full-size slide

  81. 1. RECURSION
    (REWRITING THE CLASSICS)
    finding closure without the Ancs*
    the web is a graph.
    e.g. crawlers = simple monotonic reachability
    the internet is a graph.
    e.g. routing protocols, overlay nets
    recursive queries matter!
    [Coo04,Loo04,Loo05,Loo06a,Loo06b]
    challenges:
    distributed join semantics
    asynchronous fixpoint computation
    * SIGMOD people can EMP-athize!

    View full-size slide

  82. RECURSION + CHOICE =
    DYNAMIC PROGRAMMING
    many examples
    shortest paths [Loo05,Loo06b]
    query optimization
    Evita Raced: an overlog optimizer written in overlog [Con08]
    bottom-up and top-down DP written in datalog
    Viterbi inference [Wan10]
    main challenge
    distributed stratification

    View full-size slide

  83. 2. SPACE & COMMUNICATION
    location specifiers
    partition a relation across machines
    communication “falls out”
    declare each tuple’s “resting place”

    View full-size slide

  84. link(@X,Y,C)
    path(@X,Y,Y,C) :- link(@X,Y,C)
    path(@X,Z,Y,C+D) :- link(@X,Y,C), path(@Y,Z,N,D)
    LOCSPECS INDUCE
    COMMUNICATION

    View full-size slide

  85. link(@X,Y,C)
    path(@X,Y,Y,C) :- link(@X,Y,C)
    path(@X,Z,Y,C+D) :- link(@X,Y,C), path(@Y,Z,N,D)
    LOCSPECS INDUCE
    COMMUNICATION
    a b c d
    a b 1 c b 1
    c d 1
    b a 1
    b c 1
    link: d c 1
    link(@X,Y,C)

    View full-size slide

  86. link(@X,Y,C)
    path(@X,Y,Y,C) :- link(@X,Y,C)
    path(@X,Z,Y,C+D) :- link(@X,Y,C), path(@Y,Z,N,D)
    LOCSPECS INDUCE
    COMMUNICATION
    a b c d
    a b 1 c b 1
    c d 1
    b a 1
    b c 1
    link: d c 1
    a b b 1 c b b 1
    c d d 1
    b a a 1
    b c c 1
    path: d c c 1
    path(@X,Y,Y,C) :- link(@X,Y,C)

    View full-size slide

  87. link(@X,Y,C)
    path(@X,Y,Y,C) :- link(@X,Y,C)
    path(@X,Z,Y,C+D) :- link(@X,Y,C), path(@Y,Z,N,D)
    LOCSPECS INDUCE
    COMMUNICATION
    a b c d
    a b 1 c b 1
    c d 1
    b a 1
    b c 1
    link: d c 1
    a b b 1 c b b 1
    c d d 1
    b a a 1
    b c c 1
    path: d c c 1
    path(@X,Z,Y,C+D) :- link(@X,Y,C), path(@Y,Z,N,D)

    View full-size slide

  88. link(@X,Y,C)
    path(@X,Y,Y,C) :- link(@X,Y,C)
    path(@X,Z,Y,C+D) :- link(@X,Y,C), path(@Y,Z,N,D)
    LOCSPECS INDUCE
    COMMUNICATION
    a b c d
    a b 1 c b 1
    c d 1
    b a 1
    b c 1
    link: d c 1
    a b b 1 c b b 1
    c d d 1
    b a a 1
    b c c 1
    path: d c c 1
    path(@X,Z,Y,C+D) :- link(@X,Y,C), path(@Y,Z,N,D)

    View full-size slide

  89. link(@X,Y,C)
    path(@X,Y,Y,C) :- link(@X,Y,C)
    path(@X,Z,Y,C+D) :- link(@X,Y,C), path(@Y,Z,N,D)
    LOCSPECS INDUCE
    COMMUNICATION
    a b c d
    a b 1 c b 1
    c d 1
    b a 1
    b c 1
    link: d c 1
    a b b 1 c b b 1
    c d d 1
    b a a 1
    b c c 1
    path: d c c 1
    path(@X,Z,Y,C+D) :- link(@X,Y,C), path(@Y,Z,N,D)

    View full-size slide

  90. link(@X,Y,C)
    path(@X,Y,Y,C) :- link(@X,Y,C)
    path(@X,Z,Y,C+D) :- link(@X,Y,C), path(@Y,Z,N,D)
    LOCSPECS INDUCE
    COMMUNICATION
    a b c d
    a b 1 c b 1
    c d 1
    b a 1
    b c 1
    link: d c 1
    a b b 1 c b b 1
    c d d 1
    b a a 1
    b c c 1
    path: d c c 1
    path(@X,Z,Y,C+D) :- link(@X,Y,C), path(@Y,Z,N,D)

    View full-size slide

  91. link(@X,Y,C)
    path(@X,Y,Y,C) :- link(@X,Y,C)
    path(@X,Z,Y,C+D) :- link(@X,Y,C), path(@Y,Z,N,D)
    LOCSPECS INDUCE
    COMMUNICATION
    a b c d
    a b 1 c b 1
    c d 1
    b a 1
    b c 1
    link: d c 1
    a b b 1 c b b 1
    c d d 1
    b a a 1
    b c c 1
    path: d c c 1
    path(@X,Z,Y,C+D) :- link(@X,Y,C), path(@Y,Z,N,D)

    View full-size slide

  92. link(@X,Y)
    path(@X,Y,Y,C) :- link(@X,Y,C)
    link_d(X,@Y,C) :- link(@X,Y,C)
    path(@X,Z,Y,C+D) :- link_d(X,@Y,C), path(@Y,Z,N,D)
    LOCSPECS INDUCE
    COMMUNICATION
    a b c d
    a b 1 c d 1
    c d 1
    b a 1
    b c 1
    link: d c 1
    link_d:
    a b b 1 c d d 1
    d c c 1
    b a a 1
    b c c 1
    path: d c c 1
    Localization Rewrite

    View full-size slide

  93. link(@X,Y)
    path(@X,Y,Y,C) :- link(@X,Y,C)
    link_d(X,@Y,C) :- link(@X,Y,C)
    path(@X,Z,Y,C+D) :- link_d(X,@Y,C), path(@Y,Z,N,D)
    LOCSPECS INDUCE
    COMMUNICATION
    a b c d
    a b 1 c d 1
    c d 1
    b a 1
    b c 1
    link: d c 1
    link_d:
    a b b 1 c d d 1
    d c c 1
    b a a 1
    b c c 1
    path: d c c 1
    Localization Rewrite

    View full-size slide

  94. link(@X,Y)
    path(@X,Y,Y,C) :- link(@X,Y,C)
    link_d(X,@Y,C) :- link(@X,Y,C)
    path(@X,Z,Y,C+D) :- link_d(X,@Y,C), path(@Y,Z,N,D)
    LOCSPECS INDUCE
    COMMUNICATION
    a b c d
    a b 1 c d 1
    c d 1
    b a 1
    b c 1
    link: d c 1
    link_d:
    a b b 1 c d d 1
    d c c 1
    b a a 1
    b c c 1
    path: d c c 1
    a b 1
    Localization Rewrite

    View full-size slide

  95. link(@X,Y)
    path(@X,Y,Y,C) :- link(@X,Y,C)
    link_d(X,@Y,C) :- link(@X,Y,C)
    path(@X,Z,Y,C+D) :- link_d(X,@Y,C), path(@Y,Z,N,D)
    LOCSPECS INDUCE
    COMMUNICATION
    a b c d
    a b 1 c d 1
    c d 1
    b a 1
    b c 1
    link: d c 1
    link_d: b a 1 b c 1
    d c 1
    a b 1
    c b 1
    c d 1
    a b b 1 c d d 1
    d c c 1
    b a a 1
    b c c 1
    path: d c c 1
    a b 1
    Localization Rewrite

    View full-size slide

  96. link(@X,Y)
    path(@X,Y,Y,C) :- link(@X,Y,C)
    link_d(X,@Y,C) :- link(@X,Y,C)
    path(@X,Z,Y,C+D) :- link_d(X,@Y,C), path(@Y,Z,N,D)
    LOCSPECS INDUCE
    COMMUNICATION
    a b c d
    a b 1 c d 1
    c d 1
    b a 1
    b c 1
    link: d c 1
    link_d: b a 1 b c 1
    d c 1
    a b 1
    c b 1
    c d 1
    a b b 1 c d d 1
    d c c 1
    b a a 1
    b c c 1
    path: d c c 1
    a b 1
    Localization Rewrite

    View full-size slide

  97. link(@X,Y)
    path(@X,Y,Y,C) :- link(@X,Y,C)
    link_d(X,@Y,C) :- link(@X,Y,C)
    path(@X,Z,Y,C+D) :- link_d(X,@Y,C), path(@Y,Z,N,D)
    LOCSPECS INDUCE
    COMMUNICATION
    a b c d
    a b 1 c d 1
    c d 1
    b a 1
    b c 1
    link: d c 1
    link_d: b a 1 b c 1
    d c 1
    a b 1
    c b 1
    c d 1
    a b b 1 c d d 1
    d c c 1
    b a a 1
    b c c 1
    path: d c c 1
    a b 1
    Localization Rewrite

    View full-size slide

  98. link(@X,Y)
    path(@X,Y,Y,C) :- link(@X,Y,C)
    link_d(X,@Y,C) :- link(@X,Y,C)
    path(@X,Z,Y,C+D) :- link_d(X,@Y,C), path(@Y,Z,N,D)
    LOCSPECS INDUCE
    COMMUNICATION
    a b c d
    a b 1 c d 1
    c d 1
    b a 1
    b c 1
    link: d c 1
    link_d: b a 1 b c 1
    d c 1
    a b 1
    c b 1
    c d 1
    a b b 1 c d d 1
    d c c 1
    b a a 1
    b c c 1
    path: d c c 1
    a b 1
    a c b 2
    Localization Rewrite

    View full-size slide

  99. link(@X,Y)
    path(@X,Y,Y,C) :- link(@X,Y,C)
    link_d(X,@Y,C) :- link(@X,Y,C)
    path(@X,Z,Y,C+D) :- link_d(X,@Y,C), path(@Y,Z,N,D)
    LOCSPECS INDUCE
    COMMUNICATION
    a b c d
    a b 1 c d 1
    c d 1
    b a 1
    b c 1
    link: d c 1
    link_d: b a 1 b c 1
    d c 1
    a b 1
    c b 1
    c d 1
    a b b 1 c d d 1
    d c c 1
    b a a 1
    b c c 1
    path: d c c 1
    a b 1
    a c b 2
    Localization Rewrite
    THIS IS
    DISTANCE
    V E C T O R
    xx

    View full-size slide

  100. THE MYTH OF THE
    GLOBAL DATABASE
    the problem with space?
    distributed join consistency
    path(@X,Z,Y,C+D) :-
    link(@X,Y,C), path(@Y,Z,N,D)
    needs coordination, e.g. 2PC?
    “localized” async rules more “honest”
    perils of a false abstraction

    View full-size slide

  101. THE MYTH OF THE
    GLOBAL DATABASE
    the problem with space?
    distributed join consistency
    path(@X,Z,Y,C+D) :-
    link(@X,Y,C), path(@Y,Z,N,D)
    needs coordination, e.g. 2PC?
    “localized” async rules more “honest”
    perils of a false abstraction

    View full-size slide

  102. 3. ENGINE ARCHITECTURE
    engine architecture
    threads? events?
    join!
    session state w/events
    modeling ephemera
    events, timeouts, soft-state
    in the paper

    View full-size slide

  103. 3. ENGINE ARCHITECTURE
    engine architecture
    threads? events?
    join!
    session state w/events
    modeling ephemera
    events, timeouts, soft-state
    in the paper
    Because the original of the following paper by Lauer and Needham
    widely available, we are reprinting it here. If the paper is ref
    in published work, the citation should read: "Lauer, H.C., Needh
    "On the Duality of Operating Systems Structures," in Proc. Secon
    national Symposium on Operating Systems, IRIA, Oct. 1978, reprin
    Operating Systems Review, 13,2 April 1979, pp. 3-19.
    On the Duality of Operating System Structures
    Hugh C. Lauer
    Xerox Corporation
    Palo Alto, California
    Roger M. Needham*
    Cambridge University
    Cambridge, England
    Abstract
    Many operating system designs can be placed into one of two very ro
    categories, depending upon how they implement and use the notion
    process and synchronization. One category, the "Message-oriented Syst
    is characterized by a relatively small, static number of processes with
    explicit message system for communicating among them. The other categ
    the "Procedure-oriented System," is characterized by a large, ra
    changing number of small processes and a process synchroniza
    mechanism based on shared data.
    In this paper, it is demonstrated that these two categories are duals of e
    other and that a system which is constructed according to one model h
    direct counterpart in the other. The principal conclusion is that neither m
    is inherently preferable, and the main consideration for choosing betw
    them is the nature of the machine architecture upon which the system
    being built, not the application which the system will ultimately supp
    This is an empirical paper, in the sense of empirical studies in the natural sciences
    observed a number of samples from a class of objects and identified a classification
    their properties. We have then generalized our classification and constructed abstrac

    View full-size slide

  104. TODAY
    two unfinished stories
    a dedalus primer
    experience
    implications and conjecture

    View full-size slide

  105. TODAY
    two unfinished stories
    a dedalus primer
    experience
    implications and conjecture

    View full-size slide

  106. TODAY
    two unfinished stories
    a dedalus primer
    experience
    implications and conjecture

    View full-size slide

  107. IMPLICATIONS AND
    CONJECTURES
    the CALM conjecture
    the CRON conjecture
    Coordination Complexity
    the Fateful Time conjecture

    View full-size slide

  108. IMPLICATIONS AND
    CONJECTURES
    the CALM conjecture
    the CRON conjecture
    Coordination Complexity
    the Fateful Time conjecture

    View full-size slide

  109. IMPLICATIONS AND
    CONJECTURES
    the CALM conjecture
    the CRON conjecture
    Coordination Complexity
    the Fateful Time conjecture

    View full-size slide

  110. IMPLICATIONS AND
    CONJECTURES
    the CALM conjecture
    the CRON conjecture
    Coordination Complexity
    the Fateful Time conjecture

    View full-size slide

  111. IMPLICATIONS AND
    CONJECTURES
    the CALM conjecture
    the CRON conjecture
    Coordination Complexity
    the Fateful Time conjecture

    View full-size slide

  112. BUT FIRST, THE ENDGAME!

    View full-size slide

  113. COUNTING WAITS.
    WAITING COUNTS.
    distributed aggregation?
    esp. with recursion?!
    requires coordination (consider “count-to-zero”)
    counting requires waiting
    coordination protocols?
    all entail “voting”
    2PC, Paxos, BFT
    waiting requires counting

    View full-size slide

  114. IMPLICATIONS AND
    CONJECTURES
    the CALM conjecture
    the CRON conjecture
    Coordination Complexity
    the Fateful Time conjecture

    View full-size slide

  115. THE FUSS ABOUT
    EVENTUAL CONSISTENCY
    cloud folks, etc. don’t like transactions
    they involve waiting (counting)
    eventually consistent storage
    no waiting
    loose Consistency, but Availability during network Partitions
    things work out when partitions “eventually” reconnect
    (see Brewer’s CAP Theorem)
    spawned the noSQL movement

    View full-size slide

  116. MONOTONIC?
    EVENTUALLY CONSISTENT!
    my definition of eventual consistency
    given: distributed system, finite trace of messages
    eventual consistency if the final state of the system is independent of
    message ordering
    and ensuring so does not require coordination!
    more than the usual
    typical focus is on replicas and versions of state
    we are interested in consistency of a whole program
    replication is a special case: p_rep(X, @r)@async :- p(X, @a).

    View full-size slide

  117. EXAMPLE:
    SHOPPING CART
    shopping: a growing to-do list
    e.g., “add n units of item X to cart”
    e.g., “delete m units of item Y from cart”
    easily supported by eventually-consistent infrastructure
    check-out: aggregation
    compute totals
    validate stock-on-hand, confirm with user (and move on to billing logic)
    typically supported by richer infrastructure. not e.c.
    a well-known pattern
    “general ledger”, “escrow transactions”, etc.

    View full-size slide

  118. THE CALM CONJECTURE
    CONJECTURE 1. Consistency And Logical Monotonicity (CALM).
    A program has an eventually consistent, coordination-free evaluation
    strategy iff it is expressible in (monotonic) Datalog.
    monotonic 㱺 EC
    via pipelined semi-naive evaluation (PSN)
    positive derivations can “accumulate”
    !monotonic 㱺 !EC
    distributed negation/aggregation
    the end of the game!

    View full-size slide

  119. THE CALM CONJECTURE
    CONJECTURE 1. Consistency And Logical Monotonicity (CALM).
    A program has an eventually consistent, coordination-free evaluation
    strategy iff it is expressible in (monotonic) Datalog.
    monotonic 㱺 EC
    via pipelined semi-naive evaluation (PSN)
    positive derivations can “accumulate”
    !monotonic 㱺 !EC
    distributed negation/aggregation
    the end of the game!

    View full-size slide

  120. CALM IMPLICATIONS
    NoSQL = Datalog!
    ditto lock-free data structures
    whole-program tests over e.c. storage
    automatic relaxation of consistent programs
    synthesis of coordination/compensation

    View full-size slide

  121. IMPLICATIONS AND
    CONJECTURES
    the CALM conjecture
    the CRON conjecture
    Coordination Complexity
    the Fateful Time conjecture

    View full-size slide

  122. CAUSALITY
    (WHAT ABOUT PODC?)
    Lamport and his Clock Condition
    given a partial order → (happens-before)
    and a per-node clock C
    for any events a, b
    if a → b then C(a) < C(b)
    Respect Time & the (partial) Order!

    View full-size slide

  123. TIME IS FOR
    (NON-MONOTONIC) SUCKERS!

    View full-size slide

  124. TIME IS FOR
    (NON-MONOTONIC) SUCKERS!
    Time flies like an arrow.

    View full-size slide

  125. TIME IS FOR
    (NON-MONOTONIC) SUCKERS!
    Time flies like an arrow.
    Fruit flies like a banana.
    — Groucho Marx

    View full-size slide

  126. TIME TRAVEL
    we can send things back in time!
    nobody said we couldn’t!
    theoretician@async(X) :- pods(X).
    but ... temporal paradoxes?
    e.g. the grandfather paradox

    View full-size slide

  127. THE GRANDFATHER
    PARADOX

    View full-size slide

  128. THE GRANDFATHER
    PARADOX
    parent(X, Z) :- has_baby(X,Y,Z).

    View full-size slide

  129. THE GRANDFATHER
    PARADOX
    parent(X, Z) :- has_baby(X,Y,Z).
    parent(Y, Z) :- has_baby(X,Y,Z).

    View full-size slide

  130. THE GRANDFATHER
    PARADOX
    parent(X, Z) :- has_baby(X,Y,Z).
    parent(Y, Z) :- has_baby(X,Y,Z).
    parent@next(X,Y) :- parent(X,Y),
    !del_p(X,Y).

    View full-size slide

  131. THE GRANDFATHER
    PARADOX
    parent(X, Z) :- has_baby(X,Y,Z).
    parent(Y, Z) :- has_baby(X,Y,Z).
    parent@next(X,Y) :- parent(X,Y),
    !del_p(X,Y).
    anc(X, Y) :- parent(X, Y).

    View full-size slide

  132. THE GRANDFATHER
    PARADOX
    parent(X, Z) :- has_baby(X,Y,Z).
    parent(Y, Z) :- has_baby(X,Y,Z).
    parent@next(X,Y) :- parent(X,Y),
    !del_p(X,Y).
    anc(X, Y) :- parent(X, Y).
    anc(X, Y) :- parent(X,Z),
    anc(Z,Y).

    View full-size slide

  133. THE GRANDFATHER
    PARADOX
    parent(X, Z) :- has_baby(X,Y,Z).
    parent(Y, Z) :- has_baby(X,Y,Z).
    parent@next(X,Y) :- parent(X,Y),
    !del_p(X,Y).
    anc(X, Y) :- parent(X, Y).
    anc(X, Y) :- parent(X,Z),
    anc(Z,Y).
    kill@async(X,Y) :- mistreat(Y,X).

    View full-size slide

  134. THE GRANDFATHER
    PARADOX
    parent(X, Z) :- has_baby(X,Y,Z).
    parent(Y, Z) :- has_baby(X,Y,Z).
    parent@next(X,Y) :- parent(X,Y),
    !del_p(X,Y).
    anc(X, Y) :- parent(X, Y).
    anc(X, Y) :- parent(X,Z),
    anc(Z,Y).
    kill@async(X,Y) :- mistreat(Y,X).
    del_p(Y, Z) :- kill(X, Y).

    View full-size slide

  135. THE GRANDFATHER
    PARADOX
    parent(X, Z) :- has_baby(X,Y,Z).
    parent(Y, Z) :- has_baby(X,Y,Z).
    parent@next(X,Y) :- parent(X,Y),
    !del_p(X,Y).
    anc(X, Y) :- parent(X, Y).
    anc(X, Y) :- parent(X,Z),
    anc(Z,Y).
    kill@async(X,Y) :- mistreat(Y,X).
    del_p(Y, Z) :- kill(X, Y).
    Murder is Non-Monotonic.

    View full-size slide

  136. THE CRON CONJECTURE
    CONJECTURE 2. Causality Required Only for Non-Monotonicity.
    (CRON). Program semantics require causal message ordering if and
    only if the messages participate in non-monotonic derivations.
    intuition: local stratification
    assume a cycle through non-monotonic predicates across timesteps.
    looping derivations prevented if timestamps are monotonic

    View full-size slide

  137. IMPLICATIONS AND
    CONJECTURES
    the CALM conjecture
    the CRON conjecture
    Coordination Complexity
    the Fateful Time conjecture

    View full-size slide

  138. UNSTRATIFIABLE?
    SPEND SOME TIME.

    View full-size slide

  139. UNSTRATIFIABLE?
    SPEND SOME TIME.
    this is a problem:
    p(X) :- !p(X), q(X).

    View full-size slide

  140. UNSTRATIFIABLE?
    SPEND SOME TIME.
    this is a problem:
    p(X) :- !p(X), q(X).
    this is a solution:
    q(X)@next :- q(X).
    p(X)@next :- !p(X), q(X).

    View full-size slide

  141. UNSTRATIFIABLE?
    SPEND SOME TIME.
    this is a problem:
    p(X) :- !p(X), q(X).
    this is a solution:
    q(X)@next :- q(X).
    p(X)@next :- !p(X), q(X).
    this is just dumb:
    anc(X, Y)@next :- parent(X, Y).
    anc(X, Y)@next :- parent(X,Z),
    anc(Z,Y).

    View full-size slide

  142. UNSTRATIFIABLE?
    SPEND SOME TIME.
    this is a problem:
    p(X) :- !p(X), q(X).
    this is a solution:
    q(X)@next :- q(X).
    p(X)@next :- !p(X), q(X).
    how does Dedalus time
    relate to complexity?
    this is just dumb:
    anc(X, Y)@next :- parent(X, Y).
    anc(X, Y)@next :- parent(X,Z),
    anc(Z,Y).

    View full-size slide

  143. PRACTICAL (?? !!)
    SIDENOTE

    View full-size slide

  144. PRACTICAL (?? !!)
    SIDENOTE
    Challenge: win a benchmark with free computers.

    View full-size slide

  145. PRACTICAL (?? !!)
    SIDENOTE
    Challenge: win a benchmark with free computers.
    Yahoo Petasort:

    View full-size slide

  146. PRACTICAL (?? !!)
    SIDENOTE
    Challenge: win a benchmark with free computers.
    Yahoo Petasort:
    3,800 8-core, 4-disk machines

    View full-size slide

  147. PRACTICAL (?? !!)
    SIDENOTE
    Challenge: win a benchmark with free computers.
    Yahoo Petasort:
    3,800 8-core, 4-disk machines
    i.e. each core sorted 32 MB (1/512 of RAM!)

    View full-size slide

  148. PRACTICAL (?? !!)
    SIDENOTE
    Challenge: win a benchmark with free computers.
    Yahoo Petasort:
    3,800 8-core, 4-disk machines
    i.e. each core sorted 32 MB (1/512 of RAM!)
    3799/3800 of a Petabyte streamed across the network

    View full-size slide

  149. PRACTICAL (?? !!)
    SIDENOTE
    Challenge: win a benchmark with free computers.
    Yahoo Petasort:
    3,800 8-core, 4-disk machines
    i.e. each core sorted 32 MB (1/512 of RAM!)
    3799/3800 of a Petabyte streamed across the network
    16.25 hours

    View full-size slide

  150. PRACTICAL (?? !!)
    SIDENOTE
    Challenge: win a benchmark with free computers.
    Yahoo Petasort:
    3,800 8-core, 4-disk machines
    i.e. each core sorted 32 MB (1/512 of RAM!)
    3799/3800 of a Petabyte streamed across the network
    16.25 hours
    rental cost in the cloud

    View full-size slide

  151. PRACTICAL (?? !!)
    SIDENOTE
    Challenge: win a benchmark with free computers.
    Yahoo Petasort:
    3,800 8-core, 4-disk machines
    i.e. each core sorted 32 MB (1/512 of RAM!)
    3799/3800 of a Petabyte streamed across the network
    16.25 hours
    rental cost in the cloud
    Amazon EC2 “High-CPU extra large” @ $0.84/hour

    View full-size slide

  152. PRACTICAL (?? !!)
    SIDENOTE
    Challenge: win a benchmark with free computers.
    Yahoo Petasort:
    3,800 8-core, 4-disk machines
    i.e. each core sorted 32 MB (1/512 of RAM!)
    3799/3800 of a Petabyte streamed across the network
    16.25 hours
    rental cost in the cloud
    Amazon EC2 “High-CPU extra large” @ $0.84/hour
    3800 * 0.84 * 16.25 = $51,870

    View full-size slide

  153. PRACTICAL (?? !!)
    SIDENOTE
    Challenge: win a benchmark with free computers.
    Yahoo Petasort:
    3,800 8-core, 4-disk machines
    i.e. each core sorted 32 MB (1/512 of RAM!)
    3799/3800 of a Petabyte streamed across the network
    16.25 hours
    rental cost in the cloud
    Amazon EC2 “High-CPU extra large” @ $0.84/hour
    3800 * 0.84 * 16.25 = $51,870
    not a perfect clone, but rather impressive

    View full-size slide

  154. PRACTICAL (?? !!)
    SIDENOTE
    Challenge: win a benchmark with free computers.
    Yahoo Petasort:
    3,800 8-core, 4-disk machines
    i.e. each core sorted 32 MB (1/512 of RAM!)
    3799/3800 of a Petabyte streamed across the network
    16.25 hours
    rental cost in the cloud
    Amazon EC2 “High-CPU extra large” @ $0.84/hour
    3800 * 0.84 * 16.25 = $51,870
    not a perfect clone, but rather impressive
    pretty close to free

    View full-size slide

  155. PRACTICAL (?? !!)
    SIDENOTE
    Challenge: win a benchmark with free computers.
    Yahoo Petasort:
    3,800 8-core, 4-disk machines
    i.e. each core sorted 32 MB (1/512 of RAM!)
    3799/3800 of a Petabyte streamed across the network
    16.25 hours
    rental cost in the cloud
    Amazon EC2 “High-CPU extra large” @ $0.84/hour
    3800 * 0.84 * 16.25 = $51,870
    not a perfect clone, but rather impressive
    pretty close to free
    so where’s the complexity?

    View full-size slide

  156. COORDINATION
    COMPLEXITY
    coordination the main cost
    failure/delay probabilities
    compounded by queuing effects
    coordination complexity:
    # of sequential coordination steps
    required for evaluation
    CALM: coordination manifest in logic!
    coordination at stratum boundaries

    View full-size slide

  157. DEDALUS TIME AND
    COORD COMPLEXITY
    CONJECTURE 3. Dedalus Time ⇔ Coordination Complexity. The
    minimum number of Dedalus timesteps required to evaluate a program
    on a given input data set is equivalent to the program’s Coordination
    Complexity.

    View full-size slide

  158. IMPLICATIONS AND
    CONJECTURES
    the CALM conjecture
    the CRON conjecture
    Coordination Complexity
    the Fateful Time conjecture

    View full-size slide

  159. BUT WHAT IS TIME FOR?
    we’ve seen when we don’t need it
    monotonic deduction
    we’ve seen when we do need it
    “spending time” examples
    if we need it but try to save it?
    no unique minimal model!
    multiple simultaneous worlds
    paradoxes: inconsistent assertions in time

    View full-size slide

  160. FATEFUL TIME
    CONJECTURE 4. Fateful Time. Any Dedalus program P can be
    rewritten into an equivalent temporally-minimized program P’ such that
    each inductive or asynchronous rule of P’ is necessary: converting that
    rule to a deductive rule would result in a program with no unique
    minimal model.
    the purpose of time is to seal fate:
    time = simultaneity + succession
    dedalus: timestamp unification + inductive rules
    multiple worlds 㱺 monotonic sequence of unique worlds

    View full-size slide

  161. TODAY
    two unfinished stories
    a dedalus primer
    experience
    implications and conjecture

    View full-size slide

  162. TODAY
    two unfinished stories
    a dedalus primer
    experience
    implications and conjecture

    View full-size slide

  163. WHAT NEXT? PITFALLS,
    PROMISE & POTENTIAL
    audacity of scope
    pitfall: database languages per se
    promise: data finally the central issue in computing
    potential: attack the general case, change the way software is built
    formalism
    pitfall: disconnection of theory/practice
    promise: theory embodied in useful programming tools
    potential: validate and extend a 30-year agenda
    networking
    pitfall: the walled garden
    promise: db topics connect pl, os, distributed systems, etc.
    potential: db as an intellectual crossroads

    View full-size slide

  164. CARPE DIEM
    affirm, refute, or ignore the conjectures
    (thank you for indulging me)
    but do not miss this opportunity!
    we can address a real crisis in computing
    we have the ear of the broad community
    time to sift through known results and apply them
    undoubtedly there is more to do .. jump in!

    View full-size slide

  165. JOINT WORK
    7 years
    3 systems (P2, Overlog, DSN)
    6 PhD, 2 MS students
    friends in academia, industry
    special thanks to the
    BOOM team:
    Peter ALVARO
    Ras BODÍK
    Tyson CONDIE
    Neil CONWAY
    Khaled ELMELEEGY
    Haryadi GUNAWI
    Thibaud HOTTELIER
    William MARCZAK
    Rusty SEARS

    View full-size slide

  166. web search: “springtime for datalog”
    http://boom.cs.berkeley.edu

    View full-size slide

  167. DESIGN PATTERN #3
    EVENTS AND DISPATCH
    challenge: manage thousands of sessions on a server
    A. “process” or “thread” per session.
    stack variables and PC keep context
    B: one single-threaded event-loop
    state-machine per session on heap
    problem: long tasks like I/O require care
    arguments about scaling, programmability
    session mgmt is just data mgmt!
    scale a join to thousands of tuples? big deal!!
    programmability? hmm...
    Because the original of the following paper by Lauer and Needham
    widely available, we are reprinting it here. If the paper is ref
    in published work, the citation should read: "Lauer, H.C., Needh
    "On the Duality of Operating Systems Structures," in Proc. Secon
    national Symposium on Operating Systems, IRIA, Oct. 1978, reprin
    Operating Systems Review, 13,2 April 1979, pp. 3-19.
    On the Duality of Operating System Structures
    Hugh C. Lauer
    Xerox Corporation
    Palo Alto, California
    Roger M. Needham*
    Cambridge University
    Cambridge, England
    Abstract
    Many operating system designs can be placed into one of two very ro
    categories, depending upon how they implement and use the notion
    process and synchronization. One category, the "Message-oriented Syst
    is characterized by a relatively small, static number of processes with
    explicit message system for communicating among them. The other categ
    the "Procedure-oriented System," is characterized by a large, ra
    changing number of small processes and a process synchroniza
    mechanism based on shared data.
    In this paper, it is demonstrated that these two categories are duals of e
    other and that a system which is constructed according to one model h
    direct counterpart in the other. The principal conclusion is that neither m
    is inherently preferable, and the main consideration for choosing betw

    View full-size slide

  168. A THIRD WAY
    // keep requests pending until a response is generated
    pending(Id, Clnt, P) :- request(Clnt, Id, P).
    pending(Id, Clnt, P)@next :- pending(Id, Clnt, P),
    !response(Id, Clnt, _).

    View full-size slide

  169. A THIRD WAY
    // keep requests pending until a response is generated
    pending(Id, Clnt, P) :- request(Clnt, Id, P).
    pending(Id, Clnt, P)@next :- pending(Id, Clnt, P),
    !response(Id, Clnt, _).
    // call an asynchronous service, via input “interface” service_in()
    service_out(P, Out)@async :- request(Id, Clnt, P),
    service_in(P, Out).

    View full-size slide

  170. A THIRD WAY
    // keep requests pending until a response is generated
    pending(Id, Clnt, P) :- request(Clnt, Id, P).
    pending(Id, Clnt, P)@next :- pending(Id, Clnt, P),
    !response(Id, Clnt, _).
    // call an asynchronous service, via input “interface” service_in()
    service_out(P, Out)@async :- request(Id, Clnt, P),
    service_in(P, Out).
    // join service answers back to pending to form response
    response(Clnt, Id, O) :- pending(Id, Clnt, P), service_out(P, O).

    View full-size slide

  171. EPHEMERA
    3 common distributed persistence models
    stable storage (persistent)
    event streams (ephemeral)
    soft state (bounded persistence)

    View full-size slide

  172. EPHEMERA
    3 common distributed persistence models
    stable storage (persistent)
    event streams (ephemeral)
    soft state (bounded persistence)

    View full-size slide

  173. EPHEMERA
    3 common distributed persistence models
    stable storage (persistent)
    event streams (ephemeral)
    soft state (bounded persistence)

    View full-size slide

  174. EPHEMERA
    3 common distributed persistence models
    stable storage (persistent)
    event streams (ephemeral)
    soft state (bounded persistence)

    View full-size slide

  175. EPHEMERA
    3 common distributed persistence models
    stable storage (persistent)
    event streams (ephemeral)
    soft state (bounded persistence)

    View full-size slide

  176. OVERLOG: PERIODICS
    AND PERSISTENCE
    Overlog provided metadata modifiers for persistence
    materialize(pods, infinity).
    materialize(cache, 60).
    absence of a materialize clause implies an emphemeral event stream
    Overlog’s built-in event stream:
    periodic(@Node, Id, Interval).
    a declarative construct, to be evaluated in real-time

    View full-size slide

  177. CACHING EXAMPLE
    IN OVERLOG
    materialize(pods, infinity).
    materialize(msglog, infinity).
    materialize(link, infinity).
    materialize(cache, 60).
    cache(@N, X) :- pods(@M, X), link(@M, N),
    periodic(@M, _, 40).
    msglog(@N, X) :- cache(@N, X). but what does that mean??
    cool!

    View full-size slide

  178. CACHING
    IN DEDALUS
    pods(@M, X)@next :- pods(@M,X), !del_pods(@M,X).
    msglog(@M,X)@next) , msglog(@M,X), !del_msglog(@M,X).
    link(@M, X)@next :- link(@M,X), !del_link(@M,X).
    cache(@M,X,Birth)@next :- cache(@M,X,Birth), now() - Birth > 60.
    cache(@N, X) :- pods(@M, X), link(@M, N),
    periodic(@M, _, 40).
    msglog(@N, X) :- cache(@N, X). in tandem with inductive rule above,
    msglog grounded in this base-case!
    still cool!

    View full-size slide

  179. GRAY’S TWELFTH
    CHALLENGE
    “automatic” programming
    Do What I Mean
    3 OOM “easier”
    with Memex, Turing Test, etc.
    predates multicore/cloud
    the sky had already fallen?
    44
    Automatic Programming
    Do What I Mean (not 100$ Line of code!, no programming bugs)
    The holy grail of programming languages & systems
    12.  Devise a specification language or UI
    1.  That is easy for people to express designs (1,000x easier),
    2.  That computers can compile, and
    3.  That can describe all applications (is complete).
    •  System should “reason” about application
    –  Ask about exception cases.
    –  Ask about incomplete specification.
    –  But not be onerous.
    •  This already exists in domain-specific areas.
    (i.e. 2 out of 3 already exists)
    •  An imitation game for a programming staff.

    View full-size slide

  180. MONOTONIC?
    EMBARRASSING!
    Monotonic evaluation is order-independent
    derivation trees “accumulate”
    Loo’s Pipelined Semi-Naive evaluation
    streaming (monotonic) Datalog
    same # derivations as Semi-Naive
    Intuition: network paths again

    View full-size slide

  181. Link  Table Network
    SEMI-NAIVE EVALUATION
    Slide  courtesy  Boon  Thau  Loo

    View full-size slide

  182. Path  Table
    Link  Table Network
    SEMI-NAIVE EVALUATION
    Slide  courtesy  Boon  Thau  Loo

    View full-size slide

  183. Path  Table
    1-­‐hop
    Link  Table Network
    SEMI-NAIVE EVALUATION
    Slide  courtesy  Boon  Thau  Loo

    View full-size slide

  184. Path  Table
    1-­‐hop
    Link  Table Network
    SEMI-NAIVE EVALUATION
    Slide  courtesy  Boon  Thau  Loo

    View full-size slide

  185. Path  Table
    1-­‐hop
    Link  Table Network
    SEMI-NAIVE EVALUATION
    Slide  courtesy  Boon  Thau  Loo

    View full-size slide

  186. Path  Table
    1-­‐hop
    2-­‐hop
    Link  Table Network
    SEMI-NAIVE EVALUATION
    Slide  courtesy  Boon  Thau  Loo

    View full-size slide

  187. Path  Table
    1-­‐hop
    2-­‐hop
    Link  Table Network
    SEMI-NAIVE EVALUATION
    Slide  courtesy  Boon  Thau  Loo

    View full-size slide

  188. Path  Table
    1-­‐hop
    2-­‐hop
    Link  Table Network
    SEMI-NAIVE EVALUATION
    Slide  courtesy  Boon  Thau  Loo

    View full-size slide

  189. Path  Table
    3-­‐hop
    1-­‐hop
    2-­‐hop
    Link  Table Network
    SEMI-NAIVE EVALUATION
    Slide  courtesy  Boon  Thau  Loo

    View full-size slide

  190. Path  Table
    3-­‐hop
    1-­‐hop
    2-­‐hop
    Link  Table Network
    SEMI-NAIVE EVALUATION
    Slide  courtesy  Boon  Thau  Loo

    View full-size slide

  191. Path  Table
    Link  Table Network
    0
    4
    1
    2
    3
    PIPELINED
    SEMI-NAIVE EVALUATION

    View full-size slide

  192. Path  Table
    1
    Link  Table Network
    0
    4
    1
    2
    3
    PIPELINED
    SEMI-NAIVE EVALUATION

    View full-size slide

  193. Path  Table
    2
    1
    Link  Table Network
    0
    4
    1
    2
    3
    PIPELINED
    SEMI-NAIVE EVALUATION

    View full-size slide

  194. Path  Table
    2
    1
    3
    Link  Table Network
    0
    4
    1
    2
    3
    PIPELINED
    SEMI-NAIVE EVALUATION

    View full-size slide

  195. Path  Table
    2
    1
    3
    Link  Table Network
    4
    0
    4
    1
    2
    3
    PIPELINED
    SEMI-NAIVE EVALUATION

    View full-size slide

  196. Path  Table
    2
    1
    3
    Link  Table Network
    4
    0
    4
    1
    2
    3
    PIPELINED
    SEMI-NAIVE EVALUATION

    View full-size slide

  197. BORGES SAID IT BETTER
    “The denial of time involves two negations: the negation of
    the succession of the terms of a series, negation of the
    synchronism of the terms in two different series.”
    — Jorge Luis Borges, “A New Refutation of Time”

    View full-size slide