Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Life Beyond The Illusion Of Present

Life Beyond The Illusion Of Present

The idea of the present is an illusion. Everything we see, hear and feel is just an echo from the past. But this illusion has influenced us and the way we view the world in so many ways; from Newton’s physics with a linearly progressing timeline accruing absolute knowledge along the way to the von Neumann machine with its total ordering of instructions updating mutable state with full control of the “present”. But unfortunately this is not how the world works. There is no present, all we have is facts derived from the merging of multiple pasts. The truth is closer to Einstein’s physics where everything is relative to one’s perspective.

As developers we need to wake up and break free from the perceived reality of living in a single globally consistent present. The advent of multicore and cloud computing architectures meant that most applications today are distributed systems—multiple cores separated by the memory bus or multiple nodes separated by the network—which puts a harsh end to this illusion. Facts travel at the speed of light (at best), which makes the distinction between past and perceived present even more apparent in a distributed system where latency is higher and where facts (messages) can get lost. 

Jonas Bonér

June 10, 2015
Tweet

More Decks by Jonas Bonér

Other Decks in Programming

Transcript

  1. Life BEYOND
    the ILLUSION
    of PRESENT
    Jonas Bonér
    CTO Typesafe
    @jboner

    View full-size slide

  2. "Time is what prevents everything from happening at once."
    - John Archibald Wheeler

    View full-size slide

  3. Newton's
    PHYSICS

    View full-size slide

  4. THIS SIMPLIFIED MODEL IS
    VERY APPEALING TO US

    View full-size slide

  5. von Neumann
    ARCHITECTURE

    View full-size slide

  6. BACK THEN
    LIFE WAS GOOD

    View full-size slide

  7. THEN, ALONG CAME
    CONCURRENCY
    MADE LIFE MISERABLE

    View full-size slide

  8. Jim Gray's
    TRANSACTIONS
    SAVE THE DAY

    View full-size slide

  9. WELL, ALONG CAME
    DISTRIBUTION
    MADE LIFE MISERABLE, again...

    View full-size slide

  10. But don't be surprised
    UNFORTUNATELY,
    THIS IS NOT
    HOW THE WORLD WORKS

    View full-size slide

  11. "The future is a function of the past."
    - A. P. Robertson

    View full-size slide

  12. "The (local) present is a merge function of
    multiple concurrent pasts."
    — Me

    View full-size slide

  13. val newLocalPresent = observedPasts.
    foldLeft(oldLocalPresent) { _ merge _ }

    View full-size slide

  14. WE NEED TO EXPLICITLY MODEL THE
    LOCAL PRESENT AS
    FACTS DERIVED FROM THE
    MERGING OF MULTIPLE
    CONCURRENT PASTS

    View full-size slide

  15. INFORMATION IS ALWAYS FROM THE PAST

    View full-size slide

  16. THE TRUTH
    IS CLOSER TO
    EINSTEIN'S
    PHYSICS

    View full-size slide

  17. INFORMATION HAS
    LATENCY

    View full-size slide

  18. THE COST OF MAINTAINING
    THIS ILLUSION IS INCREASED
    CONTENTION &
    COHERENCY

    View full-size slide

  19. AS LATENCY GETS HIGHER, THE
    ILLUSION CRACKS EVEN MORE

    View full-size slide

  20. DISTRIBUTED SYSTEMS
    EVERYWHERE

    View full-size slide

  21. THE NETWORK IS RELIABLE...NAT

    View full-size slide

  22. "If a tree falls in a forest and no one is around to hear it, does it make a
    sound?"
    — Charles Riborg Mann

    View full-size slide

  23. Information CAN (and will) GET LOST

    View full-size slide

  24. HOW DO WE DEAL WITH
    INFORMATION LOSS
    IN REAL LIFE?

    View full-size slide

  25. WE USE A SIMPLE PROTOCOL OF
    Confirm, Wait & Repeat

    View full-size slide

  26. We fill in THE BLANKS

    View full-size slide

  27. ...AND IF WE ARE WRONG, WE TAKE
    COMPENSATING ACTION
    APOLOGY-ORIENTED PROGRAMMING - PAT HELLAND (IN MEMORIES, GUESSES, AND APOLOGIES)

    View full-size slide

  28. The bottom line:
    WE CAN'T FORCE THE WORLD INTO A
    SINGLE GLOBALLY CONSISTENT
    PRESENT

    View full-size slide

  29. Should we just GIVE UP?

    View full-size slide

  30. I BELIEVE THAT THERE IS A
    PATH FORWARD

    View full-size slide

  31. WE NEED TO TREAT
    TIME AS A
    FIRST CLASS CONSTRUCT

    View full-size slide

  32. WHAT IS TIME, really?

    View full-size slide

  33. TIME IS THE
    SUCCESSION OF
    CAUSALLY RELATED EVENTS

    View full-size slide

  34. How can we
    MANAGE
    TIME?

    View full-size slide

  35. Think in FACTS

    View full-size slide

  36. What is a
    FACT?

    View full-size slide

  37. "A Fact is something that truly exists or happens: something that has
    actual existence, a true piece of information."
    — Merriam Webster

    View full-size slide

  38. IMMUTABILITY
    is a requirement

    View full-size slide

  39. So, do variables
    HAVE A PURPOSE IN LIFE?

    View full-size slide

  40. "The assignment statement is the von Neumann bottleneck of
    programming languages and keeps us thinking in word-at-a-time terms
    in much the same way the computer's bottleneck does."
    — John Backus (Turing Award lecture 1977)

    View full-size slide

  41. MUTABLE STATE
    NEEDS TO BE
    CONTAINED

    View full-size slide

  42. Ok, but how should we
    MANAGE FACTS?

    View full-size slide

  43. Functional
    PROGRAMMING

    View full-size slide

  44. Logic
    PROGRAMMING

    View full-size slide

  45. Dataflow
    PROGRAMMING

    View full-size slide

  46. NEVER
    DELETE
    FACTS

    View full-size slide

  47. "When bookkeeping was done with clay tablets or paper and ink,
    accountants developed some clear rules about good accounting
    practices. One never alters the books; if an error is made, it is
    annotated and a new compensating entry is made in the books. The
    books are thus a complete history of the transactions of the business.
    Update-in-place strikes many systems designers as a cardinal sin: it
    violates traditional accounting practices that have been observed for
    hundreds of years."
    — Jim Gray (1981)

    View full-size slide

  48. "Database is a cache of a subset of the log."
    — Pat Helland (2007)

    View full-size slide

  49. Store facts in an
    EVENT LOG

    View full-size slide

  50. The log allows
    TIME
    TRAVEL

    View full-size slide

  51. Can we REWRITE THE PAST?

    View full-size slide

  52. Allows us to shift our focus from
    DATA AT REST, to
    DATA IN MOTION

    View full-size slide

  53. Stream Processing

    View full-size slide

  54. CONSTRUCTING A SUFFICIENTLY CONSISTENT
    LOCAL PRESENT
    MEANS EMPLOYING
    CONSISTENCY MECHANISMS

    View full-size slide

  55. Consistency
    WHAT?
    WHY?
    WHEN?

    View full-size slide

  56. WE NEED TO DECOMPOSE THE SYSTEM USING
    CONSISTENCY BOUNDARIES

    View full-size slide

  57. INSIDE DATA: OUR CURRENT PRESENT
    OUTSIDE DATA: BLAST FROM THE PAST
    BETWEEN SERVICES: HOPE FOR THE FUTURE
    — PAT HELLAND (DATA ON THE INSIDE VS DATA ON THE OUTSIDE)

    View full-size slide

  58. MicroSERVICE

    View full-size slide

  59. AGGREGATE Root

    View full-size slide

  60. WITHIN THE CONSISTENCY BOUNDARY

    View full-size slide

  61. EVENT
    Sourcing

    View full-size slide

  62. BETWEEN THE
    Consistency
    Boundaries
    IT'S A ZOO

    View full-size slide

  63. Decoupling in
    TIME / SPACE

    View full-size slide

  64. STRONG
    CONSISTENCY
    The wrong default

    View full-size slide

  65. Here, we are living in the
    LOOMING SHADOW OF
    IMPOSSIBILITY
    THEOREMS

    View full-size slide

  66. FLP
    CONSENSUS IS IMPOSSIBLE

    View full-size slide

  67. PROTOCOLS CLIMB THE LADDER OF KNOWLEDGE
    Cɸ: Common Knowledge (infinite number of i)
    Eiɸ: (Everyone knows * i) ɸ
    E3ɸ: (Everyone knows * 3) ɸ
    E2ɸ: Everyone knows Everyone knows ɸ
    E1ɸ: Everyone knows ɸ
    Sɸ: Someone knows ɸ

    View full-size slide

  68. COMMON KNOWLEDGE
    IS NOT ATTAINABLE VIA PROTOCOL
    - JOSEPH HALPERN

    View full-size slide

  69. CAP
    CONSISTENCY IS IMPOSSIBLE

    View full-size slide

  70. Dissecting
    CAP

    View full-size slide

  71. "The first principle of successful scalability is to batter the consistency
    mechanisms down to a minimum."
    – James Hamilton

    View full-size slide

  72. EVENTUAL
    CONSISTENCY
    What does it really mean?

    View full-size slide

  73. Tracking TIME is tracking CAUSALITY

    View full-size slide

  74. RELYING ON
    TIMESTAMPS
    IS A BAD IDEA

    View full-size slide

  75. Instead, rely on
    LOGICAL TIME

    View full-size slide

  76. Lamport
    CLOCKS
    GLOBAL CAUSAL ORDERING BETWEEN

    View full-size slide

  77. Vector
    CLOCKS
    PARTIAL CAUSAL ORDERING BETWEEN EVENTS

    View full-size slide

  78. Causal
    CONSISTENCY

    View full-size slide

  79. What
    CONSISTENCY
    DO YOU REALLY NEED AND
    when?

    View full-size slide

  80. ACID 2.0
    ASSOCIATIVE
    COMMUTATIVE
    IDEMPOTENT
    DISTRIBUTED

    View full-size slide

  81. CONFLICT-FREE REPLICATED DATA TYPES

    View full-size slide

  82. DISORDERLY PROGRAMMING
    CALM THEOREM

    View full-size slide

  83. WE ARE JUST GETTING STARTED

    View full-size slide

  84. WE HAVE A LONG ROAD AHEAD OF US...

    View full-size slide

  85. Thanks
    FOR LISTENING

    View full-size slide

  86. Life BEYOND
    the ILLUSION
    of PRESENT
    Jonas Bonér
    CTO Typesafe
    @jboner

    View full-size slide