Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Life Beyond The Illusion Of Present

Life Beyond The Illusion Of Present

The idea of the present is an illusion. Everything we see, hear and feel is just an echo from the past. But this illusion has influenced us and the way we view the world in so many ways; from Newton’s physics with a linearly progressing timeline accruing absolute knowledge along the way to the von Neumann machine with its total ordering of instructions updating mutable state with full control of the “present”. But unfortunately this is not how the world works. There is no present, all we have is facts derived from the merging of multiple pasts. The truth is closer to Einstein’s physics where everything is relative to one’s perspective.

As developers we need to wake up and break free from the perceived reality of living in a single globally consistent present. The advent of multicore and cloud computing architectures meant that most applications today are distributed systems—multiple cores separated by the memory bus or multiple nodes separated by the network—which puts a harsh end to this illusion. Facts travel at the speed of light (at best), which makes the distinction between past and perceived present even more apparent in a distributed system where latency is higher and where facts (messages) can get lost. 

Jonas Bonér

June 10, 2015
Tweet

More Decks by Jonas Bonér

Other Decks in Programming

Transcript

  1. Life BEYOND
    the ILLUSION
    of PRESENT
    Jonas Bonér
    CTO Typesafe
    @jboner

    View Slide

  2. "Time is what prevents everything from happening at once."
    - John Archibald Wheeler

    View Slide

  3. Newton's
    PHYSICS

    View Slide

  4. THIS SIMPLIFIED MODEL IS
    VERY APPEALING TO US

    View Slide

  5. von Neumann
    ARCHITECTURE

    View Slide

  6. BACK THEN
    LIFE WAS GOOD

    View Slide

  7. THEN, ALONG CAME
    CONCURRENCY
    MADE LIFE MISERABLE

    View Slide

  8. Jim Gray's
    TRANSACTIONS
    SAVE THE DAY

    View Slide

  9. View Slide

  10. WELL, ALONG CAME
    DISTRIBUTION
    MADE LIFE MISERABLE, again...

    View Slide

  11. But don't be surprised
    UNFORTUNATELY,
    THIS IS NOT
    HOW THE WORLD WORKS

    View Slide

  12. "The future is a function of the past."
    - A. P. Robertson

    View Slide

  13. "The (local) present is a merge function of
    multiple concurrent pasts."
    — Me

    View Slide

  14. val newLocalPresent = observedPasts.
    foldLeft(oldLocalPresent) { _ merge _ }

    View Slide

  15. WE NEED TO EXPLICITLY MODEL THE
    LOCAL PRESENT AS
    FACTS DERIVED FROM THE
    MERGING OF MULTIPLE
    CONCURRENT PASTS

    View Slide

  16. INFORMATION IS ALWAYS FROM THE PAST

    View Slide

  17. THE TRUTH
    IS CLOSER TO
    EINSTEIN'S
    PHYSICS

    View Slide

  18. INFORMATION HAS
    LATENCY

    View Slide

  19. THE COST OF MAINTAINING
    THIS ILLUSION IS INCREASED
    CONTENTION &
    COHERENCY

    View Slide

  20. AS LATENCY GETS HIGHER, THE
    ILLUSION CRACKS EVEN MORE

    View Slide

  21. DISTRIBUTED SYSTEMS
    EVERYWHERE

    View Slide

  22. THE NETWORK IS RELIABLE...NAT

    View Slide

  23. "If a tree falls in a forest and no one is around to hear it, does it make a
    sound?"
    — Charles Riborg Mann

    View Slide

  24. Information CAN (and will) GET LOST

    View Slide

  25. HOW DO WE DEAL WITH
    INFORMATION LOSS
    IN REAL LIFE?

    View Slide

  26. WE USE A SIMPLE PROTOCOL OF
    Confirm, Wait & Repeat

    View Slide

  27. We fill in THE BLANKS

    View Slide

  28. ...AND IF WE ARE WRONG, WE TAKE
    COMPENSATING ACTION
    APOLOGY-ORIENTED PROGRAMMING - PAT HELLAND (IN MEMORIES, GUESSES, AND APOLOGIES)

    View Slide

  29. The bottom line:
    WE CAN'T FORCE THE WORLD INTO A
    SINGLE GLOBALLY CONSISTENT
    PRESENT

    View Slide

  30. Should we just GIVE UP?

    View Slide

  31. I BELIEVE THAT THERE IS A
    PATH FORWARD

    View Slide

  32. WE NEED TO TREAT
    TIME AS A
    FIRST CLASS CONSTRUCT

    View Slide

  33. WHAT IS TIME, really?

    View Slide

  34. TIME IS THE
    SUCCESSION OF
    CAUSALLY RELATED EVENTS

    View Slide

  35. How can we
    MANAGE
    TIME?

    View Slide

  36. Think in FACTS

    View Slide

  37. What is a
    FACT?

    View Slide

  38. "A Fact is something that truly exists or happens: something that has
    actual existence, a true piece of information."
    — Merriam Webster

    View Slide

  39. IMMUTABILITY
    is a requirement

    View Slide

  40. So, do variables
    HAVE A PURPOSE IN LIFE?

    View Slide

  41. "The assignment statement is the von Neumann bottleneck of
    programming languages and keeps us thinking in word-at-a-time terms
    in much the same way the computer's bottleneck does."
    — John Backus (Turing Award lecture 1977)

    View Slide

  42. MUTABLE STATE
    NEEDS TO BE
    CONTAINED

    View Slide

  43. Ok, but how should we
    MANAGE FACTS?

    View Slide

  44. Functional
    PROGRAMMING

    View Slide

  45. Logic
    PROGRAMMING

    View Slide

  46. Dataflow
    PROGRAMMING

    View Slide

  47. NEVER
    DELETE
    FACTS

    View Slide

  48. "When bookkeeping was done with clay tablets or paper and ink,
    accountants developed some clear rules about good accounting
    practices. One never alters the books; if an error is made, it is
    annotated and a new compensating entry is made in the books. The
    books are thus a complete history of the transactions of the business.
    Update-in-place strikes many systems designers as a cardinal sin: it
    violates traditional accounting practices that have been observed for
    hundreds of years."
    — Jim Gray (1981)

    View Slide

  49. CRUD

    View Slide

  50. "Database is a cache of a subset of the log."
    — Pat Helland (2007)

    View Slide

  51. Store facts in an
    EVENT LOG

    View Slide

  52. The log allows
    TIME
    TRAVEL

    View Slide

  53. Can we REWRITE THE PAST?

    View Slide

  54. Allows us to shift our focus from
    DATA AT REST, to
    DATA IN MOTION

    View Slide

  55. Stream Processing

    View Slide

  56. CONSTRUCTING A SUFFICIENTLY CONSISTENT
    LOCAL PRESENT
    MEANS EMPLOYING
    CONSISTENCY MECHANISMS

    View Slide

  57. Consistency
    WHAT?
    WHY?
    WHEN?

    View Slide

  58. WE NEED TO DECOMPOSE THE SYSTEM USING
    CONSISTENCY BOUNDARIES

    View Slide

  59. INSIDE DATA: OUR CURRENT PRESENT
    OUTSIDE DATA: BLAST FROM THE PAST
    BETWEEN SERVICES: HOPE FOR THE FUTURE
    — PAT HELLAND (DATA ON THE INSIDE VS DATA ON THE OUTSIDE)

    View Slide

  60. MicroSERVICE

    View Slide

  61. AGGREGATE Root

    View Slide

  62. WITHIN THE CONSISTENCY BOUNDARY

    View Slide

  63. EVENT
    Sourcing

    View Slide

  64. BETWEEN THE
    Consistency
    Boundaries
    IT'S A ZOO

    View Slide

  65. Decoupling in
    TIME / SPACE

    View Slide

  66. STRONG
    CONSISTENCY
    The wrong default

    View Slide

  67. Here, we are living in the
    LOOMING SHADOW OF
    IMPOSSIBILITY
    THEOREMS

    View Slide

  68. FLP
    CONSENSUS IS IMPOSSIBLE

    View Slide

  69. PROTOCOLS CLIMB THE LADDER OF KNOWLEDGE
    Cɸ: Common Knowledge (infinite number of i)
    Eiɸ: (Everyone knows * i) ɸ
    E3ɸ: (Everyone knows * 3) ɸ
    E2ɸ: Everyone knows Everyone knows ɸ
    E1ɸ: Everyone knows ɸ
    Sɸ: Someone knows ɸ

    View Slide

  70. COMMON KNOWLEDGE
    IS NOT ATTAINABLE VIA PROTOCOL
    - JOSEPH HALPERN

    View Slide

  71. CAP
    CONSISTENCY IS IMPOSSIBLE

    View Slide

  72. Dissecting
    CAP

    View Slide

  73. "The first principle of successful scalability is to batter the consistency
    mechanisms down to a minimum."
    – James Hamilton

    View Slide

  74. EVENTUAL
    CONSISTENCY
    What does it really mean?

    View Slide

  75. Tracking TIME is tracking CAUSALITY

    View Slide

  76. RELYING ON
    TIMESTAMPS
    IS A BAD IDEA

    View Slide

  77. Instead, rely on
    LOGICAL TIME

    View Slide

  78. Lamport
    CLOCKS
    GLOBAL CAUSAL ORDERING BETWEEN

    View Slide

  79. Vector
    CLOCKS
    PARTIAL CAUSAL ORDERING BETWEEN EVENTS

    View Slide

  80. Causal
    CONSISTENCY

    View Slide

  81. What
    CONSISTENCY
    DO YOU REALLY NEED AND
    when?

    View Slide

  82. ACID 2.0
    ASSOCIATIVE
    COMMUTATIVE
    IDEMPOTENT
    DISTRIBUTED

    View Slide

  83. CONFLICT-FREE REPLICATED DATA TYPES

    View Slide

  84. DISORDERLY PROGRAMMING
    CALM THEOREM

    View Slide

  85. WE ARE JUST GETTING STARTED

    View Slide

  86. WE HAVE A LONG ROAD AHEAD OF US...

    View Slide

  87. View Slide

  88. Thanks
    FOR LISTENING

    View Slide

  89. Life BEYOND
    the ILLUSION
    of PRESENT
    Jonas Bonér
    CTO Typesafe
    @jboner

    View Slide