Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How shit works: Time

How shit works: Time

A talk given at GeeCON 2018 in Prague, Czech Republic:

In this talk we'll take a hard look at one of the most commonly used, and at least as commonly misunderstood, elements in software engineering: time. Time is so fundamental to the way humans experience reality that we don't normally give it a second thought, but it's just as fundamental to software systems. Without a correct model for working with time BAD THINGS HAPPEN: data is persisted out of order, exceptions occur where they shouldn't be possible, and production systems blow up.

We'll cover the various common representations of time, acknowledge their caveats and deficiencies, and hopefully learn a few new tools and practices along the way.

0014decc65763e66f22891be724b5afa?s=128

Tomer Gabel

October 19, 2018
Tweet

More Decks by Tomer Gabel

Other Decks in Programming

Transcript

  1. How shit works: Time Tomer Gabel, WeWork Prague, 18-19 October

    2018
  2. @tomerg THE STORY OF ADI

  3. @tomerg Meet Adi.

  4. @tomerg He has a dream.

  5. @tomerg Adi is fashionable • Event sourcing is the future,

    they say • Event sourcing is the shit, he agrees
  6. @tomerg Adi is fashionable • Event sourcing is the future,

    they say • Event sourcing is the shit, he agrees • … and designs an event model Created Modified Published Archived Restored
  7. @tomerg Column Type Key Null site_id binary(16) ✔ ✖ event_time

    timestamp ✔ ✖ event_type enum(…) ✖ ✖ payload blob ✖ ✖ Adi is fashionable
  8. @tomerg HEY, KIDS Can you guess what happens next?

  9. @tomerg Adi is confused • Bad shit happens • Event

    streams exhibit:
  10. @tomerg Adi is confused • Bad shit happens • Event

    streams exhibit: – Out of order events – Conflicting events – Impossible states Created Archived Updated WTF?!
  11. How shit works: Time Tomer Gabel, WeWork @ GeeCON Prague

    2018 Image: Vera Kratochvil (public domain)
  12. @tomerg Time (noun) “… the system of those sequential relations

    that any event has to any other, as past, present, or future; indefinite and continuous duration regarded as that in which events succeed one another.” -- dictionary.com
  13. @tomerg Modeling time • Encoding – Resolution – Epoch “Real”

    time Epoch Resolution T1 T 2 T3 T4 T 5
  14. @tomerg Modeling time • Encoding – Resolution – Epoch T3

    Instant (or “event”)
  15. @tomerg Modeling time • Encoding – Resolution – Epoch T3

    A B Conflicting events
  16. @tomerg Modeling time • Encoding – Resolution – Epoch •

    Bootstrapping Epoch ???
  17. @tomerg Modeling time • Encoding – Resolution – Epoch •

    Bootstrapping – Manual Image courtesy of Jim Leonard
  18. @tomerg Modeling time • Encoding – Resolution – Epoch •

    Bootstrapping – Manual – Battery-backed Image: pc-restorer.com
  19. @tomerg Modeling time • Encoding – Resolution – Epoch •

    Bootstrapping – Manual – Battery-backed – NTP Image: FDA graphic by Michael J. Ermarth (public domain)
  20. @tomerg Modeling time • Encoding – Resolution – Epoch •

    Bootstrapping – Manual – Battery-backed – NTP • Updating T = T+1 T1 T 2 W hen?
  21. @tomerg System Clock (RTC) Modeling time • Encoding – Resolution

    – Epoch • Bootstrapping – Manual – Battery-backed – NTP • Updating Image: Jeremy Saglimbeni on Vimeo (CC BY-SA 3.0)
  22. @tomerg RTC isn’t perfect • Alas, clocks drift • Subtle

    causes – Temperature – Power supply – General relativity – Cosmic radiation – Alien gamma rays Image showing ~220µs clock drift by Luke Bigum
  23. @tomerg Distributed time Client A Client C Client B Client

    D RTC RTC RTC RTC Event Stream Event Store RTC event_time = ?
  24. @tomerg Time source: Application Client A Client C Client B

    Client D RTC RTC RTC RTC Event Stream Event Store RTC 1 2 3 4
  25. @tomerg Time source: Application Client A Client C Client B

    Client D RTC RTC RTC RTC Event Stream Event Store RTC 1 2 3 4 Clocks must be synchronized!
  26. @tomerg What about NTP? Client A Client C Client B

    Client D RTC RTC RTC RTC Event Store RTC NTP
  27. @tomerg What about NTP? Client A Client C Client B

    Client D RTC RTC RTC RTC Event Store RTC NTP Accurate within ~10ms✝ ✝ Fig. 1, “Characterizing Quality of Time and Topology in a Time Synchronization Network”, Murta et al
  28. @tomerg Time source: Database Client A Client C Client B

    Client D RTC RTC RTC RTC Event Stream Event Store RTC Doesn’t scale! Doesn’t work! Such wow!!!
  29. @tomerg DEALING WITH TIME IS HARD TL;DR

  30. @tomerg Reprise: Time (noun) “… the system of those sequential

    relations that any event has to any other, as past, present, or future; indefinite and continuous duration regarded as that in which events succeed one another.” -- dictionary.com
  31. @tomerg Reprise: Time (noun) “… the system of those sequential

    relations that any event has to any other, as past, present, or future; indefinite and continuous duration regarded as that in which events succeed one another.” -- dictionary.com
  32. @tomerg Reframing the problem • When reading events – Do

    we need wall time? – We don’t care about it – It’s just metadata • We only care about ordering events Created (2017-05-03 10:15) Updated (2017-05-03 10:27) Archived (2017-05-03 10:55)
  33. @tomerg Causality • Take any two events • What relationship

    can they have? Created Updated Updated Archived
  34. @tomerg Causality • Take any two events • What relationship

    can they have? – Happens before: A→B d B↛A A. Created B. Updated Updated Archived
  35. @tomerg Causality • Take any two events • What relationship

    can they have? – Happens before: A→B – Concurrent: A↛B and B↛A Created A. Updated B. Updated Archived
  36. @tomerg What do we need? • Our versioning scheme should…

    – Respect causality – Be totally ordered – Not rely on the RTC • We want serializability – ... on an ACID-compliant database!
  37. @tomerg MAKE ADI GREAT AGAIN! Enough theory. Let’s

  38. @tomerg Relax • It’s fairly simple in practice • First,

    the schema Column Type Key Null site_id binary(16) ✔ ✖ event_time timestamp ✔ ✖ event_type enum(…) ✖ ✖ payload blob ✖ ✖
  39. @tomerg Relax • It’s fairly simple in practice • First,

    the schema – Don’t key on timestamp Column Type Key Null site_id binary(16) ✔ ✖ event_time timestamp ✖ ✖ event_type enum(…) ✖ ✖ payload blob ✖ ✖
  40. @tomerg Relax • It’s fairly simple in practice • First,

    the schema – Don’t key on timestamp – Add explicit version Column Type Key Null site_id binary(16) ✔ ✖ version int ✔ ✖ event_time timestamp ✖ ✖ event_type enum(…) ✖ ✖ payload blob ✖ ✖
  41. @tomerg Finishing touches • On writes: MySQL Server Client C1-Cn

  42. @tomerg Finishing touches • On writes: – Read latest version

    V0 MySQL Server Client C1-Cn V0
  43. @tomerg Finishing touches • On writes: – Read latest version

    V0 – Generate events V1...Vn – Write atomically MySQL Server Client C1-Cn V1-Vn
  44. @tomerg Finishing touches • On writes: – Read latest version

    V0 – Generate events V1...Vn – Write atomically • Success! – Return new version MySQL Server Client C1-Cn V1-Vn Vn
  45. @tomerg Finishing touches • On writes: – Read latest version

    V0 – Generate events V1...Vn – Write atomically • Duplicate primary key? – Optimistic lock failure! – Retry, propagate, resolve MySQL Server Client C1-Cn V1-Vn Error PK violation
  46. @tomerg GREAT SUCCESS!

  47. @tomerg KFIR ‘ADI’ BLOCH A round of applause for our

    special guest: ... and for the terrific camera work I butchered: VAIDAS PILKAUSKAS
  48. @tomerg QUESTIONS? Thank you for listening tomer@tomergabel.com @tomerg On GitHub:

    https://github.com/holograph/examples/tree/master/event-sourcing This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
  49. Congratulations! You have found the secret level

  50. @tomerg Lamport timestamps✝ • Provides partial ordering of events •

    Advantages: – Respects causality – Low overhead – Simple to implement Event Store Host A Host C Host B ✝ “Time, Clocks, and the Ordering of Events in a Distributed System”, Leslie Lamport, 1978
  51. @tomerg Event Store Host A Host C Host B Lamport

    timestamps • Each host maintains local logical clock • Start with T=0 T=0 T=0 T=0
  52. @tomerg Event Store Host A Host C Host B Lamport

    timestamps • Each host maintains local logical clock • Start with T=0 • On send (i.e. write): – Increment local clock – Attach to message T=0 T=0 T=0 T=1
  53. @tomerg Lamport timestamps • On receive (i.e. read): – Process

    event if T < Tin Event Store Host A Host C Host B T=0 Tin =1
  54. @tomerg Lamport timestamps • On receive (i.e. read): – Process

    event if T < Tin – Update clock past the latest event: T = max(T, Tin ) + 1 • All done! Event Store Host A Host C Host B T=0 Tin =1 T=2
  55. @tomerg … well, almost • This is a partial ordering

    • Causality is dealt with – A→B ⇒ T(A) < T(B) • What about concurrency? – A↛B and B↛A ⇒ ?
  56. @tomerg Final touches • We want total ordering – Must

    be stable – Need not correspond to real time Created Updated Updated Archived T=1 T=5 T=2 T=2
  57. @tomerg Final touches • We want total ordering – Must

    be stable – Need not correspond to real time • Breaking the tie: – Use any consistent key – Host name, MAC, IP… Created Updated Updated Archived T=1 T=5 T=2 T=2