How shit works: Time

A talk given at GeeCON 2018 in Prague, Czech Republic:

In this talk we'll take a hard look at one of the most commonly used, and at least as commonly misunderstood, elements in software engineering: time. Time is so fundamental to the way humans experience reality that we don't normally give it a second thought, but it's just as fundamental to software systems. Without a correct model for working with time BAD THINGS HAPPEN: data is persisted out of order, exceptions occur where they shouldn't be possible, and production systems blow up.

We'll cover the various common representations of time, acknowledge their caveats and deficiencies, and hopefully learn a few new tools and practices along the way.

Transcript

1. How shit works: Time Tomer Gabel, WeWork Prague, 18-19 October 2018
2. @tomerg THE STORY OF ADI
3. @tomerg Meet Adi.
4. @tomerg He has a dream.
5. @tomerg Adi is fashionable • Event sourcing is the future, they say • Event sourcing is the shit, he agrees
6. @tomerg Adi is fashionable • Event sourcing is the future, they say • Event sourcing is the shit, he agrees • … and designs an event model Created Modified Published Archived Restored
7. @tomerg Column Type Key Null site_id binary(16) ✔ ✖ event_time timestamp ✔ ✖ event_type enum(…) ✖ ✖ payload blob ✖ ✖ Adi is fashionable
8. @tomerg HEY, KIDS Can you guess what happens next?
9. @tomerg Adi is confused • Bad shit happens • Event streams exhibit:
10. @tomerg Adi is confused • Bad shit happens • Event streams exhibit: – Out of order events – Conflicting events – Impossible states Created Archived Updated WTF?!
11. How shit works: Time Tomer Gabel, WeWork @ GeeCON Prague 2018 Image: Vera Kratochvil (public domain)
12. @tomerg Time (noun) “… the system of those sequential relations that any event has to any other, as past, present, or future; indefinite and continuous duration regarded as that in which events succeed one another.” -- dictionary.com
13. @tomerg Modeling time • Encoding – Resolution – Epoch “Real” time Epoch Resolution T1 T 2 T3 T4 T 5
14. @tomerg Modeling time • Encoding – Resolution – Epoch T3 Instant (or “event”)
15. @tomerg Modeling time • Encoding – Resolution – Epoch T3 A B Conflicting events
16. @tomerg Modeling time • Encoding – Resolution – Epoch • Bootstrapping Epoch ???
17. @tomerg Modeling time • Encoding – Resolution – Epoch • Bootstrapping – Manual Image courtesy of Jim Leonard
18. @tomerg Modeling time • Encoding – Resolution – Epoch • Bootstrapping – Manual – Battery-backed Image: pc-restorer.com
19. @tomerg Modeling time • Encoding – Resolution – Epoch • Bootstrapping – Manual – Battery-backed – NTP Image: FDA graphic by Michael J. Ermarth (public domain)
20. @tomerg Modeling time • Encoding – Resolution – Epoch • Bootstrapping – Manual – Battery-backed – NTP • Updating T = T+1 T1 T 2 W hen?
21. @tomerg System Clock (RTC) Modeling time • Encoding – Resolution – Epoch • Bootstrapping – Manual – Battery-backed – NTP • Updating Image: Jeremy Saglimbeni on Vimeo (CC BY-SA 3.0)
22. @tomerg RTC isn’t perfect • Alas, clocks drift • Subtle causes – Temperature – Power supply – General relativity – Cosmic radiation – Alien gamma rays Image showing ~220µs clock drift by Luke Bigum
23. @tomerg Distributed time Client A Client C Client B Client D RTC RTC RTC RTC Event Stream Event Store RTC event_time = ?
24. @tomerg Time source: Application Client A Client C Client B Client D RTC RTC RTC RTC Event Stream Event Store RTC 1 2 3 4
25. @tomerg Time source: Application Client A Client C Client B Client D RTC RTC RTC RTC Event Stream Event Store RTC 1 2 3 4 Clocks must be synchronized!
26. @tomerg What about NTP? Client A Client C Client B Client D RTC RTC RTC RTC Event Store RTC NTP
27. @tomerg What about NTP? Client A Client C Client B Client D RTC RTC RTC RTC Event Store RTC NTP Accurate within ~10ms✝ ✝ Fig. 1, “Characterizing Quality of Time and Topology in a Time Synchronization Network”, Murta et al
28. @tomerg Time source: Database Client A Client C Client B Client D RTC RTC RTC RTC Event Stream Event Store RTC Doesn’t scale! Doesn’t work! Such wow!!!
29. @tomerg DEALING WITH TIME IS HARD TL;DR
30. @tomerg Reprise: Time (noun) “… the system of those sequential relations that any event has to any other, as past, present, or future; indefinite and continuous duration regarded as that in which events succeed one another.” -- dictionary.com
31. @tomerg Reprise: Time (noun) “… the system of those sequential relations that any event has to any other, as past, present, or future; indefinite and continuous duration regarded as that in which events succeed one another.” -- dictionary.com
32. @tomerg Reframing the problem • When reading events – Do we need wall time? – We don’t care about it – It’s just metadata • We only care about ordering events Created (2017-05-03 10:15) Updated (2017-05-03 10:27) Archived (2017-05-03 10:55)
33. @tomerg Causality • Take any two events • What relationship can they have? Created Updated Updated Archived
34. @tomerg Causality • Take any two events • What relationship can they have? – Happens before: A→B d B↛A A. Created B. Updated Updated Archived
35. @tomerg Causality • Take any two events • What relationship can they have? – Happens before: A→B – Concurrent: A↛B and B↛A Created A. Updated B. Updated Archived
36. @tomerg What do we need? • Our versioning scheme should… – Respect causality – Be totally ordered – Not rely on the RTC • We want serializability – ... on an ACID-compliant database!
37. @tomerg MAKE ADI GREAT AGAIN! Enough theory. Let’s
38. @tomerg Relax • It’s fairly simple in practice • First, the schema Column Type Key Null site_id binary(16) ✔ ✖ event_time timestamp ✔ ✖ event_type enum(…) ✖ ✖ payload blob ✖ ✖
39. @tomerg Relax • It’s fairly simple in practice • First, the schema – Don’t key on timestamp Column Type Key Null site_id binary(16) ✔ ✖ event_time timestamp ✖ ✖ event_type enum(…) ✖ ✖ payload blob ✖ ✖
40. @tomerg Relax • It’s fairly simple in practice • First, the schema – Don’t key on timestamp – Add explicit version Column Type Key Null site_id binary(16) ✔ ✖ version int ✔ ✖ event_time timestamp ✖ ✖ event_type enum(…) ✖ ✖ payload blob ✖ ✖
41. @tomerg Finishing touches • On writes: MySQL Server Client C1-Cn
42. @tomerg Finishing touches • On writes: – Read latest version V0 MySQL Server Client C1-Cn V0
43. @tomerg Finishing touches • On writes: – Read latest version V0 – Generate events V1...Vn – Write atomically MySQL Server Client C1-Cn V1-Vn
44. @tomerg Finishing touches • On writes: – Read latest version V0 – Generate events V1...Vn – Write atomically • Success! – Return new version MySQL Server Client C1-Cn V1-Vn Vn
45. @tomerg Finishing touches • On writes: – Read latest version V0 – Generate events V1...Vn – Write atomically • Duplicate primary key? – Optimistic lock failure! – Retry, propagate, resolve MySQL Server Client C1-Cn V1-Vn Error PK violation
46. @tomerg GREAT SUCCESS!
47. @tomerg KFIR ‘ADI’ BLOCH A round of applause for our special guest: ... and for the terrific camera work I butchered: VAIDAS PILKAUSKAS
48. @tomerg QUESTIONS? Thank you for listening tomer@tomergabel.com @tomerg On GitHub: https://github.com/holograph/examples/tree/master/event-sourcing This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
49. Congratulations! You have found the secret level
50. @tomerg Lamport timestamps✝ • Provides partial ordering of events • Advantages: – Respects causality – Low overhead – Simple to implement Event Store Host A Host C Host B ✝ “Time, Clocks, and the Ordering of Events in a Distributed System”, Leslie Lamport, 1978
51. @tomerg Event Store Host A Host C Host B Lamport timestamps • Each host maintains local logical clock • Start with T=0 T=0 T=0 T=0
52. @tomerg Event Store Host A Host C Host B Lamport timestamps • Each host maintains local logical clock • Start with T=0 • On send (i.e. write): – Increment local clock – Attach to message T=0 T=0 T=0 T=1
53. @tomerg Lamport timestamps • On receive (i.e. read): – Process event if T < Tin Event Store Host A Host C Host B T=0 Tin =1
54. @tomerg Lamport timestamps • On receive (i.e. read): – Process event if T < Tin – Update clock past the latest event: T = max(T, Tin ) + 1 • All done! Event Store Host A Host C Host B T=0 Tin =1 T=2
55. @tomerg … well, almost • This is a partial ordering • Causality is dealt with – A→B ⇒ T(A) < T(B) • What about concurrency? – A↛B and B↛A ⇒ ?
56. @tomerg Final touches • We want total ordering – Must be stable – Need not correspond to real time Created Updated Updated Archived T=1 T=5 T=2 T=2
57. @tomerg Final touches • We want total ordering – Must be stable – Need not correspond to real time • Breaking the tie: – Use any consistent key – Host name, MAC, IP… Created Updated Updated Archived T=1 T=5 T=2 T=2