How shit works: Time - Speaker Deck

Slide 1

Slide 1 text

How shit works: Time Tomer Gabel, WeWork Prague, 18-19 October 2018

Slide 2

Slide 2 text

@tomerg THE STORY OF ADI

Slide 3

Slide 3 text

@tomerg Meet Adi.

Slide 4

Slide 4 text

@tomerg He has a dream.

Slide 5

Slide 5 text

@tomerg Adi is fashionable • Event sourcing is the future, they say • Event sourcing is the shit, he agrees

Slide 6

Slide 6 text

@tomerg Adi is fashionable • Event sourcing is the future, they say • Event sourcing is the shit, he agrees • … and designs an event model Created Modified Published Archived Restored

Slide 7

Slide 7 text

@tomerg Column Type Key Null site_id binary(16) ✔ ✖ event_time timestamp ✔ ✖ event_type enum(…) ✖ ✖ payload blob ✖ ✖ Adi is fashionable

Slide 8

Slide 8 text

@tomerg HEY, KIDS Can you guess what happens next?

Slide 9

Slide 9 text

@tomerg Adi is confused • Bad shit happens • Event streams exhibit:

Slide 10

Slide 10 text

@tomerg Adi is confused • Bad shit happens • Event streams exhibit: – Out of order events – Conflicting events – Impossible states Created Archived Updated WTF?!

Slide 11

Slide 11 text

How shit works: Time Tomer Gabel, WeWork @ GeeCON Prague 2018 Image: Vera Kratochvil (public domain)

Slide 12

Slide 12 text

@tomerg Time (noun) “… the system of those sequential relations that any event has to any other, as past, present, or future; indefinite and continuous duration regarded as that in which events succeed one another.” -- dictionary.com

Slide 13

Slide 13 text

@tomerg Modeling time • Encoding – Resolution – Epoch “Real” time Epoch Resolution T1 T 2 T3 T4 T 5

Slide 14

Slide 14 text

@tomerg Modeling time • Encoding – Resolution – Epoch T3 Instant (or “event”)

Slide 15

Slide 15 text

@tomerg Modeling time • Encoding – Resolution – Epoch T3 A B Conflicting events

Slide 16

Slide 16 text

@tomerg Modeling time • Encoding – Resolution – Epoch • Bootstrapping Epoch ???

Slide 17

Slide 17 text

@tomerg Modeling time • Encoding – Resolution – Epoch • Bootstrapping – Manual Image courtesy of Jim Leonard

Slide 18

Slide 18 text

@tomerg Modeling time • Encoding – Resolution – Epoch • Bootstrapping – Manual – Battery-backed Image: pc-restorer.com

Slide 19

Slide 19 text

@tomerg Modeling time • Encoding – Resolution – Epoch • Bootstrapping – Manual – Battery-backed – NTP Image: FDA graphic by Michael J. Ermarth (public domain)

Slide 20

Slide 20 text

@tomerg Modeling time • Encoding – Resolution – Epoch • Bootstrapping – Manual – Battery-backed – NTP • Updating T = T+1 T1 T 2 W hen?

Slide 21

Slide 21 text

@tomerg System Clock (RTC) Modeling time • Encoding – Resolution – Epoch • Bootstrapping – Manual – Battery-backed – NTP • Updating Image: Jeremy Saglimbeni on Vimeo (CC BY-SA 3.0)

Slide 22

Slide 22 text

@tomerg RTC isn’t perfect • Alas, clocks drift • Subtle causes – Temperature – Power supply – General relativity – Cosmic radiation – Alien gamma rays Image showing ~220µs clock drift by Luke Bigum

Slide 23

Slide 23 text

@tomerg Distributed time Client A Client C Client B Client D RTC RTC RTC RTC Event Stream Event Store RTC event_time = ?

Slide 24

Slide 24 text

@tomerg Time source: Application Client A Client C Client B Client D RTC RTC RTC RTC Event Stream Event Store RTC 1 2 3 4

Slide 25

Slide 25 text

@tomerg Time source: Application Client A Client C Client B Client D RTC RTC RTC RTC Event Stream Event Store RTC 1 2 3 4 Clocks must be synchronized!

Slide 26

Slide 26 text

@tomerg What about NTP? Client A Client C Client B Client D RTC RTC RTC RTC Event Store RTC NTP

Slide 27

Slide 27 text

@tomerg What about NTP? Client A Client C Client B Client D RTC RTC RTC RTC Event Store RTC NTP Accurate within ~10ms✝ ✝ Fig. 1, “Characterizing Quality of Time and Topology in a Time Synchronization Network”, Murta et al

Slide 28

Slide 28 text

@tomerg Time source: Database Client A Client C Client B Client D RTC RTC RTC RTC Event Stream Event Store RTC Doesn’t scale! Doesn’t work! Such wow!!!

Slide 29

Slide 29 text

@tomerg DEALING WITH TIME IS HARD TL;DR

Slide 30

Slide 30 text

@tomerg Reprise: Time (noun) “… the system of those sequential relations that any event has to any other, as past, present, or future; indefinite and continuous duration regarded as that in which events succeed one another.” -- dictionary.com

Slide 31

Slide 31 text

@tomerg Reprise: Time (noun) “… the system of those sequential relations that any event has to any other, as past, present, or future; indeﬁnite and continuous duration regarded as that in which events succeed one another.” -- dictionary.com

Slide 32

Slide 32 text

@tomerg Reframing the problem • When reading events – Do we need wall time? – We don’t care about it – It’s just metadata • We only care about ordering events Created (2017-05-03 10:15) Updated (2017-05-03 10:27) Archived (2017-05-03 10:55)

Slide 33

Slide 33 text

@tomerg Causality • Take any two events • What relationship can they have? Created Updated Updated Archived

Slide 34

Slide 34 text

@tomerg Causality • Take any two events • What relationship can they have? – Happens before: A→B d B↛A A. Created B. Updated Updated Archived

Slide 35

Slide 35 text

@tomerg Causality • Take any two events • What relationship can they have? – Happens before: A→B – Concurrent: A↛B and B↛A Created A. Updated B. Updated Archived

Slide 36

Slide 36 text

@tomerg What do we need? • Our versioning scheme should… – Respect causality – Be totally ordered – Not rely on the RTC • We want serializability – ... on an ACID-compliant database!

Slide 37

Slide 37 text

@tomerg MAKE ADI GREAT AGAIN! Enough theory. Let’s

Slide 38

Slide 38 text

@tomerg Relax • It’s fairly simple in practice • First, the schema Column Type Key Null site_id binary(16) ✔ ✖ event_time timestamp ✔ ✖ event_type enum(…) ✖ ✖ payload blob ✖ ✖

Slide 39

Slide 39 text

@tomerg Relax • It’s fairly simple in practice • First, the schema – Don’t key on timestamp Column Type Key Null site_id binary(16) ✔ ✖ event_time timestamp ✖ ✖ event_type enum(…) ✖ ✖ payload blob ✖ ✖

Slide 40

Slide 40 text

@tomerg Relax • It’s fairly simple in practice • First, the schema – Don’t key on timestamp – Add explicit version Column Type Key Null site_id binary(16) ✔ ✖ version int ✔ ✖ event_time timestamp ✖ ✖ event_type enum(…) ✖ ✖ payload blob ✖ ✖

Slide 41

Slide 41 text

@tomerg Finishing touches • On writes: MySQL Server Client C1-Cn

Slide 42

Slide 42 text

@tomerg Finishing touches • On writes: – Read latest version V0 MySQL Server Client C1-Cn V0

Slide 43

Slide 43 text

@tomerg Finishing touches • On writes: – Read latest version V0 – Generate events V1...Vn – Write atomically MySQL Server Client C1-Cn V1-Vn

Slide 44

Slide 44 text

@tomerg Finishing touches • On writes: – Read latest version V0 – Generate events V1...Vn – Write atomically • Success! – Return new version MySQL Server Client C1-Cn V1-Vn Vn

Slide 45

Slide 45 text

@tomerg Finishing touches • On writes: – Read latest version V0 – Generate events V1...Vn – Write atomically • Duplicate primary key? – Optimistic lock failure! – Retry, propagate, resolve MySQL Server Client C1-Cn V1-Vn Error PK violation

Slide 46

Slide 46 text

@tomerg GREAT SUCCESS!

Slide 47

Slide 47 text

@tomerg KFIR ‘ADI’ BLOCH A round of applause for our special guest: ... and for the terrific camera work I butchered: VAIDAS PILKAUSKAS

Slide 48

Slide 48 text

@tomerg QUESTIONS? Thank you for listening [email protected] @tomerg On GitHub: https://github.com/holograph/examples/tree/master/event-sourcing This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Slide 49

Slide 49 text

Congratulations! You have found the secret level

Slide 50

Slide 50 text

@tomerg Lamport timestamps✝ • Provides partial ordering of events • Advantages: – Respects causality – Low overhead – Simple to implement Event Store Host A Host C Host B ✝ “Time, Clocks, and the Ordering of Events in a Distributed System”, Leslie Lamport, 1978

Slide 51

Slide 51 text

@tomerg Event Store Host A Host C Host B Lamport timestamps • Each host maintains local logical clock • Start with T=0 T=0 T=0 T=0

Slide 52

Slide 52 text

@tomerg Event Store Host A Host C Host B Lamport timestamps • Each host maintains local logical clock • Start with T=0 • On send (i.e. write): – Increment local clock – Attach to message T=0 T=0 T=0 T=1

Slide 53

Slide 53 text

@tomerg Lamport timestamps • On receive (i.e. read): – Process event if T < Tin Event Store Host A Host C Host B T=0 Tin =1

Slide 54

Slide 54 text

@tomerg Lamport timestamps • On receive (i.e. read): – Process event if T < Tin – Update clock past the latest event: T = max(T, Tin ) + 1 • All done! Event Store Host A Host C Host B T=0 Tin =1 T=2

Slide 55

Slide 55 text

@tomerg … well, almost • This is a partial ordering • Causality is dealt with – A→B ⇒ T(A) < T(B) • What about concurrency? – A↛B and B↛A ⇒ ?

Slide 56

Slide 56 text

@tomerg Final touches • We want total ordering – Must be stable – Need not correspond to real time Created Updated Updated Archived T=1 T=5 T=2 T=2

Slide 57

Slide 57 text

@tomerg Final touches • We want total ordering – Must be stable – Need not correspond to real time • Breaking the tie: – Use any consistent key – Host name, MAC, IP… Created Updated Updated Archived T=1 T=5 T=2 T=2