Event Sourcing: Don't want to miss a thing

An introduction into Event Sourcing

Transcript

1. Event Sourcing Don't want to miss a thing @gogogarrett
2. Technology is ephemeral Data is enduring
3. #React4Lyfe #RailsScales #TypeHype
4. Data has value
5. view = fn(data)
6. oops, we had bad data
7. Greetings
8. I'm Garrett @gogogarrett
9. I work at Netflix
10. This talk is an append-only list of sentences
11. Life is a stream of immutable events
12. We can not change history
13. We can not delete things
14. Event Sourcing non-lossy architectural pattern
15. ORMs1 store current state 1 Object-relational mapping
16. Current state
17. SuperDuperDb user_id state 3 CA
18. Prev state _ _ New South Wale, aka: Sydney
19. Prev state
20. Init state
21. SuperDuperDb user_id state 3 CA
22. when the state changes we change the database to represent new state
23. change
24. change update
25. update
26. update delete
27. every update is a delete
28. Data has value Why do we delete it?
29. What if we don't know if it will have value?
30. Why would we delete something with potential value?
31. If we don't have update or delete What is left?
32. insert & select
33. append only & immutable
34. Event Sourcing
35. Save the events that lead to current state
36. Events Cart [] ! [ ]
37. Events Cart [] ! [ ] ➕ # ! [ # ]
38. Events Cart [] ! [ ] ➕ # ! [ # ] ➕ # ! [ #, # ]
39. Events Cart [] ! [ ] ➕ # ! [ # ] ➕ # ! [ #, # ] ➕ $ ! [ #, #, $ ]
40. Events Cart [] ! [ ] ➕ # ! [ # ] ➕ # ! [ #, # ] ➕ $ ! [ #, #, $ ] ➖ # ! [ #, $ ]
41. Events Cart [] ! [ ] ➕ # ! [ # ] ➕ # ! [ #, # ] ➕ $ ! [ #, #, $ ] ➖ # ! [ #, $ ] ➕ & ! [ ]
42. Data becomes an append-only list of immutable events
43. State is derived from these immtuable events
44. New state can be derived from the same events
45. Events Total computers sold [] 0
46. Events Total computers sold [] 0 ➕ " 1
47. Events Total computers sold [] 0 ➕ " 1 ➕ " 2
48. Events Total computers sold [] 0 ➕ " 1 ➕ " 2 ➕ # 2
49. Events Total computers sold [] 0 ➕ " 1 ➕ " 2 ➕ # 2 ➖ " 1
50. Events Total computers sold [] 0 ➕ " 1 ➕ " 2 ➕ # 2 ➖ " 1 ➕ % 1
51. Real world example tracking progress
53. ORM approach
56. currentState: { currentMap: 1, currentPosition: 6, currentActivity: 2 } id student_id lesson_ position activity_ position 1 3202 6 2
58. currentState: { currentMap: 1, currentPosition: 6, currentActivity: 2 } id student_id lesson_ position activity_ position quiz_ position 1 3202 6 2 1
60. currentState: { currentMap: 1, currentPosition: 3, currentActivity: 2 } id student_ id lesson_ position activity_ position quiz_ position manual_ reset_ position 1 3202 6 2 1 3
61. Kid finishes lesson 3.. currentState: { currentMap: 1, currentPosition: 4, currentActivity: 1 } id student_ id lesson_p osition activity_ position quiz_ position manual _reset _position 1 3202 4 1 1 3
62. Reporting.. ☞ What's the highest lesson a kid has done? ☞ What's the last quiz a kid has taken? ☞ What's the average growth of lessons in a month?
63. id ... last _lesson _position absolute_ highest_ position last_ quiz_ position 1 ... 6 20 8
64. This is fine
65. Event Sourcing approach
66. currentState: { currentMap: 1, currentPosition: 6, currentActivity: 1 } id student_id event_type payload 1 3202 placeStudent { lesson: 6, activity: 1 }
67. currentState: { currentMap: 1, currentPosition: 6, currentActivity: 2 } id student_id event_type payload 1 3202 placeStudent { lesson: 6, activity: 1 } 1 3202 activityComple ted { lesson: 6, activity: 1 }
68. currentState: { currentMap: 1, currentPosition: 6, currentActivity: 3 } id student_id event_type payload 1 3202 placeStudent { lesson: 6, activity: 1 } 1 3202 activityComple ted { lesson: 6, activity: 1 } 1 3202 activityComple ted { lesson: 6, activity: 2 }
69. currentState: { currentMap: 1, currentPosition: 7, currentActivity: 1 } id student_id event_type payload 1 3202 placeStudent { lesson: 6, activity: 1 } 1 3202 activityComplete d { lesson: 6, activity: 1 } 1 3202 activityComplete d { lesson: 6, activity: 2 } 1 3202 lessonCompleted { lesson: 6 }
70. currentState: { currentMap: 1, currentPosition: 3, currentActivity: 1 } id student_id event_type payload 1 3202 placeStudent { lesson: 6, activity: 1 } 1 3202 activityCompleted { lesson: 6, activity: 1 } 1 3202 activityCompleted { lesson: 6, activity: 2 } 1 3202 lessonCompleted { lesson: 6 } 1 3202 resetProgress { lesson: 3, activity: 1 }
71. Airplane view
83. The pieces ☞ Clients issue commands ☞ Aggregates build and hold current state ☞ Command handlers validate incoming commands against current state ☞ Event handlers transform the command into event/s and trigger the aggregate to update their state
84. Obligatory code snippet app.post('/progress/commands', (req: Request, res: Response) => { const currentState = new ProgressAggregate(req.user); const newCommand = new ProgressCommand(req.body); if (isValid(currentState, newCommand)) { const newEvents = commandHandler(currentState, newCommand); const newState = eventHandler(currentState, newEvents); res.send(newState); } else { res.send({ error: 'invalid command' }); } });
85. wa-lah event sourcing
86. Bonus round
87. Top questions ☞ Isn't this slow? ☞ What happens when theres too many events? ☞ What about bad data?
88. Speed - Optimize aggregates ☞ on login build the aggregates in memory ☞ partition the events table by user id / aggregation type ☞ build rollups / snapshots
89. Snapshots ☞ Always keep the raw events table ☞ After calculating current state, store that aggregation in a snapshot table. ☞ Read from snapshot table and apply the new events to that to derive new current state. ☞ Do this nightly, weekly, after every event - w/e works for you
90. Bad data ☞ Make your command handlers more strict ☞ Create new events to undo oldE events ☞ [soft] delete them (if you really have to)
91. Technology is ephemeral Data is enduring
92. Thanks @gogogarrett