Mistakes we made adopting event sourcing (and how we recovered)

Over the last few years we have been building new systems that have event-sourced architectures. Event-sourcing is a good fit for our needs because the organisation wants to preserve an accurate history of information managed by the system and analyse it for (among other things) fraud detection. When we started, however, none of us had built a system with an event-sourced architecture before. Despite reading plenty of advice on what to do and what to avoid, and experience reports from other projects, we made some significant mistakes in our design. This talk describes where we went wrong, in the hope that others can learn from our failures.

But it’s not all bad news. We were able to recover from our mistakes with an ease that surprised us. I’ll also describe the factors that allowed us to easily change our architecture, in the hope that others can learn from our successes too.

Presented at DDDEU 2020.

Transcript

1. Mistakes we made adopting event sourcing (and how we recovered) Nat Pryce natpryce.com @natpryce
2. We were building an editorial system for scientific publishing Submit Review Accept Typeset Publish Read Research Transfer Reject Revise Research Integrity Analysis
3. Containment & facing metaphors: core & peripheral applications Research Integrity Analysis
4. Technical underpinnings: boring technologies http://boringtechnology.club/
5. Mistakes we made
6. Mistakes we made • Seduced by eventual consistency Application Event History Service Event History Store Application HTTP
7. Application Current State Store Mistakes we made • Seduced by eventual consistency • Command processors both stored events and current state Event History Service Event History Store
8. Event History Service Mistakes we made • Seduced by eventual consistency • Command processors both storied events and current state • Confusion between event-driven & event-sourced architecture Application Event History Store Application
9. Mistakes we made • Seduced by eventual consistency • Command processors both stored events and current state • Confusion between event-driven & event-sourced architecture • Used the event store as a message bus Event History Service Application Event History Store Application
10. How we recognised our mistakes
11. Perhaps I was a little machiavellian I wanted us to build an intuition for the advantages, disadvantages and trade-offs inherent in an event-sourced architecture, rather than apply patterns cookie-cutter style.
12. What we decided to do
13. Change our terminology The word “event” is such an overused term! We had many discussions about how to name different kinds of event: • those that are part of the historical record • those that are emitted by our active application monitoring • those that are notifications that should trigger activity • etc. Our latest attempt: • "historical fact" not "event" • "historical record" not "event store"
14. Use REST & HTTP for integration, not events HATEOAS means that peripheral applications are context independent • We can link our legacy systems to them • We can invoke them from test tools, even in the live environment Links and HTTP content type negotiation to manage service evolution • Decouple release cadences across the organisation
15. Make command handlers transactional The application components connect directly to the event store database We kept the HTTP service, but only for reading events It turned out to be very useful for hack-days!
16. Treat snapshot of entity state as a read-through cache 1. In a read transaction: 1. load the recent state of the entity into an in-memory model 2. In a write transaction: 1. load events that occurred to the entity since the recent projection into the in-memory model 2. perform business logic 3. record events resulting from executing the command 3. In a write transaction: 1. Save the in memory state if it was created from more recent events than the latest state in the database
17. Store current entity state as JSON, not relational tables Use existing code for: • Serialisation/deserialisation between JSON and Kotlin domain model • Migrating JSON structures in the database concurrently with live transactions Alternatively: discard JSON and rebuild from events if the format is out of date Postgres can index on properties of JSON objects • No need to denormalise JSON into columns to select entities
18. How we recovered
19. Why was that so easy?
20. We used the "Hexagonal" architecture HTTP service https://alistair.cockburn.us/hexagonal-architecture/
21. We had extensive automated tests val scenario = newScenario() @Test fun `an author cannot edit another author's submission`() { val alice = scenario.newAuthor(aliceDetails, aliceUserId, theJournal) val bob = scenario.newAuthor(bobDetails, bobUserId, theJournal) val alicesSubmission = alice.canCreateSubmission( journal = theJournal, manuscriptType = article) bob.cannotUpdate(alicesSubmission, SetAuthors(bob), because = NotAuthorised(bob.userId, theJournal, alicesSubmission, alice.userId)) }
22. We integrated and deployed continuously Everyone pushed straight to master: no branches or pull requests Every commit was automatically built, tested and promoted to live. Release decoupled from deployment, controlled by feature flags. Features could be turned on for a single session with cookies, or for all users. Client-driven compatibility tests automatically inserted into build pipelines of dependencies The tests have to have good coverage!
23. Our automated tests enforced the Hexagonal architecture Direct Actor In-memory storage Test Screenplay Pattern https://ideas.riverglide.com/page-objects-refactored-12ec3541990
24. The exact same test code ran against the HTTP services HTTP service HTTP Actor Test
25. … when deployed into a cloud environment Virtual Machine Virtual Machine Virtual Machine HTTP Actor Test Load balancer Firewall
26. … through the browser Service cluster Service cluster Front-end cluster Front-end cluster Service cluster Browsing Actor Test Front-end cluster Web browser
27. … against production Data centre Data centre Data centre Browsing Actor Test Web browser CDN
28. Lessons Learned
29. The distinction between commands and events is vital The confusion between "event sourced" and "event driven" seems very common. • Developers joining the project often go down the same rabbit hole we did. Drawing a clear distinction between "things that cause side effects" and "things that happened in the past" is more important than Command/Query separation. (Even in an event-driven architecture, event handlers would perform commands in response to events and – to use the terminology we adopted – record facts in the historical record about what happened.)
30. Event sourcing combines well with other architectural styles In our case: Event Sourcing: Domain & data model Hexagonal: Intra-process software architecture CQRS: Intra-application data flow REST: Integrate applications & organisations
31. Eschew technical details in functional tests By defining our tests in terms of the domain model only, they were able to support significant changes to the technical architecture. The Hexagonal architecture and Screenplay pattern combine well to achieve this. Hexagonal architecture https://alistair.cockburn.us/hexagonal-architecture/ Screenplay Pattern https://ideas.riverglide.com/page-objects-refactored-12ec3541990
32. Thank you Nat Pryce natpryce.com @natpryce