Dutch .NET Group Meetup - Building an event sourced system in .NET

In this talk I will show how we are building a large ERP system in .NET, using CQRS and event sourcing. We will talk about the good stuff (modularization, scalability), the lessons learned (eventual consistency) and the challenges ahead (upgrades).

Transcript

1. Building an event sourced system in .NET Michiel Overeem Lead Software Architect michiel.overeem@afas.nl @michielovereem
2. 400+ employees (4 locations) 10.500 customers (companies) AFAS Software
3. HRM, CRM, finance, order management, project management, workflow, ... AFAS Profit
4. 20 years of ERP 10.000 customers 90% cloud Web Scalable CQRS Event sourcing
5. 20 years of ERP 10.000 customers 90% cloud Person Entity Customer Role Order Agreement party Spending limit Address Organisation Entity Delivery BusinessAc tivity party Invoice BusinessAc tivity party Address Address Spending limit Spending limit Own Organisation Workarea workarea Spending limit Put all our knowledge and experience in a model
6. Customized Web Scalable CQRS Event sourcing Person Entity Customer Role Order Agreement party Spending limit Address Organisation Entity Delivery BusinessAc tivity party Invoice BusinessAc tivity party Address Address Spending limit Spending limit Own Organisation Workarea workarea Spending limit
7. (2.830 tables with 126.705 columns) ERP software is inherently relational
8. Command Query Responsibility Segregation Client Command system Query system Eventing system
9. Command Query Responsibility Segregation Client Command system Query system Eventing system “PlaceOrderCommand”: { “OrderId”: “0b81b458-7671-4e9b-844b-9934255f5406”, “CustomerName”: “Linus Torvalds”, “OrderLines”: [ … ] }
10. Command Query Responsibility Segregation Client Command system Query system Eventing system “GetOrders”: { “CustomerName”: “Linus Torvalds”, }
11. Command Query Responsibility Segregation Client Command system Query system Eventing system
12. DDD
13. Aggregates - independent, self-contained business objects that handle commands and raise events.
14. Aggregates
15. Aggregates “PlaceOrderCommand”: { “OrderId”: “0b81b458-7671-4e9b-844b-9934255f5406”, “CustomerName”: “Linus Torvalds”, “OrderLines”: [ … ] }
16. Aggregates “PlaceOrderCommand”: { “OrderId”: “0b81b458-7671-4e9b-844b-9934255f5406”, “CustomerName”: “Linus Torvalds”, “OrderLines”: [ … ] } “OrderPlacedEvent”: { “OrderId”: “0b81b458-7671-4e9b-844b-9934255f5406”, “CustomerName”: “Linus Torvalds”, “OrderLines”: [ … ] }
17. Aggregates
18. Aggregates
19. Aggregates - independent, self-contained business objects that handle commands and raise events. That sounds a lot like actors ☺
21. Events – Something that has happened.
22. Events Events that need to be processed in the query system.
23. Projectors – A process that builds a projection from events.
24. Projectors
25. Projectors Needs routing to find the correct store
27. How to get the events from the command system to the query system?
28. CQRS Client Command system Query system Eventing system Some kind of message bus?
29. Event push
31. T
32. 2 phase commit Independence Command system Query system Eventing system
33. Event push
34. Event pull But we need persistence. And we still have a 2 phase commit.
35. Use the emitted events as single source of truth Event sourcing
36. CQRS Client Command system Query system Eventing system What is the data model used in the command side?
37. CQRS Client Command system Query system Eventing system
38. Event Store StreamId Stream Revision PayloadItemName Payload StreamType Sequence Number CommitTime AccountId Person1 1 PersonCreatedEvent { … } Person 1 … … Person1 2 PersonUpdatedEvent { … } Person 2 … … Person2 1 PersonCreatedEvent { … } Person 3 … … Order1 1 OrderCreatedEvent { … } Order 1 … … Order2 1 OrderCreatedEvent { … } Order 2 … … Order3 1 OrderCreatedEvent { … } Order 3 … …
39. Aggregates loading Event Store StreamId Stream Revision PayloadItemName Payload StreamType Sequence Number CommitTime AccountId Person1 1 PersonCreatedEvent { … } Person 1 … … Person1 2 PersonUpdatedEvent { … } Person 2 … … Person2 1 PersonCreatedEvent { … } Person 3 … … Order1 1 OrderCreatedEvent { … } Order 1 … … Order2 1 OrderCreatedEvent { … } Order 2 … … Order3 1 OrderCreatedEvent { … } Order 3 … …
40. Event processing Event Store StreamId Stream Revision PayloadItemName Payload StreamType Sequence Number CommitTime AccountId Person1 1 PersonCreatedEvent { … } Person 1 … … Person1 2 PersonUpdatedEvent { … } Person 2 … … Person2 1 PersonCreatedEvent { … } Person 3 … … Order1 1 OrderCreatedEvent { … } Order 1 … … Order2 1 OrderCreatedEvent { … } Order 2 … … Order3 1 OrderCreatedEvent { … } Order 3 … …
41. Event Sourcing A B A E C D D F B B A C C G H D A B C A D A A A A A A B B C C 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 1 1 2 1 1 2 3 4 5 1 1 2 1 2
42. Event Sourcing A B A E C D D F B B A C C G H D A B C A D A A A A A A B B C C 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 1 1 2 1 1 2 3 4 5 1 1 2 1 2 Sequencenumbers for projectors
43. Event Sourcing A B A E C D D F B B A C C G H D A B C A D A A A A A A B B C C 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 1 1 2 1 1 2 3 4 5 1 1 2 1 2 Revisions for aggregate loading and concurrency control
44. Event sourcing This is both the store as the messaging system.
45. Eventual consistency
46. Eventual consistency
47. Eventual consistency This is an async process
48. Eventual consistency Client Command system Query system Eventing system
49. Notifications β α γ
50. Notifications β α γ
52. Projector optimization
53. Projection Information from the customer stream Information from the order stream
54. Projection A helper-table with customer info A table for the projection A customer update could mean that we need to update 1.000.000+ rows
55. Projection: joins A table with customer info A table for the projection A query should join the two tables.
56. Projection: duplication A table with customer info A table with order info A table with customer info A table with invoice info
57. Projection: combine A table with customer info A table with order info A table with invoice info
58. Projection: chaining
59. So what have we seen?
60. Summary
61. Summary β α γ
63. Building an event sourced system in .NET michiel.overeem@afas.nl @michielovereem