DDDVienna - Applying event sourcing and CQRS in a large ERP system

DDDVienna - Applying event sourcing and CQRS in a large ERP system

In the last 7 years we have applied event sourcing and CQRS in the development of our new ERP system. During this process we have run into some interesting challenges and we have come up with solutions. In this session I would like to present these patterns and get your feedback on them.

03dc1d993fa7d5de422c1f35d53f80e6?s=128

Michiel Overeem

October 04, 2019
Tweet

Transcript

  1. Applying event sourcing and CQRS in a large ERP system

    Our work at AFAS Software
  2. 400+ employees (4 locations) 10.500 customers (companies) AFAS Software

  3. HRM, CRM, finance, order management, project management, workflow, ... AFAS

    Profit
  4. 1996 – AFAS starts 2011 (jan) – First steps in

    CQRS and event sourcing 2011 (dec) – Started at AFAS 2020 – Live Via JTeam/Trifork/Axon. 2016 – Started on evolution
  5. 20 years of ERP 10.000 customers 90% cloud Web Scalable

    CQRS Event sourcing
  6. 20 years of ERP 10.000 customers 90% cloud Person Entity

    Customer Role Order Agreement party Spending limit Address Organisation Entity Delivery BusinessAc tivity party Invoice BusinessAc tivity party Address Address Spending limit Spending limit Own Organisation Workarea workarea Spending limit Put all our knowledge and experience in a model
  7. Customized Web Scalable CQRS Event sourcing Person Entity Customer Role

    Order Agreement party Spending limit Address Organisation Entity Delivery BusinessAc tivity party Invoice BusinessAc tivity party Address Address Spending limit Spending limit Own Organisation Workarea workarea Spending limit
  8. (2.830 tables with 126.705 columns) ERP software is inherently relational

  9. Our basic CQRS/Event Sourcing solution

  10. Aggregate root Command Event store Event Projector Query store Event

    Queue Query request Query handler
  11. Event store Aggregate root Command Event store Event Projector Query

    store Event Queue Query request Query handler Aggregate root Command Projector Event Queue Query store
  12. Event store Aggregate root Command Event store Event Projector Query

    store Event Queue Query request Query handler Aggregate root Command Projector Event Queue Query store Data of different projectors can not be joined. Tables are fully inlined, no joins.
  13. Aggregate root Command Event store Event Projector Query store Event

    Queue Query request Query handler • Aimed at CRUD • MongoDB, *SQL, DocumentDB • MS Orleans, MS Service Fabric • SPA-frontend
  14. How about evolution? Part 1

  15. Evolution as a feature of the model? Person Entity Customer

    Role Order Agreement party Spending limit Address Organisation Entity Delivery BusinessAc tivity party Invoice BusinessAc tivity party Address Address Spending limit Spending limit Own Organisation Workarea workarea Spending limit
  16. Data upgrade Deployed with Executed by Executed by Data upgrade

    Application upgrade Lazy transformation Upcasting In place transfor- mation Multiple versions Copy and transfor- mation Basic & Complex Event Basic & Complex Stream Basic Store Complex Store Big Flip Rolling Upgrade Blue-Green Expand- Contract Blue- Green Deployed with Combined with Weak schema Expensive!
  17. Event Publishing part 2

  18. Aggregate root Command Event store Event Projector Query store Event

    Queue Query request Query handler
  19. Aggregate root Command Event store Event Projector Query store Event

    Queue Query request Query handler Can give error (needs retry). Command-side in control over publishing.
  20. Aggregate root Command Event store Event Projector Query store Query

    request Query handler
  21. Aggregate root Command Event store Event Projector Query store Query

    request Query handler Projector in control. More load on the event store.
  22. When CRUD is not enough business processes

  23. Aggregate root Command Event store Event Projector Query store Query

    request Query handler
  24. Aggregate root Command Event store Event Projector Query store Query

    request Query handler Process manager
  25. Aggregate root Command Event store Event Projector Query store Query

    request Query handler Process manager But what if these commands fail?
  26. Aggregate root Command Event store Event Projector Query store Query

    request Query handler Process manager Event store
  27. Aggregate root Command Event store Event Projector Query store Query

    request Query handler Process manager Event store Commands that cannot fail become events. We still have some internal commands.
  28. Aggregate root Command Event store Event Projector Query store Query

    request Query handler Process manager Event store Different instances to scale the work. Event sourced for state.
  29. Aggregate root / Process manager Command Event store Event Projector

    Query store Query request Query handler Is this what we want?
  30. Aggregate root / Process manager Command Event store Event Projector

    Query store Query request Query handler Is this what we want? Technically there is no difference between aggregates and processmanagers
  31. The problem of relational data

  32. Aggregate root Command Event store Event Projector Query store Query

    request Query handler Process manager Event store
  33. Aggregate root Command Event store Event Projector Query store Query

    request Query handler Process manager Event store Process manager Process manager Process manager Aggregate root Aggregate root Aggregate root Aggregate root
  34. Aggregate root Command Event store Event Projector Query store Query

    request Query handler Process manager Event store Process manager Process manager Process manager Aggregate root Aggregate root Aggregate root Aggregate root These ProcessManager all need the same data (lots of data)
  35. Aggregate root Command Event store Event Projector Query store Query

    request Query handler Process manager Event store Stream Projector Query store Event store
  36. Aggregate root Command Event store Event Projector Query store Query

    request Query handler Process manager Event store Stream Projector Query store Event store StreamProjectors are single-instanced, non- event-sourced processmanagers
  37. Eventual consistency communication between client and server

  38. Aggregate root Command Event store Event Projector Query store Query

    request Query handler Client
  39. Counter Process Manager Count Command Counted Event Event store Counter

    Projector Query store Get Counter Count Query handler Getting the next consecutive number in a series.
  40. Counter Process Manager Count Command Counted Event Event store Counter

    Projector Query store Get Counter Count Query handler When is the value ready? Correlationid?
  41. Aggregate root Command Event store Event Projector Query store Query

    request Query handler Client Using websockets to notify the Client
  42. Aggregate root Command Event store Event Projector Query store Query

    request Query handler Client Returning values from the produced event directly
  43. Counter Process Manager Count Command Counted Event Event store Counter

    Projector Query store Get Counter Count Query handler
  44. How large is our system anyway?

  45. Aggregate root Command Event store Event Projector Query store Query

    request Query handler Stream Projector Query store Event store 557 464 1927 4849 1075 2718 801 1479
  46. Store (19) S M L Traffic (19) S M L

    Schema (14) S M L
  47. Dealing with bounded contexts

  48. None
  49. Aggregate root Command Event store Event Aggregate root Event store

    Event
  50. Aggregate root Command Event store Event Aggregate root Event store

    Event Team B Team A
  51. Aggregate root Command Event store Event Aggregate root Event store

    Event These events now serve multiple goals. Team B Team A
  52. Aggregate root Command Event store Event Aggregate root Event store

    Event Team B Team A Sync Event store Event
  53. Aggregate root Command Event store Event Aggregate root Event store

    Event Team B Team A Sync Event store Event Introduce synchronization events.
  54. How about evolution? Part 2

  55. Versioned Events Weak Schema Upcasters In-Place Copy- Transform

  56. We will go live in January… No guarantees are given

  57. Copyright Nasa Goddard Michiel Overeem @michielovereem • No event bus

    • Commands do have results • Projectors can publish events • Aggregates can listen to events • You can do what you want ☺ i.e. you do not have to listen to a consultant that does not know your context.