Upgrade to Pro — share decks privately, control downloads, hide ads and more …

DDDVienna - Applying event sourcing and CQRS in a large ERP system

DDDVienna - Applying event sourcing and CQRS in a large ERP system

In the last 7 years we have applied event sourcing and CQRS in the development of our new ERP system. During this process we have run into some interesting challenges and we have come up with solutions. In this session I would like to present these patterns and get your feedback on them.

Michiel Overeem

October 04, 2019
Tweet

More Decks by Michiel Overeem

Other Decks in Technology

Transcript

  1. Applying event sourcing and CQRS in a large ERP system
    Our work at AFAS Software

    View Slide

  2. 400+ employees (4 locations)
    10.500 customers (companies)
    AFAS Software

    View Slide

  3. HRM, CRM, finance, order
    management, project
    management, workflow, ...
    AFAS Profit

    View Slide

  4. 1996 – AFAS starts
    2011 (jan) – First steps in CQRS and event sourcing
    2011 (dec) – Started at AFAS
    2020 – Live
    Via JTeam/Trifork/Axon.
    2016 – Started on evolution

    View Slide

  5. 20 years of ERP
    10.000 customers
    90% cloud
    Web
    Scalable
    CQRS
    Event sourcing

    View Slide

  6. 20 years of ERP
    10.000 customers
    90% cloud
    Person
    Entity
    Customer
    Role
    Order
    Agreement
    party
    Spending limit
    Address
    Organisation
    Entity
    Delivery
    BusinessAc tivity
    party
    Invoice
    BusinessAc tivity
    party
    Address
    Address
    Spending limit
    Spending limit
    Own Organisation
    Workarea
    workarea
    Spending limit
    Put all our knowledge and
    experience in a model

    View Slide

  7. Customized
    Web
    Scalable
    CQRS
    Event sourcing
    Person
    Entity
    Customer
    Role
    Order
    Agreement
    party
    Spending limit
    Address
    Organisation
    Entity
    Delivery
    BusinessAc tivity
    party
    Invoice
    BusinessAc tivity
    party
    Address
    Address
    Spending limit
    Spending limit
    Own Organisation
    Workarea
    workarea
    Spending limit

    View Slide

  8. (2.830 tables with 126.705 columns)
    ERP software is inherently relational

    View Slide

  9. Our basic CQRS/Event Sourcing solution

    View Slide

  10. Aggregate
    root
    Command
    Event store
    Event
    Projector
    Query store
    Event Queue
    Query
    request
    Query
    handler

    View Slide

  11. Event store
    Aggregate
    root
    Command
    Event store
    Event
    Projector
    Query store
    Event Queue
    Query
    request
    Query
    handler
    Aggregate
    root
    Command Projector
    Event Queue
    Query store

    View Slide

  12. Event store
    Aggregate
    root
    Command
    Event store
    Event
    Projector
    Query store
    Event Queue
    Query
    request
    Query
    handler
    Aggregate
    root
    Command Projector
    Event Queue
    Query store
    Data of different projectors can not be joined.
    Tables are fully inlined, no joins.

    View Slide

  13. Aggregate
    root
    Command
    Event store
    Event
    Projector
    Query store
    Event Queue
    Query
    request
    Query
    handler
    • Aimed at CRUD
    • MongoDB, *SQL, DocumentDB
    • MS Orleans, MS Service Fabric
    • SPA-frontend

    View Slide

  14. How about evolution? Part 1

    View Slide

  15. Evolution as a feature of the model?
    Person
    Entity
    Customer
    Role
    Order
    Agreement
    party
    Spending limit
    Address
    Organisation
    Entity
    Delivery
    BusinessAc tivity
    party
    Invoice
    BusinessAc tivity
    party
    Address
    Address
    Spending limit
    Spending limit
    Own Organisation
    Workarea
    workarea
    Spending limit

    View Slide

  16. Data
    upgrade
    Deployed with
    Executed by
    Executed by
    Data
    upgrade
    Application upgrade
    Lazy transformation
    Upcasting
    In place
    transfor-
    mation
    Multiple versions
    Copy and
    transfor-
    mation
    Basic &
    Complex
    Event
    Basic &
    Complex
    Stream
    Basic
    Store
    Complex
    Store
    Big Flip
    Rolling
    Upgrade
    Blue-Green
    Expand-
    Contract
    Blue-
    Green
    Deployed with
    Combined with
    Weak schema
    Expensive!

    View Slide

  17. Event Publishing part 2

    View Slide

  18. Aggregate
    root
    Command
    Event store
    Event
    Projector
    Query store
    Event Queue
    Query
    request
    Query
    handler

    View Slide

  19. Aggregate
    root
    Command
    Event store
    Event
    Projector
    Query store
    Event Queue
    Query
    request
    Query
    handler
    Can give error (needs retry).
    Command-side in control
    over publishing.

    View Slide

  20. Aggregate
    root
    Command
    Event store
    Event
    Projector
    Query store
    Query
    request
    Query
    handler

    View Slide

  21. Aggregate
    root
    Command
    Event store
    Event
    Projector
    Query store
    Query
    request
    Query
    handler
    Projector in control.
    More load on the event store.

    View Slide

  22. When CRUD is not enough
    business processes

    View Slide

  23. Aggregate
    root
    Command
    Event store
    Event
    Projector
    Query store
    Query
    request
    Query
    handler

    View Slide

  24. Aggregate
    root
    Command
    Event store
    Event
    Projector
    Query store
    Query
    request
    Query
    handler
    Process
    manager

    View Slide

  25. Aggregate
    root
    Command
    Event store
    Event
    Projector
    Query store
    Query
    request
    Query
    handler
    Process
    manager
    But what if these
    commands fail?

    View Slide

  26. Aggregate
    root
    Command
    Event store
    Event
    Projector
    Query store
    Query
    request
    Query
    handler
    Process
    manager
    Event store

    View Slide

  27. Aggregate
    root
    Command
    Event store
    Event
    Projector
    Query store
    Query
    request
    Query
    handler
    Process
    manager
    Event store
    Commands that cannot
    fail become events. We
    still have some internal
    commands.

    View Slide

  28. Aggregate
    root
    Command
    Event store
    Event
    Projector
    Query store
    Query
    request
    Query
    handler
    Process
    manager
    Event store
    Different instances to
    scale the work.
    Event sourced for state.

    View Slide

  29. Aggregate
    root /
    Process
    manager
    Command
    Event store
    Event
    Projector
    Query store
    Query
    request
    Query
    handler
    Is this what we want?

    View Slide

  30. Aggregate
    root /
    Process
    manager
    Command
    Event store
    Event
    Projector
    Query store
    Query
    request
    Query
    handler
    Is this what we want?
    Technically there is no
    difference between
    aggregates and
    processmanagers

    View Slide

  31. The problem of relational data

    View Slide

  32. Aggregate
    root
    Command
    Event store
    Event
    Projector
    Query store
    Query
    request
    Query
    handler
    Process
    manager
    Event store

    View Slide

  33. Aggregate
    root
    Command
    Event store
    Event
    Projector
    Query store
    Query
    request
    Query
    handler
    Process
    manager
    Event store
    Process
    manager
    Process
    manager
    Process
    manager
    Aggregate
    root
    Aggregate
    root
    Aggregate
    root
    Aggregate
    root

    View Slide

  34. Aggregate
    root
    Command
    Event store
    Event
    Projector
    Query store
    Query
    request
    Query
    handler
    Process
    manager
    Event store
    Process
    manager
    Process
    manager
    Process
    manager
    Aggregate
    root
    Aggregate
    root
    Aggregate
    root
    Aggregate
    root
    These ProcessManager
    all need the same data
    (lots of data)

    View Slide

  35. Aggregate
    root
    Command
    Event store
    Event
    Projector
    Query store
    Query
    request
    Query
    handler
    Process
    manager
    Event store
    Stream
    Projector Query store
    Event store

    View Slide

  36. Aggregate
    root
    Command
    Event store
    Event
    Projector
    Query store
    Query
    request
    Query
    handler
    Process
    manager
    Event store
    Stream
    Projector Query store
    Event store
    StreamProjectors are
    single-instanced, non-
    event-sourced
    processmanagers

    View Slide

  37. Eventual consistency
    communication between client and server

    View Slide

  38. Aggregate
    root
    Command
    Event store
    Event
    Projector
    Query store
    Query
    request
    Query
    handler
    Client

    View Slide

  39. Counter
    Process
    Manager
    Count
    Command
    Counted
    Event
    Event store
    Counter
    Projector
    Query store
    Get
    Counter
    Count
    Query
    handler
    Getting the next
    consecutive number in a
    series.

    View Slide

  40. Counter
    Process
    Manager
    Count
    Command
    Counted
    Event
    Event store
    Counter
    Projector
    Query store
    Get
    Counter
    Count
    Query
    handler
    When is the value ready?
    Correlationid?

    View Slide

  41. Aggregate
    root
    Command
    Event store
    Event
    Projector
    Query store
    Query
    request
    Query
    handler
    Client
    Using websockets to
    notify the Client

    View Slide

  42. Aggregate
    root
    Command
    Event store
    Event
    Projector
    Query store
    Query
    request
    Query
    handler
    Client
    Returning values from
    the produced event
    directly

    View Slide

  43. Counter
    Process
    Manager
    Count
    Command
    Counted
    Event
    Event store
    Counter
    Projector
    Query store
    Get
    Counter
    Count
    Query
    handler

    View Slide

  44. How large is our system anyway?

    View Slide

  45. Aggregate
    root
    Command
    Event store
    Event
    Projector
    Query store
    Query
    request
    Query
    handler
    Stream
    Projector Query store
    Event store
    557
    464
    1927
    4849
    1075
    2718
    801
    1479

    View Slide

  46. Store (19)
    S M L
    Traffic (19)
    S M L
    Schema (14)
    S M L

    View Slide

  47. Dealing with bounded contexts

    View Slide

  48. View Slide

  49. Aggregate
    root
    Command
    Event store
    Event
    Aggregate
    root
    Event store
    Event

    View Slide

  50. Aggregate
    root
    Command
    Event store
    Event
    Aggregate
    root
    Event store
    Event
    Team B
    Team A

    View Slide

  51. Aggregate
    root
    Command
    Event store
    Event
    Aggregate
    root
    Event store
    Event
    These events now serve multiple goals.
    Team B
    Team A

    View Slide

  52. Aggregate
    root
    Command
    Event store
    Event
    Aggregate
    root
    Event store
    Event
    Team B
    Team A
    Sync
    Event store
    Event

    View Slide

  53. Aggregate
    root
    Command
    Event store
    Event
    Aggregate
    root
    Event store
    Event
    Team B
    Team A
    Sync
    Event store
    Event
    Introduce synchronization events.

    View Slide

  54. How about evolution? Part 2

    View Slide

  55. Versioned
    Events
    Weak
    Schema
    Upcasters In-Place
    Copy-
    Transform

    View Slide

  56. We will go live in January…
    No guarantees are given ☺

    View Slide

  57. Copyright Nasa Goddard
    Michiel Overeem
    @michielovereem
    • No event bus
    • Commands do have results
    • Projectors can publish events
    • Aggregates can listen to events
    • You can do what you want ☺
    i.e. you do not have to
    listen to a consultant that
    does not know your
    context.

    View Slide