CQRS and Event Sourcing in the Wild (IPC Berlin 2017)

CQRS and Event Sourcing in the Wild (IPC Berlin 2017)

CQRS and event sourcing are currently very popular topics in the PHP community, with good reason. However, most blogs and talks focus on the theory, simple applications or introductions to one of the frameworks currently available, not necessarily the challenges of using and maintaining it in production.
This session bridges that gap and looks at some of the pitfalls of a real-world deployment. I’ll discuss topics like concurrency and scale, refactoring events and updating read models. Attend this talk to learn from my experiences and be better prepared when you face these challenges.

2f4800411154a8c66dde489448a044d2?s=128

Michiel Rook

May 31, 2017
Tweet

Transcript

  1. CQRS & EVENT SOURCING IN THE WILD Michiel Rook -

    @michieltcs
  2. ➤ Developer, consultant, trainer, speaker ➤ @michieltcs

  3. YOU

  4. RAISE YOUR HAND IF YOU HAVE

  5. read CQRS / Event Sourcing theory RAISE YOUR HAND IF

    YOU HAVE
  6. read CQRS / Event Sourcing theory followed a tutorial, built

    a hobby project RAISE YOUR HAND IF YOU HAVE
  7. read CQRS / Event Sourcing theory followed a tutorial, built

    a hobby project used it in production RAISE YOUR HAND IF YOU HAVE
  8. QUICK RECAP

  9. ' Event Sourcing ensures that all changes to application state

    are stored as a sequence of events. -Martin Fowler
  10. ACTIVE RECORD VS. EVENT SOURCING Account number Balance 12345678 €€50.00

    ... ... Money Withdrawn Account number 12345678 Amount €€50.00 Money Deposited Account number 12345678 Amount €€100.00 Account Created Account number 12345678 @michieltcs
  11. COMMANDS TO EVENTS Deposit Money Account number 12345678 Amount €€100.00

    class DepositMoney {
 public $accountNumber;
 public $amount;
 } @michieltcs
  12. COMMANDS TO EVENTS Deposit Money Account number 12345678 Amount €€100.00

    function depositMoney(DepositMoney $command) {
 $this->apply(new MoneyDeposited(
 $command->accountNumber,
 $command->amount,
 date()));
 } command
 handler @michieltcs
  13. COMMANDS TO EVENTS Deposit Money Account number 12345678 Amount €€100.00

    Money Deposited Account number 12345678 Amount €€100.00 class MoneyDeposited {
 public $accountNumber;
 public $amount;
 public $timestamp;
 } command
 handler @michieltcs
  14. AGGREGATES class BankAccount {
 public $accountNumber;
 public $balance;
 
 //

    ...
 
 public function applyMoneyDeposited(
 MoneyDeposited $event) {
 $this->balance += $event->amount;
 }
 @michieltcs
  15. AGGREGATE STATE Account number Balance 12345678 €€0.00 Money Withdrawn Account

    number 12345678 Amount €€50.00 Money Deposited Account number 12345678 Amount €€100.00 Account Created Account number 12345678 Account number Balance 12345678 €€100.00 Account number Balance 12345678 €€50.00 event
 handler event
 handler event
 handler @michieltcs
  16. Domain UI Event Bus Event Handlers Command Repository Data Layer

    Database Database Event Store commands events events queries DTOs Aggregates @michieltcs
  17. DISCLAIMER

  18. REPLAYS AND REBUILDS

  19. ANSWERING QUERIES

  20. BASED ON EVENTS

  21. QUERIES User Registered User Registered User Unregistered Number of active

    users? @michieltcs
  22. QUERIES User Registered User Deactivated User Reactivated Number of active

    users? User Registered User Unregistered @michieltcs
  23. PROJECTION User Registered Event Handler Number of active users +1

    User Unregistered Event Handler Number of active users -1 @michieltcs
  24. PROJECTION Events Event Handler(s) Storage @michieltcs

  25. ELASTICSEARCH class CompanyRegistered {
 public $companyId;
 public $name;
 public $street;


    public $city;
 } @michieltcs
  26. ELASTICSEARCH function handleCompanyRegistered(
 CompanyRegistered $event) {
 
 $this->elasticsearch->index([
 'index' =>

    'companies',
 'type' => 'company',
 'id' => $event->companyId,
 'body' => [
 'name' => $event->name,
 'address' => [
 'street' => $event->street,
 'city' => $event->city
 ]
 ]
 ]);
 } @michieltcs
  27. NEW PROJECTION

  28. NEW STRUCTURE

  29. BASED ON EXISTING EVENTS

  30. REBUILDING Stop app Cleanup Loop over events Apply to projection

    Start app @michieltcs
  31. ZERO DOWNTIME Loop over existing events Apply to new projection

    Use projection @michieltcs
  32. ZERO DOWNTIME New events Queue Loop over existing events Apply

    to new projection Apply queued events Use projection @michieltcs
  33. ZERO DOWNTIME Get next event Apply to new projection Last

    event? Use projection yes no @michieltcs
  34. LONG RUNNING REBUILDS

  35. IN MEMORY

  36. DISTRIBUTED

  37. PARTIAL

  38. BACKGROUND TASK

  39. SIDE EFFECTS User Registered User Id 123abc Email Address test@example.

    net @michieltcs
  40. SIDE EFFECTS User Registered User Id 123abc Email Address test@example.

    net Event Handler @michieltcs
  41. SIDE EFFECTS User Registered User Id 123abc Email Address test@example.

    net Event Handler @michieltcs
  42. SIDE EFFECTS User Registered User Id 123abc Email Address test@example.

    net Event Handler Exclude during replays! @michieltcs
  43. MULTIPLE EVENT HANDLERS Event Handler Event @michieltcs

  44. MULTIPLE EVENT HANDLERS Event Handler Event Handler Event @michieltcs

  45. MULTIPLE EVENT HANDLERS Event Handler Event Handler Event Event Handler

    @michieltcs
  46. MULTIPLE EVENT HANDLERS Event Handler Event Handler Event Event Handler

    Event Handler ? @michieltcs
  47. EVENTUAL CONSISTENCY ➤ Reads eventually return the same value ➤

    Asynchronous event handlers ➤ Projections in event handlers? @michieltcs
  48. UI ➤ Optimistic UI ➤ Websockets ➤ React + Redux

    (Saga) @michieltcs
  49. EVENT VERSIONING

  50. NEW BUSINESS REQUIREMENTS

  51. CHANGING VIEW ON EVENTS

  52. IRRELEVANT

  53. DIFFERENT FIELDS

  54. WRONG NAME

  55. TOO COARSE

  56. TOO FINE

  57. SUPPORT YOUR LEGACY?

  58. Commands can be renamed 1 @michieltcs

  59. Commands can be renamed 1 Events are immutable 2 @michieltcs

  60. Commands can be renamed 1 Events are immutable Correct old

    events with new events 2 3 @michieltcs
  61. UPCASTING

  62. UPCASTING Event Store @michieltcs

  63. UPCASTING class UserRegistered_V1 {
 public $userId;
 public $name;
 public $timestamp;


    } @michieltcs
  64. UPCASTING Event Store UserRegistered_V1 Event Handler @michieltcs

  65. UPCASTING class UserRegistered_V2 {
 public $userId;
 public $name;
 public $date;


    } @michieltcs
  66. UPCASTING Event Store UserRegistered_V1 Upcaster UserRegistered_V2 Event Handler @michieltcs

  67. UPCASTING function upcast($event): array {
 if (!$event instanceof UserRegistered_V1) {


    return [];
 }
 return [
 new UserRegistered_V2(
 $event->userId, $event->name,
 $event->timestamp->format("Y-m-d"))
 ];
 } @michieltcs
  68. VERSIONED EVENT STORE

  69. VERSIONED EVENT STORE events_v1 [
 {
 "id": "12345678",
 "type": "UserRegistered_V1",


    "aggregateType": "User",
 "aggregateIdentifier": "1234",
 "sequenceNumber": 0,
 "payload": { ... },
 "timestamp": ...
 ...
 },
 ...
 ] @michieltcs
  70. COPY & REPLACE

  71. VERSIONED EVENT STORE Loop over existing events Apply upcaster Add

    queued events Use new event store New events Queue @michieltcs
  72. VERSIONED EVENT STORE events_v2 [
 {
 "id": "12345678",
 "type": "UserRegistered_V2",


    "aggregateType": "User",
 "aggregateIdentifier": "1234",
 "sequenceNumber": 0,
 "payload": { ... },
 "timestamp": ...
 ...
 },
 ...
 ] @michieltcs
  73. REWRITING HISTORY Load (subset of) events Deserialize Apply upcaster Serialize

    Save/replace @michieltcs
  74. THINGS TO BE AWARE OF Upcasting Performance @michieltcs

  75. THINGS TO BE AWARE OF Upcasting Performance Complexity @michieltcs

  76. THINGS TO BE AWARE OF Upcasting Rewriting events Performance Complexity

    Rewriting history @michieltcs
  77. THINGS TO BE AWARE OF Upcasting Rewriting events Performance Complexity

    Rewriting history Code for old events @michieltcs
  78. THINGS TO BE AWARE OF Upcasting Rewriting events Performance Complexity

    Rewriting history Code for old events Breaking serialization @michieltcs
  79. THINGS TO BE AWARE OF Upcasting Rewriting events Performance Complexity

    Rewriting history Code for old events Breaking serialization Existing projections not automatically updated! @michieltcs
  80. CONCURRENCY

  81. CONCURRENT COMMANDS Withdraw Money Account number 12345678 Amount €€50.00 Deposit

    Money Account number 12345678 Amount €€100.00 ? @michieltcs
  82. PESSIMISTIC LOCKING Withdraw Money Account number 12345678 Amount €€50.00 Deposit

    Money Account number 12345678 Amount €€100.00 Account number Balance 12345678 €€100.00 Account number Balance 12345678 €€50.00 wait for lock lock @michieltcs
  83. OPTIMISTIC LOCKING Withdraw Money Account number 12345678 Amount €€50.00 version

    1 Deposit Money Account number 12345678 Amount €€100.00 version 1 Account number Balance 12345678 €€100.00 version 2 ConcurrencyException @michieltcs
  84. SCALE

  85. PERFORMANCE Server @michieltcs

  86. PERFORMANCE Server Database @michieltcs

  87. PERFORMANCE Server Database Framework @michieltcs

  88. PERFORMANCE Server Database Framework Language @michieltcs

  89. PERFORMANCE Server Database Framework Language Serializer @michieltcs

  90. STORAGE #events @michieltcs

  91. STORAGE #events #aggregates @michieltcs

  92. STORAGE #events #aggregates #events per aggregate

  93. STORAGE #events #aggregates #events per aggregate Serializer @michieltcs

  94. STORAGE #events #aggregates #events per aggregate Serializer Payload @michieltcs

  95. SNAPSHOTS Events ... ... 1001 Account Created 1002 Money Deposited

    1003 Money Withdrawn 1004 Money Deposited @michieltcs
  96. SNAPSHOTS Events ... ... 1001 Account Created 1002 Money Deposited

    1003 Money Withdrawn 1004 Money Deposited 1005 Money Withdrawn @michieltcs
  97. SNAPSHOTS Events ... ... 1001 Account Created 1002 Money Deposited

    1003 Money Withdrawn 1004 Money Deposited SNAPSHOT 1005 Money Withdrawn Events ... ... 1001 Account Created 1002 Money Deposited 1003 Money Withdrawn 1004 Money Deposited 1005 Money Withdrawn @michieltcs
  98. ARCHIVING EVENTS

  99. Reduce working set @michieltcs

  100. Reduce working set Inactive aggregates @michieltcs

  101. Reduce working set Inactive aggregates Historic events @michieltcs

  102. Reduce working set Inactive aggregates Historic events Cheaper storage @michieltcs

  103. FRAMEWORK COMPARISON Upcasting Snapshots Replaying Conflict detection Broadway (PHP) No

    (PR) No (PR) Not in core Not in core Prooph (PHP) Upcaster / UpcasterChain Yes, triggers on event count Example code, off line Depends on store Axon (Java/Scala) Upcaster / UpcasterChain Yes, triggers on event count Replaying Cluster / Tracking Processor Yes Akka Persistence (Java/Scala) Event Adapter Yes, decided by actor Yes Yes @michieltcs
  104. PROOPH class TodoItem {
 /**
 * @AggregateIdentifier
 */
 private $itemId;


    
 /**
 * @CommandHandler
 */
 public function __construct(PostTodo $command) {
 AggregateLifecycle::recordThat(
 new TodoPosted($command->getItemId()));
 }
 
 /**
 * @EventHandler
 */
 public function onTodoPosted(TodoPosted $event) {
 echo "Posted: " . $event->getItemId() . "\n";
 }
 } @michieltcs
  105. https://github.com/prooph/annotations

  106. CLOSING WORDS

  107. CQRS + ES = AWESOME

  108. NO SILVER BULLET

  109. COMPLEXITY

  110. FRAMEWORK SUPPORT

  111. INFRASTRUCTURE

  112. AUDIT TRAIL

  113. SCALABILITY

  114. TESTING

  115. DOMAIN FIT

  116. LITERATURE

  117. THANK YOU! @michieltcs / mrook@fourscouts.nl
 
 www.michielrook.nl