Dealing with change in event sourced applications (Devoxx UK 2018)

Dealing with change in event sourced applications (Devoxx UK 2018)

In software development, change is pretty much the only constant factor. In fact, embracing change is one of the twelve principles behind the Agile Manifesto. As time passes, our understanding of the domain we are working in evolves. We develop based on new requirements, (better) insights, opportunities, changes in the market or legislation, or other factors.

These inputs eventually all lead to modifications to our application, which can be very challenging to implement if the application uses event sourcing. Indeed, when applied very strictly, event sourcing can be quite resistant to change. And unfortunately, there’s not a lot of literature on this subject (yet).

In this talk, we’ll explore how to deal with projection updates, event versioning and the GDPR, and the tools offered by a popular framework (Axon).

2f4800411154a8c66dde489448a044d2?s=128

Michiel Rook

May 11, 2018
Tweet

Transcript

  1. DEALING WITH CHANGE
 IN EVENT SOURCED APPLICATIONS Michiel Rook -

    @michieltcs
  2. None
  3. CHANGE

  4. ' Welcome changing requirements, even late in development. -Agile Manifesto

  5. MODIFICATIONS

  6. EVENT SOURCING

  7. CHALLENGE!

  8. YOU

  9. RAISE YOUR HAND IF YOU HAVE

  10. read CQRS / Event Sourcing theory RAISE YOUR HAND IF

    YOU HAVE
  11. read CQRS / Event Sourcing theory followed a tutorial, built

    a hobby project RAISE YOUR HAND IF YOU HAVE
  12. read CQRS / Event Sourcing theory followed a tutorial, built

    a hobby project used it in production RAISE YOUR HAND IF YOU HAVE
  13. None
  14. QUICK RECAP

  15. ' Event Sourcing ensures that all changes to application state

    are stored as a sequence of events. -Martin Fowler
  16. ACTIVE RECORD VS. EVENT SOURCING Account Id Account number Balance

    1234 12345678 €50,00 ... ... ... Money Withdrawn Account Id 1234 Amount €50,00 Money Deposited Account Id 1234 Amount €100,00 Account Opened Account Id 1234 Account number 12345678 @michieltcs
  17. COMMANDS TO EVENTS Deposit Money Account Id 1234 Amount €100,00

    @michieltcs 1 @Value 2 public class DepositMoney { 3 @TargetAggregateIdentifier 4 String accountId; 5 BigDecimal amount; 6 }
  18. COMMANDS TO EVENTS Deposit Money Account Id 1234 Amount €100,00

    command
 handler @michieltcs 1 @CommandHandler 2 public void depositMoney(DepositMoney command) { 3 apply(new MoneyDeposited( 4 command.getAccountId(), 5 command.getAmount(), 6 ZonedDateTime.now())); 7 }
  19. COMMANDS TO EVENTS Deposit Money Account Id 1234 Amount €100,00

    Money Deposited Account Id 1234 Amount €100,00 command
 handler @michieltcs 1 @Value 2 public class MoneyDeposited { 3 String accountId; 4 BigDecimal amount; 5 ZonedDateTime timestamp; 6 }
  20. AGGREGATES @michieltcs an Aggregate handles Commands and generates Events based

    on the current state
  21. AGGREGATES @michieltcs 1 class BankAccount { 2 @AggregateIdentifier 3 private

    String accountId; 4 private String accountNumber; 5 private BigDecimal balance; 6 7 // ... 8 @EventHandler 9 public void accountOpened(AccountOpened event) { 10 this.accountId = event.getAccountId(); 11 this.accountNumber = event.getAccountNumber(); 12 this.balance = BigDecimal.valueOf(0); 13 } 14 15 @EventHandler 16 public void moneyDeposited(MoneyDeposited event) { 17 this.balance = this.balance.add(event.getAmount()); 18 } 19 }
  22. AGGREGATE STATE Account number Balance 12345678 €0,00 Account number Balance

    12345678 €100,00 Account number Balance 12345678 €50,00 event
 handler event
 handler event
 handler @michieltcs Money Withdrawn Account Id 1234 Amount €50,00 Money Deposited Account Id 1234 Amount €100,00 Account Opened Account Id 1234 Account number 12345678
  23. VALIDATING COMMANDS @michieltcs 1 @CommandHandler 2 public void withdrawMoney(WithdrawMoney command)

    throws 3 OverdraftDetectedException { 4 if (balance.compareTo(command.getAmount()) >= 0) { 5 apply(new MoneyWithdrawn( 6 command.getAccountId(), 7 command.getAmount(), 8 ZonedDateTime.now())); 9 } else { 10 throw new OverdraftDetectedException(accountNumber, balance, command. 11 getAmount()); 12 } 13 }
  24. TESTING AGGREGATES @michieltcs 1 public class BankAccountTest { 2 private

    FixtureConfiguration<BankAccount> fixture; 3 4 @Before 5 public void createFixture() { 6 fixture = new AggregateTestFixture<>(BankAccount.class); 7 } 8 9 @Test 10 public void noOverdraftsOnEmptyAccount() { 11 fixture.given(new AccountOpened(ACCOUNT_ID, ACCOUNT_NUMBER)) 12 .when(new WithdrawMoney(ACCOUNT_ID, new BigDecimal(20))) 13 .expectException(OverdraftDetectedException.class); 14 } 15 16 private final static String ACCOUNT_ID = "accountId"; 17 private final static String ACCOUNT_NUMBER = "accountNumber"; 18 }
  25. REPLAYS AND REBUILDS

  26. ANSWERING QUERIES

  27. BASED ON EVENTS

  28. QUERIES Account Opened Account Opened Account Closed Number of active

    accounts? @michieltcs
  29. QUERIES Money Deposited Money Withdrawn Interest Received Accounts with balance

    > €100? Money Deposited Money Withdrawn @michieltcs
  30. PROJECTION Account Opened Event Handler # of active accounts +1

    Account Closed Event Handler # of active accounts -1 @michieltcs
  31. PROJECTION Events Event Handler(s) Storage @michieltcs

  32. CQRS

  33. UI @michieltcs

  34. Domain UI Command commands Aggregates @michieltcs

  35. Domain UI Command Repository Event Store commands events Aggregates @michieltcs

  36. Domain UI Event Bus Event Handlers Command Repository Database Database

    Event Store commands events events Aggregates @michieltcs
  37. Domain UI Event Bus Event Handlers Command Repository Data Layer

    Database Database Event Store commands events events queries DTOs Aggregates @michieltcs
  38. MULTIPLE AGGREGATES Seller Event Seller Event Seller Event Seller Event

    Seller Event @michieltcs Listing Event Listing Event Listing Event Listing Event Listing Event Seller Listing Seller Name Listing Date Listing Description ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
  39. PROJECTION @michieltcs 1 public class BankAccountProjections { 2 private Map<String,

    String> activeAccounts = new HashMap<>(); 3 4 @EventHandler 5 public void onAccountOpened(AccountOpened accountOpened) { 6 activeAccounts.put(accountOpened.getAccountId(), accountOpened. 7 getAccountNumber()); 8 } 9 10 @EventHandler 11 public void onAccountClosed(AccountClosed accountClosed) { 12 activeAccounts.remove(accountClosed.getAccountId()); 13 } 14 15 public String getAccountNumberForAccountId(String accountId) { 16 return activeAccounts.get(accountId); 17 } 18 }
  40. PROJECTION @michieltcs 1 @GetMapping("accounts/{accountId}") 2 public String getAccountNumber(@PathVariable("accountId") String accountId)

    { 3 return bankAccountProjections.getAccountNumberForAccountId(accountId); 4 }
  41. NEW PROJECTION

  42. NEW STRUCTURE

  43. BASED ON EXISTING EVENTS

  44. REBUILDING Stop app Cleanup Loop over events Apply to projection

    Start app @michieltcs
  45. ZERO DOWNTIME Loop over existing events Apply to new projection

    Use projection @michieltcs
  46. ZERO DOWNTIME New events Queue Loop over existing events Apply

    to new projection Apply queued events Use projection @michieltcs
  47. ZERO DOWNTIME Get next event Apply to new projection Last

    event? Use projection yes no @michieltcs
  48. LONG RUNNING REBUILDS?

  49. IN MEMORY

  50. DISTRIBUTED

  51. DIVIDING THE WORK Event Event Event Event Event @michieltcs Aggregate

    Event
  52. DIVIDING THE WORK Event Event Event Event Event @michieltcs Aggregate

    Instance Event Instance
  53. DIVIDING THE WORK Event Event Event Event Event @michieltcs Aggregate

    Instance Event Instance
  54. DIVIDING THE WORK Event Event Event Event Event @michieltcs Aggregate

    Instance Event Event Event Event Instance
  55. DIVIDING THE WORK Event Event Event Event Event @michieltcs Aggregate

    Instance Event Event Event Event Instance Event Event Event
  56. DIVIDING THE WORK Seller Event Seller Event Seller Event Seller

    Event Seller Event @michieltcs Listing Event Listing Event Listing Event Listing Event Listing Event Seller Listing Seller Name Listing Date Listing Description ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
  57. DIVIDING THE WORK Seller Event Seller Event Seller Event Seller

    Event Seller Event @michieltcs Listing Event Listing Event Listing Event Listing Event Listing Event Seller Listing Instance Instance
  58. DIVIDING THE WORK Seller Event Seller Event Seller Event Seller

    Event Seller Event @michieltcs Listing Event Listing Event Listing Event Listing Event Listing Event Seller Listing Instance Instance ?
  59. BACKGROUND TASK

  60. TRACKING EVENTS @michieltcs Event Event Event Event Event Event .

    . . . . . . . . . . . . . Event Event Event
  61. TRACKING EVENTS @michieltcs Event Event Event Event Event Event .

    . . . . . . . . . . . . . Event Event Event
  62. TRACKING EVENTS @michieltcs Event Event Event Event Event Event .

    . . . . . . . . . . . . . Event Event Event
  63. TRACKING EVENTS @michieltcs Event Event Event Event Event Event .

    . . . . . . . . . . . . . Event Event Event
  64. TRACKING EVENTS @michieltcs Event Event Event Event Event Event .

    . . . . . . . . . . . . . Event Event Event
  65. TRACKING EVENTS @michieltcs Event Event Event Event Event Event .

    . . . . . . . . . . . . . Event Event Event Token Store
  66. TRACKING EVENTS @michieltcs Event Event Event Event Event Event .

    . . . . . . . . . . . . . Event Event Event
  67. TRACKING EVENTS @michieltcs Event Event Event Event Event Event .

    . . . . . . . . . . . . . Event Event Event
  68. TRACKING EVENTS @michieltcs Event Event Event Event Event Event .

    . . . . . . . . . . . . . Event Event Event
  69. TRACKING EVENTS @michieltcs Event Event Event Event Event Event .

    . . . . . . . . . . . . . Event Event Event
  70. TRACKING EVENT PROCESSOR @michieltcs 1 @Target(ElementType.TYPE) 2 @Retention(RetentionPolicy.RUNTIME) 3 public

    @interface RebuildableProjection { 4 String version() default ""; 5 boolean rebuild() default false; 6 }
  71. TRACKING EVENT PROCESSOR @michieltcs 1 @Configuration 2 public class ProjectionsConfiguration

    { 3 @Autowired 4 private EventHandlingConfiguration eventHandlingConfiguration;
  72. TRACKING EVENT PROCESSOR @michieltcs 1 @PostConstruct 2 public void startTrackingProjections()

    throws ClassNotFoundException { 3 ClassPathScanningCandidateComponentProvider scanner = 4 new ClassPathScanningCandidateComponentProvider(false); 5 scanner.addIncludeFilter(new AnnotationTypeFilter(RebuildableProjection. 6 class)); 7 8 for (BeanDefinition bd : scanner.findCandidateComponents("org.demo")) { 9 Class<?> aClass = Class.forName(bd.getBeanClassName()); 10 RebuildableProjection rebuildableProjection = aClass.getAnnotation( 11 RebuildableProjection. 12 class); 13 14 if (rebuildableProjection.rebuild()) { 15 registerRebuildableProjection(aClass, rebuildableProjection); 16 } 17 } 18 }
  73. TRACKING EVENT PROCESSOR @michieltcs 1 private void registerRebuildableProjection(Class<?> aClass, 2

    RebuildableProjection 3 rebuildableProjection) { 4 ProcessingGroup processingGroup = aClass.getAnnotation(ProcessingGroup. 5 class); 6 7 String name = Optional.ofNullable(processingGroup).map(ProcessingGroup:: 8 value) 9 .orElse(aClass.getName() + "/" + rebuildableProjection.version()); 10 11 eventHandlingConfiguration.assignHandlersMatching( 12 name, 13 Integer.MAX_VALUE, 14 (eventHandler) -> aClass.isAssignableFrom(eventHandler.getClass())); 15 16 eventHandlingConfiguration.registerTrackingProcessor(name); 17 }
  74. EVENT VERSIONING

  75. NEW BUSINESS REQUIREMENTS

  76. CHANGING VIEW ON EVENTS

  77. IRRELEVANT

  78. DIFFERENT FIELDS

  79. WRONG NAME

  80. TOO COARSE

  81. TOO FINE

  82. SUPPORT YOUR LEGACY?

  83. Commands can be renamed 1 @michieltcs

  84. Commands can be renamed 1 Events are immutable 2 @michieltcs

  85. Commands can be renamed 1 Events are immutable Correct old

    events with new events 2 3 @michieltcs
  86. @michieltcs Ledger Entry Aug 14 Inventory €15600,00 Accounts Payable €15600,00

  87. @michieltcs Ledger Entry Aug 14 Inventory €15600,00 Accounts Payable €15600,00

    Ledger Entry Aug 14 Inventory €16500,00 Accounts Payable €16500,00
  88. @michieltcs Ledger Entry Aug 14 Inventory €15600,00 Accounts Payable €15600,00

    Ledger Entry Aug 14 Inventory €16500,00 Accounts Payable €16500,00 Ledger Correction Entry Aug 14 Inventory €900,00 Accounts Payable €900,00
  89. COMPENSATING ACTIONS class MoneyWithdrawn {
 String accountId;
 BigDecimal amount;
 }

    class WithdrawalRolledBack {
 String accountId;
 BigDecimal amount;
 } Typo: too much withdrawn!
  90. COMPENSATING ACTIONS class AccountOpened {
 String accountId;
 String accountNumber;
 }

    class DuplicateAccountClosed {
 String accountId;
 } Duplicate account number!
  91. UPCASTING

  92. UPCASTING Event Store @michieltcs

  93. UPCASTING @michieltcs 1 @Value 2 @Revision("1.0") 3 public class AccountOpened

    { 4 String accountId; 5 String accountNumber; 6 }
  94. UPCASTING Event Store AccountOpened (1.0) Event Handler @michieltcs

  95. UPCASTING @michieltcs 1 @Value 2 @Revision("2.0") 3 public class AccountOpened

    { 4 String accountId; 5 String accountNumberIban; 6 }
  96. UPCASTING Event Store AccountOpened (1.0) Upcaster AccountOpened (2.0) Event Handler

    @michieltcs
  97. UPCASTING @michieltcs 1 @Getter 2 public class AccountOpenedUpcaster implements EventUpcaster

    { 3 private final SerializedType typeConsumed 4 = new SimpleSerializedType(AccountOpened.class.getTypeName(), "1.0"); 5 private final SerializedType typeProduced 6 = new SimpleSerializedType(AccountOpened.class.getTypeName(), "2.0");
  98. UPCASTING @michieltcs 1 @Override 2 public final Stream<IntermediateEventRepresentation> upcast( 3

    Stream<IntermediateEventRepresentation> intermediateRepresentations) { 4 return intermediateRepresentations.map(evt -> { 5 if (evt.getType().equals(getTypeConsumed())) { 6 return evt.upcastPayload(getTypeProduced(), Document.class, 7 document -> { 8 Element rootElement = document.getRootElement(); 9 Element accountNumberElement = rootElement.element( 10 "accountNumber"); 11 rootElement.remove(accountNumberElement); 12 rootElement.addElement("accountNumberIban") 13 .setText(toIban(accountNumberElement.getText())); 14 return document; 15 }); 16 } else { 17 return evt; 18 } 19 }); 20 }
  99. VERSIONED EVENT STORE

  100. VERSIONED EVENT STORE events_v1 [
 {
 "id": "12345678",
 "type": "AccountOpened",


    "aggregateType": "Account",
 "aggregateIdentifier": "1234",
 "sequenceNumber": 0,
 "payloadRevision": "1.0",
 "payload": { ... },
 "timestamp": ...
 ...
 },
 ...
 ] @michieltcs
  101. COPY & REPLACE

  102. VERSIONED EVENT STORE Loop over existing events Apply upcaster Add

    queued events Use new event store New events Queue @michieltcs
  103. VERSIONED EVENT STORE events_v2 [
 {
 "id": "12345678",
 "type": "AccountOpened",


    "aggregateType": "Account",
 "aggregateIdentifier": "1234",
 "sequenceNumber": 0,
 "payloadRevision": "2.0",
 "payload": { ... },
 "timestamp": ...
 ...
 },
 ...
 ] @michieltcs
  104. GDPR

  105. "RIGHT TO ERASURE"

  106. ' ... shall have the right to obtain ... the

    erasure of personal data concerning him or her without undue delay -GDPR, Article 17
  107. PERSONALLY IDENTIFIABLE INFORMATION

  108. REMOVED

  109. ANONYMIZED?

  110. PROCESSING GDPR ART. 17 REQUESTS @michieltcs RightToErasureInvoked Remove from event

    store Remove from read models Notify 3rd parties
  111. PROCESSING GDPR ART. 17 REQUESTS @michieltcs RightToErasureInvoked Remove from event

    store ? Remove from read models Notify 3rd parties
  112. IMMUTABLE EVENTS?

  113. MODIFY DIRECTLY

  114. COPY WITH FILTER

  115. STORE PII EXTERNALLY

  116. STORE PII EXTERNALLY @michieltcs AccountOpened External Storage 1 @Value 2

    public class AccountOpened { 3 String accountId; 4 } Account Id Account number Name 1234 12345678 John Doe ... ... ...
  117. CRYPTO ERASURE

  118. AXON GDPR MODULE @michieltcs 1 @Value 2 public class AccountOpened

    { 3 @DataSubjectId 4 String accountId; 5 6 @PersonalData 7 String accountNumber; 8 9 @PersonalData 10 String fullName; 11 }
  119. CLOSING WORDS

  120. CQRS + ES = AWESOME

  121. NO SILVER BULLET

  122. CHALLENGES

  123. (IM)MUTABILITY

  124. AUDIT TRAIL

  125. SCALABILITY

  126. TESTING

  127. THANK YOU! @michieltcs / michiel@michielrook.nl
 
 www.michielrook.nl