Forget me, please? Event sourcing and the GDPR

In May 2018, a new piece of EU legislation called the General Data Protection Regulation (GDPR) has come into effect. The GDPR attempts to regulate data protection for individuals within the EU and has very interesting and specific implications for applications that use event sourcing. In that context, the most interesting and challenging part of the regulation is without a doubt Article 17, the ‘right to erasure’: the right for individuals to request that their data be removed from the databases of a company.

In this talk, I’ll discuss my thoughts on the GDPR and give a few helpful pointers and tips!

Transcript

1. FORGET ME PLEASE?

   EVENT SOURCING & THE GDPR
   Michiel Rook - @michieltcs
   

   View Slide

2. DISCLAIMER:
   I AM NOT A LAWYER
   

   View Slide

3. GDPR
   

   View Slide

4. GENERAL DATA
   PROTECTION REGULATION
   

   View Slide

5. '
   Regulation (EU) 2016/679 of the European
   Parliament and of the Council of 27 April 2016 on
   the protection of natural persons with regard to the
   processing of personal data and on the free
   movement of such data, and repealing Directive
   95/46/EC (Data Protection Directive)
   -General Data Protection Regulation
   

   View Slide

6. A SHORT HISTORY
   

   View Slide

7. 1995
   Data Protection Directive
   

   View Slide

8. 1995
   Data Protection Directive
   2012
   GDPR proposal
   

   View Slide

9. 1995
   Data Protection Directive
   2012
   GDPR proposal
   2016
   GDPR adopted
   

   View Slide

10. 1995
    Data Protection Directive
    2012
    GDPR proposal
    2016
    GDPR adopted
    25 May 2018
    GDPR enforceable
    

    View Slide

11. REGULATION
    

    View Slide

12. PROTECTS EU CITIZENS
    

    View Slide

13. DATA PROTECTION ACT
    

    View Slide

14. BROAD & VAGUE
    

    View Slide

15. PRIVACY BY DESIGN
    

    View Slide

16. '
    The controller shall implement
    appropriate technical and organisational
    measures for ensuring that, by default,
    only personal data which are necessary for
    each specific purpose of the processing are
    processed. -GDPR, Article 25
    

    View Slide

17. DATA PROTECTION
    OFFICER
    

    View Slide

18. SUPERVISORY
    AUTHORITY
    

    View Slide

19. FINES
    

    View Slide

20. €20 MILLION OR 4% OF
    ANNUAL TURNOVER
    

    View Slide

21. YOU
    

    View Slide

22. RAISE YOUR HAND
    IF YOU HAVE
    

    View Slide

23. read CQRS / Event Sourcing theory
    RAISE YOUR HAND
    IF YOU HAVE
    

    View Slide

24. read CQRS / Event Sourcing theory
    followed a tutorial, built a hobby project
    RAISE YOUR HAND
    IF YOU HAVE
    

    View Slide

25. read CQRS / Event Sourcing theory
    followed a tutorial, built a hobby project
    used it in production
    RAISE YOUR HAND
    IF YOU HAVE
    

    View Slide

26. Axon Framework Spring Boot
    

    View Slide

27. QUICK RECAP
    CQRS + EVENT SOURCING
    

    View Slide

28. CQRS
    

    View Slide

29. COMMAND QUERY
    RESPONSIBILITY
    SEGREGATION
    

    View Slide

30. STORAGE SIDE

    VS.

    QUERY SIDE
    

    View Slide

31. UI
    @michieltcs
    

    View Slide

32. Domain
    UI
    Command
    commands
    Aggregates
    @michieltcs
    

    View Slide

33. Domain
    UI
    Command
    Repository
    Event
    Store
    commands
    events
    Aggregates
    @michieltcs
    

    View Slide

34. Domain
    UI
    Event Bus
    Event Handlers
    Command
    Repository
    Database Database
    Event
    Store
    commands
    events
    events
    Aggregates
    @michieltcs
    

    View Slide

35. Domain
    UI
    Event Bus
    Event Handlers
    Command
    Repository
    Data Layer
    Database Database
    Event
    Store
    commands
    events
    events
    queries DTOs
    Aggregates
    @michieltcs
    

    View Slide

36. EVENT SOURCING
    

    View Slide

37. '
    Event Sourcing ensures that all
    changes to application state are stored
    as a sequence of events.
    -Martin Fowler
    

    View Slide

38. ACTIVE RECORD VS. EVENT SOURCING
    Account Id Account number Balance
    1234 12345678 €50,00
    ... ... ...
    Money Withdrawn
    Account Id 1234
    Amount €50,00
    Money Deposited
    Account Id 1234
    Amount €100,00
    Account Opened
    Account Id 1234
    Account number 12345678
    @michieltcs
    

    View Slide

39. COMMANDS TO EVENTS
    Deposit Money
    Account Id 1234
    Amount €100,00
    @michieltcs
    1 @Value
    2 public class DepositMoney {
    3 @TargetAggregateIdentifier
    4 String accountId;
    5 BigDecimal amount;
    6 }
    

    View Slide

40. COMMANDS TO EVENTS
    Deposit Money
    Account Id 1234
    Amount €100,00
    command

    handler
    @michieltcs
    1 @CommandHandler
    2 public void depositMoney(DepositMoney command) {
    3 apply(new MoneyDeposited(
    4 command.getAccountId(),
    5 command.getAmount(),
    6 ZonedDateTime.now()));
    7 }
    

    View Slide

41. COMMANDS TO EVENTS
    Deposit Money
    Account Id 1234
    Amount €100,00
    Money Deposited
    Account Id 1234
    Amount €100,00
    command

    handler
    @michieltcs
    1 @Value
    2 public class MoneyDeposited {
    3 String accountId;
    4 BigDecimal amount;
    5 ZonedDateTime timestamp;
    6 }
    

    View Slide

42. AGGREGATES
    @michieltcs
    an Aggregate handles Commands and
    generates Events based on the current state
    

    View Slide

43. AGGREGATES
    @michieltcs
    1 class BankAccount {
    2 @AggregateIdentifier
    3 private String accountId;
    4 private String accountNumber;
    5 private BigDecimal balance;
    6
    7 // ...
    8 @EventHandler
    9 public void accountOpened(AccountOpened event) {
    10 this.accountId = event.getAccountId();
    11 this.accountNumber = event.getAccountNumber();
    12 this.balance = BigDecimal.valueOf(0);
    13 }
    14
    15 @EventHandler
    16 public void moneyDeposited(MoneyDeposited event) {
    17 this.balance = this.balance.add(event.getAmount());
    18 }
    19 }
    

    View Slide

44. AGGREGATE STATE
    Account number Balance
    12345678 €0,00
    Account number Balance
    12345678 €100,00
    Account number Balance
    12345678 €50,00
    event

    handler
    event

    handler
    event

    handler
    @michieltcs
    Money Withdrawn
    Account Id 1234
    Amount €50,00
    Money Deposited
    Account Id 1234
    Amount €100,00
    Account Opened
    Account Id 1234
    Account number 12345678
    

    View Slide

45. VALIDATING COMMANDS
    @michieltcs
    1 @CommandHandler
    2 public void withdrawMoney(WithdrawMoney command) throws
    3 OverdraftDetectedException {
    4 if (balance.compareTo(command.getAmount()) >= 0) {
    5 apply(new MoneyWithdrawn(
    6 command.getAccountId(),
    7 command.getAmount(),
    8 ZonedDateTime.now()));
    9 } else {
    10 throw new OverdraftDetectedException(accountNumber, balance, command.
    11 getAmount());
    12 }
    13 }
    

    View Slide

46. TESTING AGGREGATES
    @michieltcs
    1 public class BankAccountTest {
    2 private FixtureConfiguration fixture;
    3
    4 @Before
    5 public void createFixture() {
    6 fixture = new AggregateTestFixture<>(BankAccount.class);
    7 }
    8
    9 @Test
    10 public void noOverdraftsOnEmptyAccount() {
    11 fixture.given(new AccountOpened(ACCOUNT_ID, ACCOUNT_NUMBER))
    12 .when(new WithdrawMoney(ACCOUNT_ID, new BigDecimal(20)))
    13 .expectException(OverdraftDetectedException.class);
    14 }
    15
    16 private final static String ACCOUNT_ID = "accountId";
    17 private final static String ACCOUNT_NUMBER = "accountNumber";
    18 }
    

    View Slide

47. EVENT SOURCING

    & GDPR
    

    View Slide

48. CONSENT
    

    View Slide

49. '
    Where processing is based on consent,
    the controller shall be able to
    demonstrate that the data subject has
    consented to processing of his or her
    personal data.
    -GDPR, Article 7
    

    View Slide

50. REGISTERING CONSENT
    

    View Slide

51. '
    ...the request for consent shall be
    presented in a manner which is clearly
    distinguishable from the other
    matters...
    -GDPR, Article 7
    

    View Slide

52. REVOKING CONSENT
    

    View Slide

53. '
    The data subject shall have the right to
    withdraw his or her consent at any
    time. ... It shall be as easy to withdraw
    as to give consent.
    -GDPR, Article 7
    

    View Slide

54. WHY USE EVENT
    SOURCING / CQRS?
    

    View Slide

55. CAPTURE INTENT
    

    View Slide

56. DEMONSTRATING
    CONSENT
    

    View Slide

57. EVENT LOG

    AS AUDIT LOG
    

    View Slide

58. NEW READ MODELS
    

    View Slide

59. EASIER DEBUGGING
    

    View Slide

60. EVENT LOG AS AUDIT LOG
    @michieltcs
    ConsentedToNewsletters
    

    View Slide

61. EVENT LOG AS AUDIT LOG
    @michieltcs
    ConsentedToNewsletters
    ConsentedToDataGathering
    

    View Slide

62. EVENT LOG AS AUDIT LOG
    @michieltcs
    ConsentedToNewsletters
    ConsentedToDataGathering
    RevokedConsentToNewsletters
    

    View Slide

63. "RIGHT TO ACCESS"
    

    View Slide

64. '
    The data subject shall have the right to
    obtain from the controller
    confirmation as to whether or not
    personal data ... are being processed,
    and ... access to the personal data ...
    -GDPR, Article 15
    

    View Slide

65. "RIGHT TO ERASURE"
    

    View Slide

66. '
    The data subject shall have the right to
    obtain from the controller the erasure
    of personal data concerning him or
    her without undue delay
    -GDPR, Article 17
    

    View Slide

67. PERSONALLY
    IDENTIFIABLE
    INFORMATION
    

    View Slide

68. '
    ‘personal data’ means any information
    relating to an identified or identifiable
    natural person; an identifiable natural
    person is one who can be identified,
    directly or indirectly
    -GDPR, Article 4
    

    View Slide

69. GROUNDS
    

    View Slide

70. '
    .. the personal data are no longer
    necessary
    .. the data subject withdraws consent
    on which the processing is based
    -GDPR, Article 17
    

    View Slide

71. EXCEPTIONS
    

    View Slide

72. '
    .. to comply with a legal obligation
    .. for the establishment, exercise or
    defence of legal claims (*)
    -GDPR, Article 17
    

    View Slide

73. UNDUE DELAY
    

    View Slide

74. INFORM 3RD PARTIES
    

    View Slide

75. BACKUPS?
    

    View Slide

76. PROCESSING GDPR ART. 17 REQUESTS
    @michieltcs
    RightToErasureInvoked
    

    View Slide

77. PROCESSING GDPR ART. 17 REQUESTS
    @michieltcs
    RightToErasureInvoked
    Notify 3rd
    parties
    

    View Slide

78. PROCESSING GDPR ART. 17 REQUESTS
    @michieltcs
    RightToErasureInvoked
    Remove
    from read
    models
    Notify 3rd
    parties
    

    View Slide

79. PROCESSING GDPR ART. 17 REQUESTS
    @michieltcs
    RightToErasureInvoked
    Remove
    from event
    store
    Remove
    from read
    models
    Notify 3rd
    parties
    

    View Slide

80. PROCESSING GDPR ART. 17 REQUESTS
    @michieltcs
    RightToErasureInvoked
    Remove
    from event
    store
    ?
    Remove
    from read
    models
    Notify 3rd
    parties
    

    View Slide

81. IMMUTABLE EVENTS?
    

    View Slide

82. COMPENSATING ACTIONS
    

    View Slide

83. @michieltcs
    Ledger Entry
    Aug 14 Inventory €15600,00
    Accounts Payable €15600,00
    

    View Slide

84. @michieltcs
    Ledger Entry
    Aug 14 Inventory €15600,00
    Accounts Payable €15600,00
    Ledger Entry
    Aug 14 Inventory €16500,00
    Accounts Payable €16500,00
    

    View Slide

85. @michieltcs
    Ledger Entry
    Aug 14 Inventory €15600,00
    Accounts Payable €15600,00
    Ledger Entry
    Aug 14 Inventory €16500,00
    Accounts Payable €16500,00
    Ledger Correction Entry
    Aug 14 Inventory €900,00
    Accounts Payable €900,00
    

    View Slide

86. COMPENSATING ACTIONS
    class MoneyWithdrawn {

    String accountId;

    BigDecimal amount;

    }
    class WithdrawalRolledBack {

    String accountId;

    BigDecimal amount;

    }
    Typo: too much withdrawn!
    

    View Slide

87. COMPENSATING ACTIONS
    class AccountOpened {

    String accountId;

    String accountNumber;

    }
    class DuplicateAccountClosed {

    String accountId;

    }
    Duplicate account number!
    

    View Slide

88. GDPR?
    

    View Slide

89. STRATEGIES
    

    View Slide

90. ONLY REMOVE FROM
    PROJECTION?
    

    View Slide

91. LEGAL DEFENCE?
    

    View Slide

92. '
    .. adequate, relevant and limited to
    what is necessary in relation to the
    purposes for which they are processed
    (‘data minimisation’)
    -GDPR, Article 5
    

    View Slide

93. UPCASTING?
    

    View Slide

94. UPCASTING
    Event Store
    Event_V1
    Upcaster
    Event_V2
    Event Handler
    @michieltcs
    

    View Slide

95. UPCASTING
    Event Store
    Event_V1
    Upcaster
    Event_V2
    Event Handler
    @michieltcs
    Event_V2 = f(Event_V1)
    

    View Slide

96. UPCASTING
    Event Store
    Event_V1
    Upcaster
    Event_V2
    Event Handler
    @michieltcs
    Event_V2 = f(Event_V1)
    

    View Slide

97. DELETING EVENTS
    

    View Slide

98. DELETING EVENTS
    

    View Slide

99. MODIFYING EVENTS
    

    View Slide

100. MODIFYING EVENTS
     

     View Slide

101. COPY & FILTER
     

     View Slide

102. VERSIONED EVENT STORE
     

     View Slide

103. VERSIONED EVENT STORE
     events_v1
     [

     {

     "id": "12345678",

     "type": "AccountOpened",

     "aggregateType": "Account",

     "aggregateIdentifier": "1234",

     "sequenceNumber": 0,

     "payloadRevision": "1.0",

     "payload": { ... },

     "timestamp": ...

     ...

     },

     ...

     ]
     @michieltcs
     

     View Slide

104. COPY & REPLACE
     

     View Slide

105. VERSIONED EVENT STORE
     Loop over
     existing
     events
     Apply
     upcaster
     Add
     queued
     events
     Use new
     event store
     New events Queue
     @michieltcs
     

     View Slide

106. VERSIONED EVENT STORE
     events_v2
     [

     {

     "id": "12345678",

     "type": "AccountOpened",

     "aggregateType": "Account",

     "aggregateIdentifier": "1234",

     "sequenceNumber": 0,

     "payloadRevision": "2.0",

     "payload": { ... },

     "timestamp": ...

     ...

     },

     ...

     ]
     @michieltcs
     

     View Slide

107. STORE PII EXTERNALLY
     

     View Slide

108. STORE PII EXTERNALLY
     @michieltcs
     1 @Value
     2 public class AccountOpened {
     3 String accountId;
     4 String accountNumber;
     5 String name;
     6 }
     

     View Slide

109. STORE PII EXTERNALLY
     @michieltcs
     1 @Value
     2 public class AccountOpened {
     3 String accountId;
     4 }
     

     View Slide

110. STORE PII EXTERNALLY
     @michieltcs
     AccountOpened External Storage
     1 @Value
     2 public class AccountOpened {
     3 String accountId;
     4 }
     Account Id Account number Name
     1234 12345678 John Doe
     ... ... ...
     

     View Slide

111. STORE PII EXTERNALLY
     @michieltcs
     AccountOpened External Storage
     1 @Value
     2 public class AccountOpened {
     3 String accountId;
     4 }
     Account Id Account number Name
     1234 12345678 ANON
     ... ... ...
     

     View Slide

112. STORE PII EXTERNALLY
     @michieltcs
     AccountOpened External Storage
     1 @Value
     2 public class AccountOpened {
     3 String accountId;
     4 }
     Account Id Account number Name
     1234 12345678 ANON
     ... ... ...
     

     View Slide

113. CRYPTO ERASURE
     

     View Slide

114. ENCRYPT EVENTS
     

     View Slide

115. DECRYPT EVENTS
     

     View Slide

116. ENCRYPT FIELD VALUES
     

     View Slide

117. DECRYPT FIELD VALUES
     

     View Slide

118. ENCRYPTING EVENTS
     @michieltcs
     

     80f49161

     NL00ABNA012345678

     Foo

     Bar
     ...

     
     

     View Slide

119. ENCRYPTING EVENTS
     @michieltcs
     

     80f49161

     2dqjHkY8Mc8+cek4vs/9hzgkob4J3fZJNIJh2sAXlJ0=

     N5Y27vd0UbKo6FIu5c7QGQ==

     OSKrzfuuuayuUNXYS5YUug==
     ...

     
     

     View Slide

120. ENCRYPTING EVENTS
     Generate
     event
     Find
     / create
     encryption
     key
     Encrypt
     payload
     values
     Store

     event
     @michieltcs
     

     View Slide

121. DECRYPTING EVENTS
     Load

     event
     Find
     associated
     encryption
     key
     Decrypt
     payload
     values
     Process

     event
     @michieltcs
     

     View Slide

122. SHEDDING THE KEY
     Load

     event
     Find
     associated
     encryption
     key
     Decrypt
     payload
     values
     Process

     event
     @michieltcs
     X
     

     View Slide

123. AXON GDPR MODULE
     @michieltcs
     1 @Value
     2 public class AccountOpened {
     3 @DataSubjectId
     4 String accountId;
     5
     6 @PersonalData
     7 String accountNumberIban;
     8
     9 @PersonalData
     10 String firstName;
     11
     12 @PersonalData
     13 String lastName;

     14 }
     

     View Slide

124. KEY MANAGEMENT
     

     View Slide

125. PERFORMANCE
     

     View Slide

126. RE-ENCRYPT DATA AT REST
     

     View Slide

127. CLOSING WORDS
     

     View Slide

128. GDPR
     

     View Slide

129. CHALLENGES
     

     View Slide

130. FRAMEWORK SUPPORT
     

     View Slide

131. (IM)MUTABILITY
     

     View Slide

132. AUDIT TRAIL
     

     View Slide

133. DEMONSTRATING
     CONSENT
     

     View Slide

134. FUTURE?
     

     View Slide

135. THANK YOU!
     @michieltcs / michiel@michielrook.nl

     

     www.michielrook.nl
     

     View Slide