Forget me, please? Event sourcing and the GDPR

Slide 1

Slide 1 text

FORGET ME PLEASE?  EVENT SOURCING & THE GDPR Michiel Rook - @michieltcs

Slide 2

Slide 2 text

DISCLAIMER: I AM NOT A LAWYER

Slide 3

Slide 3 text

GDPR

Slide 4

Slide 4 text

GENERAL DATA PROTECTION REGULATION

Slide 5

Slide 5 text

' Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (Data Protection Directive) -General Data Protection Regulation

Slide 6

Slide 6 text

A SHORT HISTORY

Slide 7

Slide 7 text

1995 Data Protection Directive

Slide 8

Slide 8 text

1995 Data Protection Directive 2012 GDPR proposal

Slide 9

Slide 9 text

1995 Data Protection Directive 2012 GDPR proposal 2016 GDPR adopted

Slide 10

Slide 10 text

1995 Data Protection Directive 2012 GDPR proposal 2016 GDPR adopted 25 May 2018 GDPR enforceable

Slide 11

Slide 11 text

REGULATION

Slide 12

Slide 12 text

PROTECTS EU CITIZENS

Slide 13

Slide 13 text

DATA PROTECTION ACT

Slide 14

Slide 14 text

BROAD & VAGUE

Slide 15

Slide 15 text

PRIVACY BY DESIGN

Slide 16

Slide 16 text

' The controller shall implement appropriate technical and organisational measures for ensuring that, by default, only personal data which are necessary for each speciﬁc purpose of the processing are processed. -GDPR, Article 25

Slide 17

Slide 17 text

DATA PROTECTION OFFICER

Slide 18

Slide 18 text

SUPERVISORY AUTHORITY

Slide 19

Slide 19 text

FINES

Slide 20

Slide 20 text

€20 MILLION OR 4% OF ANNUAL TURNOVER

Slide 21

Slide 21 text

YOU

Slide 22

Slide 22 text

RAISE YOUR HAND IF YOU HAVE

Slide 23

Slide 23 text

read CQRS / Event Sourcing theory RAISE YOUR HAND IF YOU HAVE

Slide 24

Slide 24 text

read CQRS / Event Sourcing theory followed a tutorial, built a hobby project RAISE YOUR HAND IF YOU HAVE

Slide 25

Slide 25 text

read CQRS / Event Sourcing theory followed a tutorial, built a hobby project used it in production RAISE YOUR HAND IF YOU HAVE

Slide 26

Slide 26 text

Axon Framework Spring Boot

Slide 27

Slide 27 text

QUICK RECAP CQRS + EVENT SOURCING

Slide 28

Slide 28 text

CQRS

Slide 29

Slide 29 text

COMMAND QUERY RESPONSIBILITY SEGREGATION

Slide 30

Slide 30 text

STORAGE SIDE  VS.  QUERY SIDE

Slide 31

Slide 31 text

UI @michieltcs

Slide 32

Slide 32 text

Domain UI Command commands Aggregates @michieltcs

Slide 33

Slide 33 text

Domain UI Command Repository Event Store commands events Aggregates @michieltcs

Slide 34

Slide 34 text

Domain UI Event Bus Event Handlers Command Repository Database Database Event Store commands events events Aggregates @michieltcs

Slide 35

Slide 35 text

Domain UI Event Bus Event Handlers Command Repository Data Layer Database Database Event Store commands events events queries DTOs Aggregates @michieltcs

Slide 36

Slide 36 text

EVENT SOURCING

Slide 37

Slide 37 text

' Event Sourcing ensures that all changes to application state are stored as a sequence of events. -Martin Fowler

Slide 38

Slide 38 text

ACTIVE RECORD VS. EVENT SOURCING Account Id Account number Balance 1234 12345678 50,00 ... ... ... Money Withdrawn Account Id 1234 Amount 50,00 Money Deposited Account Id 1234 Amount 100,00 Account Opened Account Id 1234 Account number 12345678 @michieltcs

Slide 39

Slide 39 text

COMMANDS TO EVENTS Deposit Money Account Id 1234 Amount 100,00 @michieltcs 1 @Value 2 public class DepositMoney { 3 @TargetAggregateIdentifier 4 String accountId; 5 BigDecimal amount; 6 }

Slide 40

Slide 40 text

COMMANDS TO EVENTS Deposit Money Account Id 1234 Amount 100,00 command  handler @michieltcs 1 @CommandHandler 2 public void depositMoney(DepositMoney command) { 3 apply(new MoneyDeposited( 4 command.getAccountId(), 5 command.getAmount(), 6 ZonedDateTime.now())); 7 }

Slide 41

Slide 41 text

COMMANDS TO EVENTS Deposit Money Account Id 1234 Amount 100,00 Money Deposited Account Id 1234 Amount 100,00 command  handler @michieltcs 1 @Value 2 public class MoneyDeposited { 3 String accountId; 4 BigDecimal amount; 5 ZonedDateTime timestamp; 6 }

Slide 42

Slide 42 text

AGGREGATES @michieltcs an Aggregate handles Commands and generates Events based on the current state

Slide 43

Slide 43 text

AGGREGATES @michieltcs 1 class BankAccount { 2 @AggregateIdentifier 3 private String accountId; 4 private String accountNumber; 5 private BigDecimal balance; 6 7 // ... 8 @EventHandler 9 public void accountOpened(AccountOpened event) { 10 this.accountId = event.getAccountId(); 11 this.accountNumber = event.getAccountNumber(); 12 this.balance = BigDecimal.valueOf(0); 13 } 14 15 @EventHandler 16 public void moneyDeposited(MoneyDeposited event) { 17 this.balance = this.balance.add(event.getAmount()); 18 } 19 }

Slide 44

Slide 44 text

AGGREGATE STATE Account number Balance 12345678 0,00 Account number Balance 12345678 100,00 Account number Balance 12345678 50,00 event  handler event  handler event  handler @michieltcs Money Withdrawn Account Id 1234 Amount 50,00 Money Deposited Account Id 1234 Amount 100,00 Account Opened Account Id 1234 Account number 12345678

Slide 45

Slide 45 text

VALIDATING COMMANDS @michieltcs 1 @CommandHandler 2 public void withdrawMoney(WithdrawMoney command) throws 3 OverdraftDetectedException { 4 if (balance.compareTo(command.getAmount()) >= 0) { 5 apply(new MoneyWithdrawn( 6 command.getAccountId(), 7 command.getAmount(), 8 ZonedDateTime.now())); 9 } else { 10 throw new OverdraftDetectedException(accountNumber, balance, command. 11 getAmount()); 12 } 13 }

Slide 46

Slide 46 text

TESTING AGGREGATES @michieltcs 1 public class BankAccountTest { 2 private FixtureConfiguration fixture; 3 4 @Before 5 public void createFixture() { 6 fixture = new AggregateTestFixture<>(BankAccount.class); 7 } 8 9 @Test 10 public void noOverdraftsOnEmptyAccount() { 11 fixture.given(new AccountOpened(ACCOUNT_ID, ACCOUNT_NUMBER)) 12 .when(new WithdrawMoney(ACCOUNT_ID, new BigDecimal(20))) 13 .expectException(OverdraftDetectedException.class); 14 } 15 16 private final static String ACCOUNT_ID = "accountId"; 17 private final static String ACCOUNT_NUMBER = "accountNumber"; 18 }

Slide 47

Slide 47 text

EVENT SOURCING  & GDPR

Slide 48

Slide 48 text

CONSENT

Slide 49

Slide 49 text

' Where processing is based on consent, the controller shall be able to demonstrate that the data subject has consented to processing of his or her personal data. -GDPR, Article 7

Slide 50

Slide 50 text

REGISTERING CONSENT

Slide 51

Slide 51 text

' ...the request for consent shall be presented in a manner which is clearly distinguishable from the other matters... -GDPR, Article 7

Slide 52

Slide 52 text

REVOKING CONSENT

Slide 53

Slide 53 text

' The data subject shall have the right to withdraw his or her consent at any time. ... It shall be as easy to withdraw as to give consent. -GDPR, Article 7

Slide 54

Slide 54 text

WHY USE EVENT SOURCING / CQRS?

Slide 55

Slide 55 text

CAPTURE INTENT

Slide 56

Slide 56 text

DEMONSTRATING CONSENT

Slide 57

Slide 57 text

EVENT LOG  AS AUDIT LOG

Slide 58

Slide 58 text

NEW READ MODELS

Slide 59

Slide 59 text

EASIER DEBUGGING

Slide 60

Slide 60 text

EVENT LOG AS AUDIT LOG @michieltcs ConsentedToNewsletters

Slide 61

Slide 61 text

EVENT LOG AS AUDIT LOG @michieltcs ConsentedToNewsletters ConsentedToDataGathering

Slide 62

Slide 62 text

EVENT LOG AS AUDIT LOG @michieltcs ConsentedToNewsletters ConsentedToDataGathering RevokedConsentToNewsletters

Slide 63

Slide 63 text

"RIGHT TO ACCESS"

Slide 64

Slide 64 text

' The data subject shall have the right to obtain from the controller conﬁrmation as to whether or not personal data ... are being processed, and ... access to the personal data ... -GDPR, Article 15

Slide 65

Slide 65 text

"RIGHT TO ERASURE"

Slide 66

Slide 66 text

' The data subject shall have the right to obtain from the controller the erasure of personal data concerning him or her without undue delay -GDPR, Article 17

Slide 67

Slide 67 text

PERSONALLY IDENTIFIABLE INFORMATION

Slide 68

Slide 68 text

' ‘personal data’ means any information relating to an identified or identifiable natural person; an identifiable natural person is one who can be identified, directly or indirectly -GDPR, Article 4

Slide 69

Slide 69 text

GROUNDS

Slide 70

Slide 70 text

' .. the personal data are no longer necessary .. the data subject withdraws consent on which the processing is based -GDPR, Article 17

Slide 71

Slide 71 text

EXCEPTIONS

Slide 72

Slide 72 text

' .. to comply with a legal obligation .. for the establishment, exercise or defence of legal claims (*) -GDPR, Article 17

Slide 73

Slide 73 text

UNDUE DELAY

Slide 74

Slide 74 text

INFORM 3RD PARTIES

Slide 75

Slide 75 text

BACKUPS?

Slide 76

Slide 76 text

PROCESSING GDPR ART. 17 REQUESTS @michieltcs RightToErasureInvoked

Slide 77

Slide 77 text

PROCESSING GDPR ART. 17 REQUESTS @michieltcs RightToErasureInvoked Notify 3rd parties

Slide 78

Slide 78 text

PROCESSING GDPR ART. 17 REQUESTS @michieltcs RightToErasureInvoked Remove from read models Notify 3rd parties

Slide 79

Slide 79 text

PROCESSING GDPR ART. 17 REQUESTS @michieltcs RightToErasureInvoked Remove from event store Remove from read models Notify 3rd parties

Slide 80

Slide 80 text

PROCESSING GDPR ART. 17 REQUESTS @michieltcs RightToErasureInvoked Remove from event store ? Remove from read models Notify 3rd parties

Slide 81

Slide 81 text

IMMUTABLE EVENTS?

Slide 82

Slide 82 text

COMPENSATING ACTIONS

Slide 83

Slide 83 text

@michieltcs Ledger Entry Aug 14 Inventory 15600,00 Accounts Payable 15600,00

Slide 84

Slide 84 text

@michieltcs Ledger Entry Aug 14 Inventory 15600,00 Accounts Payable 15600,00 Ledger Entry Aug 14 Inventory 16500,00 Accounts Payable 16500,00

Slide 85

Slide 85 text

@michieltcs Ledger Entry Aug 14 Inventory 15600,00 Accounts Payable 15600,00 Ledger Entry Aug 14 Inventory 16500,00 Accounts Payable 16500,00 Ledger Correction Entry Aug 14 Inventory 900,00 Accounts Payable 900,00

Slide 86

Slide 86 text

COMPENSATING ACTIONS class MoneyWithdrawn {  String accountId;  BigDecimal amount;  } class WithdrawalRolledBack {  String accountId;  BigDecimal amount;  } Typo: too much withdrawn!

Slide 87

Slide 87 text

COMPENSATING ACTIONS class AccountOpened {  String accountId;  String accountNumber;  } class DuplicateAccountClosed {  String accountId;  } Duplicate account number!

Slide 88

Slide 88 text

GDPR?

Slide 89

Slide 89 text

STRATEGIES

Slide 90

Slide 90 text

ONLY REMOVE FROM PROJECTION?

Slide 91

Slide 91 text

LEGAL DEFENCE?

Slide 92

Slide 92 text

' .. adequate, relevant and limited to what is necessary in relation to the purposes for which they are processed (‘data minimisation’) -GDPR, Article 5

Slide 93

Slide 93 text

UPCASTING?

Slide 94

Slide 94 text

UPCASTING Event Store Event_V1 Upcaster Event_V2 Event Handler @michieltcs

Slide 95

Slide 95 text

UPCASTING Event Store Event_V1 Upcaster Event_V2 Event Handler @michieltcs Event_V2 = f(Event_V1)

Slide 96

Slide 96 text

UPCASTING Event Store Event_V1 Upcaster Event_V2 Event Handler @michieltcs Event_V2 = f(Event_V1)

Slide 97

Slide 97 text

DELETING EVENTS

Slide 98

Slide 98 text

DELETING EVENTS

Slide 99

Slide 99 text

MODIFYING EVENTS

Slide 100

Slide 100 text

MODIFYING EVENTS

Slide 101

Slide 101 text

COPY & FILTER

Slide 102

Slide 102 text

VERSIONED EVENT STORE

Slide 103

Slide 103 text

VERSIONED EVENT STORE events_v1 [  {  "id": "12345678",  "type": "AccountOpened",  "aggregateType": "Account",  "aggregateIdentifier": "1234",  "sequenceNumber": 0,  "payloadRevision": "1.0",  "payload": { ... },  "timestamp": ...  ...  },  ...  ] @michieltcs

Slide 104

Slide 104 text

COPY & REPLACE

Slide 105

Slide 105 text

VERSIONED EVENT STORE Loop over existing events Apply upcaster Add queued events Use new event store New events Queue @michieltcs

Slide 106

Slide 106 text

VERSIONED EVENT STORE events_v2 [  {  "id": "12345678",  "type": "AccountOpened",  "aggregateType": "Account",  "aggregateIdentifier": "1234",  "sequenceNumber": 0,  "payloadRevision": "2.0",  "payload": { ... },  "timestamp": ...  ...  },  ...  ] @michieltcs

Slide 107

Slide 107 text

STORE PII EXTERNALLY

Slide 108

Slide 108 text

STORE PII EXTERNALLY @michieltcs 1 @Value 2 public class AccountOpened { 3 String accountId; 4 String accountNumber; 5 String name; 6 }

Slide 109

Slide 109 text

STORE PII EXTERNALLY @michieltcs 1 @Value 2 public class AccountOpened { 3 String accountId; 4 }

Slide 110

Slide 110 text

STORE PII EXTERNALLY @michieltcs AccountOpened External Storage 1 @Value 2 public class AccountOpened { 3 String accountId; 4 } Account Id Account number Name 1234 12345678 John Doe ... ... ...

Slide 111

Slide 111 text

STORE PII EXTERNALLY @michieltcs AccountOpened External Storage 1 @Value 2 public class AccountOpened { 3 String accountId; 4 } Account Id Account number Name 1234 12345678 ANON ... ... ...

Slide 112

Slide 112 text

STORE PII EXTERNALLY @michieltcs AccountOpened External Storage 1 @Value 2 public class AccountOpened { 3 String accountId; 4 } Account Id Account number Name 1234 12345678 ANON ... ... ...

Slide 113

Slide 113 text

CRYPTO ERASURE

Slide 114

Slide 114 text

ENCRYPT EVENTS

Slide 115

Slide 115 text

DECRYPT EVENTS

Slide 116

Slide 116 text

ENCRYPT FIELD VALUES

Slide 117

Slide 117 text

DECRYPT FIELD VALUES

Slide 118

Slide 118 text

ENCRYPTING EVENTS @michieltcs   80f49161  NL00ABNA012345678  Foo  Bar ... 

Slide 119

Slide 119 text

ENCRYPTING EVENTS @michieltcs   80f49161  2dqjHkY8Mc8+cek4vs/9hzgkob4J3fZJNIJh2sAXlJ0=  N5Y27vd0UbKo6FIu5c7QGQ==  OSKrzfuuuayuUNXYS5YUug== ... 

Slide 120

Slide 120 text

ENCRYPTING EVENTS Generate event Find / create encryption key Encrypt payload values Store  event @michieltcs

Slide 121

Slide 121 text

DECRYPTING EVENTS Load  event Find associated encryption key Decrypt payload values Process  event @michieltcs

Slide 122

Slide 122 text

SHEDDING THE KEY Load  event Find associated encryption key Decrypt payload values Process  event @michieltcs X

Slide 123

Slide 123 text

AXON GDPR MODULE @michieltcs 1 @Value 2 public class AccountOpened { 3 @DataSubjectId 4 String accountId; 5 6 @PersonalData 7 String accountNumberIban; 8 9 @PersonalData 10 String firstName; 11 12 @PersonalData 13 String lastName;  14 }

Slide 124

Slide 124 text

KEY MANAGEMENT

Slide 125

Slide 125 text

PERFORMANCE

Slide 126

Slide 126 text

RE-ENCRYPT DATA AT REST

Slide 127

Slide 127 text

CLOSING WORDS

Slide 128

Slide 128 text

GDPR

Slide 129

Slide 129 text

CHALLENGES

Slide 130

Slide 130 text

FRAMEWORK SUPPORT

Slide 131

Slide 131 text

(IM)MUTABILITY

Slide 132

Slide 132 text

AUDIT TRAIL

Slide 133

Slide 133 text

DEMONSTRATING CONSENT

Slide 134

Slide 134 text

FUTURE?

Slide 135

Slide 135 text

THANK YOU! @michieltcs / michiel@michielrook.nl    www.michielrook.nl