Slide 1

Slide 1 text

A presentation by @stuherbert
 for @GanbaroDigital Event-Sourcing & GDPR When Immutability Meets Reality

Slide 2

Slide 2 text

Industry veteran: architect, engineer, leader, manager, mentor F/OSS contributor since 1994 Talking and writing about PHP since 2004 Chief Software Archaeologist Building Quality @GanbaroDigital About Stuart

Slide 3

Slide 3 text

Follow me I do tweet a lot about non-tech stuff though :) @stuherbert

Slide 4

Slide 4 text

@GanbaroDigital ?? ?? Do you currently use event-sourcing?

Slide 5

Slide 5 text

@GanbaroDigital ?? ?? Are you planning on adopting event-sourcing?

Slide 6

Slide 6 text

@GanbaroDigital ?? ?? Do you currently work in a regulated industry?

Slide 7

Slide 7 text

@GanbaroDigital In This Talk 1. Event Sourcing 2. GDPR 3. How GDPR Impacts Event Sourcing 4. Summary

Slide 8

Slide 8 text

@GanbaroDigital Please ask questions as we go!

Slide 9

Slide 9 text

@GanbaroDigital Event Sourcing

Slide 10

Slide 10 text

@GanbaroDigital What Is It?

Slide 11

Slide 11 text

@GanbaroDigital Event Sourcing is a data architecture.

Slide 12

Slide 12 text

@GanbaroDigital All state changes are represented as events.

Slide 13

Slide 13 text

@GanbaroDigital “ An event is something that has happened.

Slide 14

Slide 14 text

@GanbaroDigital Some Example Events • User added item to basket • User completed basket checkout • User paid for order • Order shipped

Slide 15

Slide 15 text

@GanbaroDigital

Slide 16

Slide 16 text

@GanbaroDigital UI

Slide 17

Slide 17 text

@GanbaroDigital UI API

Slide 18

Slide 18 text

@GanbaroDigital Business Model & Data Model UI API

Slide 19

Slide 19 text

@GanbaroDigital So far, that looks like traditional software systems.

Slide 20

Slide 20 text

@GanbaroDigital In a traditional software system, the database holds the current state.

Slide 21

Slide 21 text

@GanbaroDigital Current state is the result of all the operations that have already happened.

Slide 22

Slide 22 text

@GanbaroDigital The database stores the result of what has happened. It doesn't store what has happened.

Slide 23

Slide 23 text

@GanbaroDigital Business Model & Data Model UI API

Slide 24

Slide 24 text

@GanbaroDigital Business Model & Data Model UI API Database

Slide 25

Slide 25 text

@GanbaroDigital Event Source systems store events in the database ... ... not the current state (and not the operations either).

Slide 26

Slide 26 text

@GanbaroDigital Business Model & Data Model UI API

Slide 27

Slide 27 text

@GanbaroDigital Business Model & Data Model UI API Event Store

Slide 28

Slide 28 text

@GanbaroDigital Current state isn't stored in the Event Store. It has to be built.

Slide 29

Slide 29 text

@GanbaroDigital Current state is built by playback of the stored events.

Slide 30

Slide 30 text

@GanbaroDigital Business Model & Data Model UI API Event Store

Slide 31

Slide 31 text

@GanbaroDigital Event Validation UI Event Store API Event Playback

Slide 32

Slide 32 text

@GanbaroDigital “ Event-Sourcing guarantees that you can build any state at any time through event playback.

Slide 33

Slide 33 text

@GanbaroDigital We're going to put that guarantee under a microscope later in this talk.

Slide 34

Slide 34 text

@GanbaroDigital Events are stored in, and played back from, the Event Store.

Slide 35

Slide 35 text

@GanbaroDigital Event Validation UI Event Store API Event Playback

Slide 36

Slide 36 text

@GanbaroDigital An Event Store is, ultimately, a database. It may be a general purpose RDBMS, a NoSQL datastore, or a specialist ESDB.

Slide 37

Slide 37 text

@GanbaroDigital The Event Store is subject to the same performance constraints that govern all databases.

Slide 38

Slide 38 text

@GanbaroDigital Performance Constraints • IOPS in production • Network bandwidth & latency • Concurrency • Maintenance operations

Slide 39

Slide 39 text

@GanbaroDigital Performance Constraints • IOPS in production • Network bandwidth & latency • Concurrency • Maintenance operations

Slide 40

Slide 40 text

@GanbaroDigital Performance Constraints • IOPS in production • Network bandwidth & latency • Concurrency • Maintenance operations

Slide 41

Slide 41 text

@GanbaroDigital Performance Constraints • IOPS in production • Network bandwidth & latency • Concurrency • Maintenance operations

Slide 42

Slide 42 text

@GanbaroDigital One way to minimise these performance constraints is to use an append-only / log datastore.

Slide 43

Slide 43 text

@GanbaroDigital Append-only / log datastores can be immutable.

Slide 44

Slide 44 text

@GanbaroDigital Event playback is too slow, too expensive to use all the time.

Slide 45

Slide 45 text

@GanbaroDigital ES-CQRS To The Rescue

Slide 46

Slide 46 text

@GanbaroDigital Command Query Responsibility Separation

Slide 47

Slide 47 text

@GanbaroDigital Command Query Responsibility Separation

Slide 48

Slide 48 text

@GanbaroDigital Command Query Responsibility Separation

Slide 49

Slide 49 text

@GanbaroDigital Command Query Responsibility Separation

Slide 50

Slide 50 text

@GanbaroDigital CQRS is a code architecture.

Slide 51

Slide 51 text

@GanbaroDigital CQRS separates read operations from create, update & delete operations.

Slide 52

Slide 52 text

@GanbaroDigital Reads and writes operate on separate business models.

Slide 53

Slide 53 text

@GanbaroDigital https://martinfowler.com/bliki/CQRS.html

Slide 54

Slide 54 text

@GanbaroDigital CQRS can also be a data architecture.

Slide 55

Slide 55 text

@GanbaroDigital Your reads can be against from a different datastore.

Slide 56

Slide 56 text

@GanbaroDigital Your read datastore can have a different data model to your write datastore.

Slide 57

Slide 57 text

@GanbaroDigital That's where ES-CQRS comes in.

Slide 58

Slide 58 text

@GanbaroDigital Event Sourcing is a data architecture. ES-CQRS is a data architecture too.

Slide 59

Slide 59 text

@GanbaroDigital Event playback is too slow, too expensive to use all the time.

Slide 60

Slide 60 text

@GanbaroDigital Speed up operations by caching current state in a datastore.

Slide 61

Slide 61 text

@GanbaroDigital Event Validation UI Event Store API Event Playback

Slide 62

Slide 62 text

@GanbaroDigital Event Validation UI API Event Playback Projection Cache Event Store Cache Lookups

Slide 63

Slide 63 text

@GanbaroDigital “ Event-Sourcing guarantees that you can build any state at any time through event playback.

Slide 64

Slide 64 text

@GanbaroDigital If the Projection Cache is lost, it can be rebuilt by playing back the events from the Event Store.

Slide 65

Slide 65 text

@GanbaroDigital Event Validation UI API Event Playback Projection Cache Event Store Cache Lookups

Slide 66

Slide 66 text

@GanbaroDigital Event Validation UI API Event Playback Projection Cache Event Store Cache Lookups ✗

Slide 67

Slide 67 text

@GanbaroDigital The Projection Cache is built using code that changes over time.

Slide 68

Slide 68 text

@GanbaroDigital Event Validation UI API Event Playback Projection Cache Event Store Cache Lookups

Slide 69

Slide 69 text

@GanbaroDigital Event Validation UI API Event Playback Projection Cache Event Store Cache Lookups

Slide 70

Slide 70 text

@GanbaroDigital “ Event-Sourcing guarantees that you can build any state at any time through event playback.

Slide 71

Slide 71 text

@GanbaroDigital To rebuild state for any moment in time, you need to know which version of the code was applied to each event.

Slide 72

Slide 72 text

@GanbaroDigital Snapshots of the Projection Cache are used to reduce this burden.

Slide 73

Slide 73 text

@GanbaroDigital Snapshots contain projections at a point in time.

Slide 74

Slide 74 text

@GanbaroDigital tl;dr

Slide 75

Slide 75 text

@GanbaroDigital An ES-CQRS system stores events as its primary data, not state.

Slide 76

Slide 76 text

@GanbaroDigital An ES-CQRS system builds state by playing back events.

Slide 77

Slide 77 text

@GanbaroDigital Some of these events will hold personal data.

Slide 78

Slide 78 text

@GanbaroDigital In ES-CQRS, personal data is stored in: - Event Store - Projection Cache - Snapshots

Slide 79

Slide 79 text

@GanbaroDigital Event Stores may be immutable.

Slide 80

Slide 80 text

@GanbaroDigital The Projection Cache is built using code that changes over time.

Slide 81

Slide 81 text

@GanbaroDigital Snapshots contain projections at a point in time.

Slide 82

Slide 82 text

@GanbaroDigital GDPR

Slide 83

Slide 83 text

@GanbaroDigital What Is GDPR?

Slide 84

Slide 84 text

@GanbaroDigital General
 Data Protection Regulation

Slide 85

Slide 85 text

@GanbaroDigital General
 Data Protection Regulation

Slide 86

Slide 86 text

@GanbaroDigital General
 Data Protection Regulation

Slide 87

Slide 87 text

@GanbaroDigital General
 Data Protection Regulation

Slide 88

Slide 88 text

@GanbaroDigital It came into effect May 25th 2018.

Slide 89

Slide 89 text

@GanbaroDigital In the UK, it was enshrined in law by the Data Protection Act 2018.

Slide 90

Slide 90 text

@GanbaroDigital In-scope: the personal data of all European Union citizens WORLDWIDE

Slide 91

Slide 91 text

@GanbaroDigital For many developers, GDPR will be the first time they have worked in a regulated environment.

Slide 92

Slide 92 text

@GanbaroDigital “ GDPR is the beginning of the end of the wild, wild west of unregulated software development.

Slide 93

Slide 93 text

@GanbaroDigital GDPR applies to free / open-source software too. You can't get around that in your LICENSE.md file.

Slide 94

Slide 94 text

@GanbaroDigital https://github.com/webdevlaw/open-source-privacy- standards

Slide 95

Slide 95 text

@GanbaroDigital Here are just some* of the requirements that GDPR and DPA 2018 place on data processing. * IANAL etc etc

Slide 96

Slide 96 text

@GanbaroDigital https://ico.org.uk/for-organisations/guide-to-data-protection/ guide-to-the-general-data-protection-regulation-gdpr/

Slide 97

Slide 97 text

@GanbaroDigital Breaking Down GDPR • Obligations on Organisations • Rights of Individuals

Slide 98

Slide 98 text

@GanbaroDigital Breaking Down GDPR • Obligations on Organisations • Rights of Individuals

Slide 99

Slide 99 text

@GanbaroDigital Obligations on Organisations

Slide 100

Slide 100 text

@GanbaroDigital It is illegal to hold personal data without a lawful basis.

Slide 101

Slide 101 text

@GanbaroDigital Identify the lawful basis for each piece of personal data.

Slide 102

Slide 102 text

@GanbaroDigital Maintain records of personal data.

Slide 103

Slide 103 text

@GanbaroDigital Maintain records of processing activities.

Slide 104

Slide 104 text

@GanbaroDigital Use personal data in a way that is fair.

Slide 105

Slide 105 text

@GanbaroDigital Consent is one lawful basis for storing personal data.

Slide 106

Slide 106 text

@GanbaroDigital Use personal data only for what you have explicit consent for.

Slide 107

Slide 107 text

@GanbaroDigital Obtain new consent if you want to use personal data for new purposes.

Slide 108

Slide 108 text

@GanbaroDigital Only collect personal data that you need for the processing you have consent for.

Slide 109

Slide 109 text

@GanbaroDigital Correct personal data that is factually inaccurate or misleading. Or delete it.

Slide 110

Slide 110 text

@GanbaroDigital You must not keep personal data any longer than required.

Slide 111

Slide 111 text

@GanbaroDigital Delete all personal data that you no longer need.

Slide 112

Slide 112 text

@GanbaroDigital The personal data must be erased from backups and archives too.

Slide 113

Slide 113 text

@GanbaroDigital Inform all third-parties that you have deleted personal data that you have passed to them. And tell the individual about those third-parties.

Slide 114

Slide 114 text

@GanbaroDigital Take appropriate security measures to protect personal data.

Slide 115

Slide 115 text

@GanbaroDigital Have evidence to demonstrate your compliance with GDPR.

Slide 116

Slide 116 text

@GanbaroDigital Rights of Individuals

Slide 117

Slide 117 text

@GanbaroDigital • Right to be informed • Right of access • Right to rectification • Right to erasure • Right to restrict processing • Right to data portability • Right to object • Rights related to automated processing Individual Rights

Slide 118

Slide 118 text

@GanbaroDigital Provide individuals with privacy information at the point of collection.

Slide 119

Slide 119 text

@GanbaroDigital If you obtain personal data from third-party sources, you must* provide individuals with your privacy information within 1 month.

Slide 120

Slide 120 text

@GanbaroDigital Provide subject access to personal data within 1 month of a request.

Slide 121

Slide 121 text

@GanbaroDigital Make sure a subject access request does not disclose personal data about anyone else.

Slide 122

Slide 122 text

@GanbaroDigital Correct factually inaccurate personal data within 1 month of a rectification request.

Slide 123

Slide 123 text

@GanbaroDigital Erase all personal data you can no longer hold within 1 month of an erasure request.

Slide 124

Slide 124 text

@GanbaroDigital The 'right to be forgotten' has stronger obligations if the personal data is about children.

Slide 125

Slide 125 text

@GanbaroDigital Do not use personal data that is subject to a processing restriction request. But you can still store it.

Slide 126

Slide 126 text

@GanbaroDigital Provide personal data in commonly-used machine-readable formats*.

Slide 127

Slide 127 text

@GanbaroDigital *but only when lawful basis is consent or by contract, and only when personal data is processed by automated means.

Slide 128

Slide 128 text

@GanbaroDigital We'll look at the Right to Object in a moment.

Slide 129

Slide 129 text

@GanbaroDigital Provide individuals with information about solely-automated decision making.

Slide 130

Slide 130 text

@GanbaroDigital Provide individuals with the means to request human intervention.

Slide 131

Slide 131 text

@GanbaroDigital Provide individuals with the means to challenge solely-automated decisions.

Slide 132

Slide 132 text

@GanbaroDigital Perform regular checks to ensure solely-automated decisions are working as intended.

Slide 133

Slide 133 text

@GanbaroDigital Exemptions

Slide 134

Slide 134 text

@GanbaroDigital Individuals have the right to object about the data held and how it is being used.

Slide 135

Slide 135 text

@GanbaroDigital https://ico.org.uk/for-organisations/guide-to-data-protection/ guide-to-the-general-data-protection-regulation-gdpr/ individual-rights/right-to-object/

Slide 136

Slide 136 text

@GanbaroDigital But wait, there's more!

Slide 137

Slide 137 text

@GanbaroDigital In-scope: the personal data of all European citizens WORLDWIDE

Slide 138

Slide 138 text

@GanbaroDigital In-scope: the personal data* of all European citizens WORLDWIDE

Slide 139

Slide 139 text

@GanbaroDigital * there are exemptions

Slide 140

Slide 140 text

@GanbaroDigital The list of exemptions was defined by the Data Protection Act 2018.

Slide 141

Slide 141 text

@GanbaroDigital https://ico.org.uk/for-organisations/guide-to-data-protection/ guide-to-the-general-data-protection-regulation-gdpr/ exemptions/

Slide 142

Slide 142 text

@GanbaroDigital Those are just the UK's exemptions. Each EU28 nation will have its own list.

Slide 143

Slide 143 text

@GanbaroDigital “ GDPR is a complex regulatory framework. Obtain, and follow, qualified advice.

Slide 144

Slide 144 text

@GanbaroDigital We've looked at Event Sourcing. We've looked at GDPR (and the DPA 2018).

Slide 145

Slide 145 text

@GanbaroDigital ?? ?? What happens when immutability meets the reality of personal data regulation?

Slide 146

Slide 146 text

@GanbaroDigital How GDPR Impacts ES-CQRS

Slide 147

Slide 147 text

@GanbaroDigital Required Capabilities

Slide 148

Slide 148 text

@GanbaroDigital ?? ?? What are the high-level requirements for GDPR compliance?

Slide 149

Slide 149 text

@GanbaroDigital Here's my current list.

Slide 150

Slide 150 text

@GanbaroDigital 'When' / 'if' is a separate topic for you and your legal advice.

Slide 151

Slide 151 text

@GanbaroDigital I am sure that this is not an exhaustive list!

Slide 152

Slide 152 text

@GanbaroDigital GDPR enforcement will identify gaps in the requirements and clarify acceptable practices.

Slide 153

Slide 153 text

@GanbaroDigital You should also assume that future legislation will change the requirements too.

Slide 154

Slide 154 text

@GanbaroDigital Requirement: You must store all personal data securely.

Slide 155

Slide 155 text

@GanbaroDigital Requirement: You must be able to trace all personal data back to a lawful purpose.

Slide 156

Slide 156 text

@GanbaroDigital Requirement: You must be able to trace all personal data back to a lawful purpose for each processing use.

Slide 157

Slide 157 text

@GanbaroDigital Implies: You may need to track which items of personal data have been used for each piece of processing.

Slide 158

Slide 158 text

@GanbaroDigital Requirement: You must be able to retrieve all personal data about any individual.

Slide 159

Slide 159 text

@GanbaroDigital Requirement: You must be able to update any piece of personal data.

Slide 160

Slide 160 text

@GanbaroDigital Requirement: You must be able to drop any piece of personal data. ... as if you never held it in the first place.

Slide 161

Slide 161 text

@GanbaroDigital Requirement: You must be able to remove personal data from everywhere (inc backups and archives).

Slide 162

Slide 162 text

@GanbaroDigital Requirement: You must be able to avoid processing personal data that you already have.

Slide 163

Slide 163 text

@GanbaroDigital Requirement: You must be able to review any solely-automated decision.

Slide 164

Slide 164 text

@GanbaroDigital Implies: You may need to track which items of personal data have been used for each piece of processing.

Slide 165

Slide 165 text

@GanbaroDigital Requirement: You must be able to override any solely-automated decision.

Slide 166

Slide 166 text

@GanbaroDigital None of these requirements are unique to ES-CQRS systems.

Slide 167

Slide 167 text

@GanbaroDigital ES-CQRS systems have unique challenges to overcome.

Slide 168

Slide 168 text

@GanbaroDigital ?? ?? What happens when immutability meets the reality of personal data regulation?

Slide 169

Slide 169 text

@GanbaroDigital The Problem is State

Slide 170

Slide 170 text

@GanbaroDigital In a traditional system, many of these GDPR requirements are met by amending state.

Slide 171

Slide 171 text

@GanbaroDigital An ES-CQRS system stores events as its primary data, not state.

Slide 172

Slide 172 text

@GanbaroDigital An ES-CQRS system builds state by playing back events.

Slide 173

Slide 173 text

@GanbaroDigital Some of these events will hold personal data.

Slide 174

Slide 174 text

@GanbaroDigital The interaction of events can be complex.

Slide 175

Slide 175 text

@GanbaroDigital In ES-CQRS, personal data is stored in: - Event Store - Projection Cache - Snapshots

Slide 176

Slide 176 text

@GanbaroDigital Here's the GDRP requirements that uniquely challenge ES-CQRS, and some questions to consider when adopting.

Slide 177

Slide 177 text

@GanbaroDigital Requirement: You must store all personal data securely.

Slide 178

Slide 178 text

@GanbaroDigital ?? ?? Does a specialist Event Store meet this requirement?

Slide 179

Slide 179 text

@GanbaroDigital ?? ?? Do the Projection Cache and Snapshot storage meet this requirement?

Slide 180

Slide 180 text

@GanbaroDigital Requirement: You must be able to update any piece of personal data.

Slide 181

Slide 181 text

@GanbaroDigital ?? ?? How do you correct data in an append-only system?

Slide 182

Slide 182 text

@GanbaroDigital ?? ?? If you still have the incorrect data, does that meet this requirement?

Slide 183

Slide 183 text

@GanbaroDigital Requirement: You must be able to drop any piece of personal data. ... as if you never held it in the first place.

Slide 184

Slide 184 text

@GanbaroDigital ?? ?? Can you hard-delete personal data from your Event Store?

Slide 185

Slide 185 text

@GanbaroDigital ?? ?? Can you purge any piece of personal data from the Projection Cache and any snapshots?

Slide 186

Slide 186 text

@GanbaroDigital ?? ?? If you do so by rebuilding the Projection Cache, are you sure you won't change anyone else's personal data?

Slide 187

Slide 187 text

@GanbaroDigital Some Event Stores encrypt personal data, and "delete it" by throwing away the encryption keys.

Slide 188

Slide 188 text

@GanbaroDigital ?? ?? If you still have the data, but cannot read it today, does that prevent it being read in the future?

Slide 189

Slide 189 text

@GanbaroDigital ... and don't forget ...

Slide 190

Slide 190 text

@GanbaroDigital Requirement: You must be able to remove personal data from everywhere (inc backups and archives).

Slide 191

Slide 191 text

@GanbaroDigital Requirement: You must be able to avoid processing personal data that you already have.

Slide 192

Slide 192 text

@GanbaroDigital ?? ?? If you rebuild state for an earlier time, how do you honour processing restrictions?

Slide 193

Slide 193 text

@GanbaroDigital ?? ?? How do you ensure processing restrictions do not change anyone else's personal data after a projection rebuild?

Slide 194

Slide 194 text

@GanbaroDigital Implies: You may need to track which items of personal data have been used for each piece of processing.

Slide 195

Slide 195 text

@GanbaroDigital ?? ?? Can you reproduce the state used at any point in time? With 100% accuracy?

Slide 196

Slide 196 text

@GanbaroDigital ?? ?? Do you need to archive state whenever it is used?

Slide 197

Slide 197 text

@GanbaroDigital ?? ?? Can you have the benefits of Event Sourcing and be GDPR-compliant? And is it worth it?

Slide 198

Slide 198 text

@GanbaroDigital Summing Up

Slide 199

Slide 199 text

@GanbaroDigital “GDPR is foundational. Compliance touches every aspect of how your business works.

Slide 200

Slide 200 text

@GanbaroDigital “ GDPR is a complex regulatory framework. Obtain, and follow, qualified advice.

Slide 201

Slide 201 text

@GanbaroDigital We're in the early stage of GDPR enforcement. Enforcement actions (or inaction!) will shape future advice.

Slide 202

Slide 202 text

@GanbaroDigital GDPR and immutability appear to be FUNDAMENTALLY incompatible.

Slide 203

Slide 203 text

@GanbaroDigital As a CTO, I would not adopt any framework / approach that relies on immutability if it will store personal data.

Slide 204

Slide 204 text

@GanbaroDigital If you are going to adopt Event Sourcing ...

Slide 205

Slide 205 text

@GanbaroDigital When evaluating an ES framework, ask the question: where are the docs on how to achieve GDPR compliance?

Slide 206

Slide 206 text

@GanbaroDigital When evaluating an ES framework, ask the question: where is the legal advice that it is GDPR compliant?

Slide 207

Slide 207 text

@GanbaroDigital In an ES workshop, ask the question: how do you achieve GDPR compliance using what is being taught?

Slide 208

Slide 208 text

@GanbaroDigital In an ES workshop, ask the question: where is the legal advice that the approach being taught achieves GDPR compliance?

Slide 209

Slide 209 text

@GanbaroDigital Your organisation is legally liable for GDPR compliance, not the ES framework, not the ES consultant.

Slide 210

Slide 210 text

@GanbaroDigital

Slide 211

Slide 211 text

Thank You How Can We Help You? A presentation by @stuherbert
 for @GanbaroDigital