ES-CQRS and GDPR: When Immutability Meets Reality

A presentation by @stuherbert  for @GanbaroDigital Event-Sourcing & GDPR When
Immutability Meets Reality

Industry veteran: architect, engineer, leader, manager, mentor F/OSS contributor since
1994 Talking and writing about PHP since 2004 Chief Software Archaeologist Building Quality @GanbaroDigital About Stuart

Follow me I do tweet a lot about non-tech stuff
though :) @stuherbert

@GanbaroDigital ?? ?? Do you currently use event-sourcing?

@GanbaroDigital ?? ?? Are you planning on adopting event-sourcing?

@GanbaroDigital ?? ?? Do you currently work in a regulated
industry?

@GanbaroDigital In This Talk 1. Event Sourcing 2. GDPR 3.
How GDPR Impacts Event Sourcing 4. Summary

@GanbaroDigital Please ask questions as we go!

@GanbaroDigital Event Sourcing

@GanbaroDigital What Is It?

@GanbaroDigital Event Sourcing is a data architecture.

@GanbaroDigital All state changes are represented as events.

@GanbaroDigital “ An event is something that has happened.

@GanbaroDigital Some Example Events • User added item to basket
• User completed basket checkout • User paid for order • Order shipped

@GanbaroDigital

@GanbaroDigital UI

@GanbaroDigital UI API

@GanbaroDigital Business Model & Data Model UI API

@GanbaroDigital So far, that looks like traditional software systems.

@GanbaroDigital In a traditional software system, the database holds the
current state.

@GanbaroDigital Current state is the result of all the operations
that have already happened.

@GanbaroDigital The database stores the result of what has happened.
It doesn't store what has happened.

@GanbaroDigital Business Model & Data Model UI API Database

@GanbaroDigital Event Source systems store events in the database ...
... not the current state (and not the operations either).

@GanbaroDigital Business Model & Data Model UI API Event Store

@GanbaroDigital Current state isn't stored in the Event Store. It
has to be built.

@GanbaroDigital Current state is built by playback of the stored
events.

@GanbaroDigital Business Model & Data Model UI API Event Store

@GanbaroDigital Event Validation UI Event Store API Event Playback

@GanbaroDigital “ Event-Sourcing guarantees that you can build any state
at any time through event playback.

@GanbaroDigital We're going to put that guarantee under a microscope
later in this talk.

@GanbaroDigital Events are stored in, and played back from, the
Event Store.

@GanbaroDigital An Event Store is, ultimately, a database. It may
be a general purpose RDBMS, a NoSQL datastore, or a specialist ESDB.

@GanbaroDigital The Event Store is subject to the same performance
constraints that govern all databases.

@GanbaroDigital Performance Constraints • IOPS in production • Network bandwidth
& latency • Concurrency • Maintenance operations

@GanbaroDigital One way to minimise these performance constraints is to
use an append-only / log datastore.

@GanbaroDigital Append-only / log datastores can be immutable.

@GanbaroDigital Event playback is too slow, too expensive to use
all the time.

@GanbaroDigital ES-CQRS To The Rescue

@GanbaroDigital Command Query Responsibility Separation

@GanbaroDigital CQRS is a code architecture.

@GanbaroDigital CQRS separates read operations from create, update & delete
operations.

@GanbaroDigital Reads and writes operate on separate business models.

@GanbaroDigital https://martinfowler.com/bliki/CQRS.html

@GanbaroDigital CQRS can also be a data architecture.

@GanbaroDigital Your reads can be against from a different datastore.

@GanbaroDigital Your read datastore can have a different data model
to your write datastore.

@GanbaroDigital That's where ES-CQRS comes in.

@GanbaroDigital Event Sourcing is a data architecture. ES-CQRS is a
data architecture too.

@GanbaroDigital Event playback is too slow, too expensive to use
all the time.

@GanbaroDigital Speed up operations by caching current state in a
datastore.

@GanbaroDigital Event Validation UI API Event Playback Projection Cache Event
Store Cache Lookups

@GanbaroDigital If the Projection Cache is lost, it can be
rebuilt by playing back the events from the Event Store.

Store Cache Lookups

Store Cache Lookups ✗

@GanbaroDigital The Projection Cache is built using code that changes
over time.

Store Cache Lookups

@GanbaroDigital To rebuild state for any moment in time, you
need to know which version of the code was applied to each event.

@GanbaroDigital Snapshots of the Projection Cache are used to reduce
this burden.

@GanbaroDigital Snapshots contain projections at a point in time.

@GanbaroDigital tl;dr

@GanbaroDigital An ES-CQRS system stores events as its primary data,
not state.

@GanbaroDigital An ES-CQRS system builds state by playing back events.

@GanbaroDigital Some of these events will hold personal data.

@GanbaroDigital In ES-CQRS, personal data is stored in: - Event
Store - Projection Cache - Snapshots

@GanbaroDigital Event Stores may be immutable.

@GanbaroDigital The Projection Cache is built using code that changes
over time.

@GanbaroDigital Snapshots contain projections at a point in time.

@GanbaroDigital GDPR

@GanbaroDigital What Is GDPR?

@GanbaroDigital General  Data Protection Regulation

@GanbaroDigital It came into effect May 25th 2018.

@GanbaroDigital In the UK, it was enshrined in law by
the Data Protection Act 2018.

@GanbaroDigital In-scope: the personal data of all European Union citizens
WORLDWIDE

@GanbaroDigital For many developers, GDPR will be the ﬁrst time
they have worked in a regulated environment.

@GanbaroDigital “ GDPR is the beginning of the end of
the wild, wild west of unregulated software development.

@GanbaroDigital GDPR applies to free / open-source software too. You
can't get around that in your LICENSE.md ﬁle.

@GanbaroDigital https://github.com/webdevlaw/open-source-privacy- standards

@GanbaroDigital Here are just some* of the requirements that GDPR
and DPA 2018 place on data processing. * IANAL etc etc

@GanbaroDigital https://ico.org.uk/for-organisations/guide-to-data-protection/ guide-to-the-general-data-protection-regulation-gdpr/

@GanbaroDigital Breaking Down GDPR • Obligations on Organisations • Rights
of Individuals

@GanbaroDigital Obligations on Organisations

@GanbaroDigital It is illegal to hold personal data without a
lawful basis.

@GanbaroDigital Identify the lawful basis for each piece of personal
data.

@GanbaroDigital Maintain records of personal data.

@GanbaroDigital Maintain records of processing activities.

@GanbaroDigital Use personal data in a way that is fair.

@GanbaroDigital Consent is one lawful basis for storing personal data.

@GanbaroDigital Use personal data only for what you have explicit
consent for.

@GanbaroDigital Obtain new consent if you want to use personal
data for new purposes.

@GanbaroDigital Only collect personal data that you need for the
processing you have consent for.

@GanbaroDigital Correct personal data that is factually inaccurate or misleading.
Or delete it.

@GanbaroDigital You must not keep personal data any longer than
required.

@GanbaroDigital Delete all personal data that you no longer need.

@GanbaroDigital The personal data must be erased from backups and
archives too.

@GanbaroDigital Inform all third-parties that you have deleted personal data
that you have passed to them. And tell the individual about those third-parties.

@GanbaroDigital Take appropriate security measures to protect personal data.

@GanbaroDigital Have evidence to demonstrate your compliance with GDPR.

@GanbaroDigital Rights of Individuals

@GanbaroDigital • Right to be informed • Right of access
• Right to rectiﬁcation • Right to erasure • Right to restrict processing • Right to data portability • Right to object • Rights related to automated processing Individual Rights

@GanbaroDigital Provide individuals with privacy information at the point of
collection.

@GanbaroDigital If you obtain personal data from third-party sources, you
must* provide individuals with your privacy information within 1 month.

@GanbaroDigital Provide subject access to personal data within 1 month
of a request.

@GanbaroDigital Make sure a subject access request does not disclose
personal data about anyone else.

@GanbaroDigital Correct factually inaccurate personal data within 1 month of
a rectiﬁcation request.

@GanbaroDigital Erase all personal data you can no longer hold
within 1 month of an erasure request.

@GanbaroDigital The 'right to be forgotten' has stronger obligations if
the personal data is about children.

@GanbaroDigital Do not use personal data that is subject to
a processing restriction request. But you can still store it.

@GanbaroDigital Provide personal data in commonly-used machine-readable formats*.

@GanbaroDigital *but only when lawful basis is consent or by
contract, and only when personal data is processed by automated means.

@GanbaroDigital We'll look at the Right to Object in a
moment.

@GanbaroDigital Provide individuals with information about solely-automated decision making.

@GanbaroDigital Provide individuals with the means to request human intervention.

@GanbaroDigital Provide individuals with the means to challenge solely-automated decisions.

@GanbaroDigital Perform regular checks to ensure solely-automated decisions are working
as intended.

@GanbaroDigital Exemptions

@GanbaroDigital Individuals have the right to object about the data
held and how it is being used.

@GanbaroDigital https://ico.org.uk/for-organisations/guide-to-data-protection/ guide-to-the-general-data-protection-regulation-gdpr/ individual-rights/right-to-object/

@GanbaroDigital But wait, there's more!

@GanbaroDigital In-scope: the personal data of all European citizens WORLDWIDE

@GanbaroDigital In-scope: the personal data* of all European citizens WORLDWIDE

@GanbaroDigital * there are exemptions

@GanbaroDigital The list of exemptions was deﬁned by the Data
Protection Act 2018.

@GanbaroDigital https://ico.org.uk/for-organisations/guide-to-data-protection/ guide-to-the-general-data-protection-regulation-gdpr/ exemptions/

@GanbaroDigital Those are just the UK's exemptions. Each EU28 nation
will have its own list.

@GanbaroDigital “ GDPR is a complex regulatory framework. Obtain, and
follow, qualiﬁed advice.

@GanbaroDigital We've looked at Event Sourcing. We've looked at GDPR
(and the DPA 2018).

@GanbaroDigital ?? ?? What happens when immutability meets the reality
of personal data regulation?

@GanbaroDigital How GDPR Impacts ES-CQRS

@GanbaroDigital Required Capabilities

@GanbaroDigital ?? ?? What are the high-level requirements for GDPR
compliance?

@GanbaroDigital Here's my current list.

@GanbaroDigital 'When' / 'if' is a separate topic for you
and your legal advice.

@GanbaroDigital I am sure that this is not an exhaustive
list!

@GanbaroDigital GDPR enforcement will identify gaps in the requirements and
clarify acceptable practices.

@GanbaroDigital You should also assume that future legislation will change
the requirements too.

@GanbaroDigital Requirement: You must store all personal data securely.

@GanbaroDigital Requirement: You must be able to trace all personal
data back to a lawful purpose.

@GanbaroDigital Requirement: You must be able to trace all personal
data back to a lawful purpose for each processing use.

@GanbaroDigital Implies: You may need to track which items of
personal data have been used for each piece of processing.

@GanbaroDigital Requirement: You must be able to retrieve all personal
data about any individual.

@GanbaroDigital Requirement: You must be able to update any piece
of personal data.

@GanbaroDigital Requirement: You must be able to drop any piece
of personal data. ... as if you never held it in the ﬁrst place.

@GanbaroDigital Requirement: You must be able to remove personal data
from everywhere (inc backups and archives).

@GanbaroDigital Requirement: You must be able to avoid processing personal
data that you already have.

@GanbaroDigital Requirement: You must be able to review any solely-automated
decision.

@GanbaroDigital Requirement: You must be able to override any solely-automated
decision.

@GanbaroDigital None of these requirements are unique to ES-CQRS systems.

@GanbaroDigital ES-CQRS systems have unique challenges to overcome.

@GanbaroDigital ?? ?? What happens when immutability meets the reality
of personal data regulation?

@GanbaroDigital The Problem is State

@GanbaroDigital In a traditional system, many of these GDPR requirements
are met by amending state.

@GanbaroDigital An ES-CQRS system stores events as its primary data,
not state.

@GanbaroDigital An ES-CQRS system builds state by playing back events.

@GanbaroDigital Some of these events will hold personal data.

@GanbaroDigital The interaction of events can be complex.

@GanbaroDigital In ES-CQRS, personal data is stored in: - Event
Store - Projection Cache - Snapshots

@GanbaroDigital Here's the GDRP requirements that uniquely challenge ES-CQRS, and
some questions to consider when adopting.

@GanbaroDigital Requirement: You must store all personal data securely.

@GanbaroDigital ?? ?? Does a specialist Event Store meet this
requirement?

@GanbaroDigital ?? ?? Do the Projection Cache and Snapshot storage
meet this requirement?

@GanbaroDigital Requirement: You must be able to update any piece
of personal data.

@GanbaroDigital ?? ?? How do you correct data in an
append-only system?

@GanbaroDigital ?? ?? If you still have the incorrect data,
does that meet this requirement?

@GanbaroDigital Requirement: You must be able to drop any piece
of personal data. ... as if you never held it in the ﬁrst place.

@GanbaroDigital ?? ?? Can you hard-delete personal data from your
Event Store?

@GanbaroDigital ?? ?? Can you purge any piece of personal
data from the Projection Cache and any snapshots?

@GanbaroDigital ?? ?? If you do so by rebuilding the
Projection Cache, are you sure you won't change anyone else's personal data?

@GanbaroDigital Some Event Stores encrypt personal data, and "delete it"
by throwing away the encryption keys.

@GanbaroDigital ?? ?? If you still have the data, but
cannot read it today, does that prevent it being read in the future?

@GanbaroDigital ... and don't forget ...

@GanbaroDigital Requirement: You must be able to remove personal data
from everywhere (inc backups and archives).

@GanbaroDigital Requirement: You must be able to avoid processing personal
data that you already have.

@GanbaroDigital ?? ?? If you rebuild state for an earlier
time, how do you honour processing restrictions?

@GanbaroDigital ?? ?? How do you ensure processing restrictions do
not change anyone else's personal data after a projection rebuild?

@GanbaroDigital ?? ?? Can you reproduce the state used at
any point in time? With 100% accuracy?

@GanbaroDigital ?? ?? Do you need to archive state whenever
it is used?

@GanbaroDigital ?? ?? Can you have the beneﬁts of Event
Sourcing and be GDPR-compliant? And is it worth it?

@GanbaroDigital Summing Up

@GanbaroDigital “GDPR is foundational. Compliance touches every aspect of how
your business works.

@GanbaroDigital “ GDPR is a complex regulatory framework. Obtain, and
follow, qualiﬁed advice.

@GanbaroDigital We're in the early stage of GDPR enforcement. Enforcement
actions (or inaction!) will shape future advice.

@GanbaroDigital GDPR and immutability appear to be FUNDAMENTALLY incompatible.

@GanbaroDigital As a CTO, I would not adopt any framework
/ approach that relies on immutability if it will store personal data.

@GanbaroDigital If you are going to adopt Event Sourcing ...

@GanbaroDigital When evaluating an ES framework, ask the question: where
are the docs on how to achieve GDPR compliance?

@GanbaroDigital When evaluating an ES framework, ask the question: where
is the legal advice that it is GDPR compliant?

@GanbaroDigital In an ES workshop, ask the question: how do
you achieve GDPR compliance using what is being taught?

@GanbaroDigital In an ES workshop, ask the question: where is
the legal advice that the approach being taught achieves GDPR compliance?

@GanbaroDigital Your organisation is legally liable for GDPR compliance, not
the ES framework, not the ES consultant.

@GanbaroDigital

Thank You How Can We Help You? A presentation by
@stuherbert  for @GanbaroDigital

ES-CQRS and GDPR: When Immutability Meets Reality

ES-CQRS and GDPR: When Immutability Meets Reality

More Decks by Stuart Herbert

Other Decks in Programming

Featured

Transcript