Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ES-CQRS and GDPR: When Immutability Meets Reality

ES-CQRS and GDPR: When Immutability Meets Reality

The last couple of years has seen a surge of interest in Event Sourcing in the PHP community. This has coincided with the introduction of GDPR, the new data privacy regulations rolled out across all EU28 nations.

In this talk, Stuart will introduce you to the core principles of GDPR and the Data Protection Act 2018. He’ll look at what capabilities you need to add to your application/service to allow your business to comply with this landmark legislation. And then he’ll look at how these requirements do—or don’t—map into the Event Sourcing world.

Presented at PHP North East User Group on 19th March 2019.

2c1dc90ff7bf69097a151677624777d2?s=128

Stuart Herbert

March 19, 2019
Tweet

Transcript

  1. A presentation by @stuherbert
 for @GanbaroDigital Event-Sourcing & GDPR When

    Immutability Meets Reality
  2. Industry veteran: architect, engineer, leader, manager, mentor F/OSS contributor since

    1994 Talking and writing about PHP since 2004 Chief Software Archaeologist Building Quality @GanbaroDigital About Stuart
  3. Follow me I do tweet a lot about non-tech stuff

    though :) @stuherbert
  4. @GanbaroDigital ?? ?? Do you currently use event-sourcing?

  5. @GanbaroDigital ?? ?? Are you planning on adopting event-sourcing?

  6. @GanbaroDigital ?? ?? Do you currently work in a regulated

    industry?
  7. @GanbaroDigital In This Talk 1. Event Sourcing 2. GDPR 3.

    How GDPR Impacts Event Sourcing 4. Summary
  8. @GanbaroDigital Please ask questions as we go!

  9. @GanbaroDigital Event Sourcing

  10. @GanbaroDigital What Is It?

  11. @GanbaroDigital Event Sourcing is a data architecture.

  12. @GanbaroDigital All state changes are represented as events.

  13. @GanbaroDigital “ An event is something that has happened.

  14. @GanbaroDigital Some Example Events • User added item to basket

    • User completed basket checkout • User paid for order • Order shipped
  15. @GanbaroDigital

  16. @GanbaroDigital UI

  17. @GanbaroDigital UI API

  18. @GanbaroDigital Business Model & Data Model UI API

  19. @GanbaroDigital So far, that looks like traditional software systems.

  20. @GanbaroDigital In a traditional software system, the database holds the

    current state.
  21. @GanbaroDigital Current state is the result of all the operations

    that have already happened.
  22. @GanbaroDigital The database stores the result of what has happened.

    It doesn't store what has happened.
  23. @GanbaroDigital Business Model & Data Model UI API

  24. @GanbaroDigital Business Model & Data Model UI API Database

  25. @GanbaroDigital Event Source systems store events in the database ...

    ... not the current state (and not the operations either).
  26. @GanbaroDigital Business Model & Data Model UI API

  27. @GanbaroDigital Business Model & Data Model UI API Event Store

  28. @GanbaroDigital Current state isn't stored in the Event Store. It

    has to be built.
  29. @GanbaroDigital Current state is built by playback of the stored

    events.
  30. @GanbaroDigital Business Model & Data Model UI API Event Store

  31. @GanbaroDigital Event Validation UI Event Store API Event Playback

  32. @GanbaroDigital “ Event-Sourcing guarantees that you can build any state

    at any time through event playback.
  33. @GanbaroDigital We're going to put that guarantee under a microscope

    later in this talk.
  34. @GanbaroDigital Events are stored in, and played back from, the

    Event Store.
  35. @GanbaroDigital Event Validation UI Event Store API Event Playback

  36. @GanbaroDigital An Event Store is, ultimately, a database. It may

    be a general purpose RDBMS, a NoSQL datastore, or a specialist ESDB.
  37. @GanbaroDigital The Event Store is subject to the same performance

    constraints that govern all databases.
  38. @GanbaroDigital Performance Constraints • IOPS in production • Network bandwidth

    & latency • Concurrency • Maintenance operations
  39. @GanbaroDigital Performance Constraints • IOPS in production • Network bandwidth

    & latency • Concurrency • Maintenance operations
  40. @GanbaroDigital Performance Constraints • IOPS in production • Network bandwidth

    & latency • Concurrency • Maintenance operations
  41. @GanbaroDigital Performance Constraints • IOPS in production • Network bandwidth

    & latency • Concurrency • Maintenance operations
  42. @GanbaroDigital One way to minimise these performance constraints is to

    use an append-only / log datastore.
  43. @GanbaroDigital Append-only / log datastores can be immutable.

  44. @GanbaroDigital Event playback is too slow, too expensive to use

    all the time.
  45. @GanbaroDigital ES-CQRS To The Rescue

  46. @GanbaroDigital Command Query Responsibility Separation

  47. @GanbaroDigital Command Query Responsibility Separation

  48. @GanbaroDigital Command Query Responsibility Separation

  49. @GanbaroDigital Command Query Responsibility Separation

  50. @GanbaroDigital CQRS is a code architecture.

  51. @GanbaroDigital CQRS separates read operations from create, update & delete

    operations.
  52. @GanbaroDigital Reads and writes operate on separate business models.

  53. @GanbaroDigital https://martinfowler.com/bliki/CQRS.html

  54. @GanbaroDigital CQRS can also be a data architecture.

  55. @GanbaroDigital Your reads can be against from a different datastore.

  56. @GanbaroDigital Your read datastore can have a different data model

    to your write datastore.
  57. @GanbaroDigital That's where ES-CQRS comes in.

  58. @GanbaroDigital Event Sourcing is a data architecture. ES-CQRS is a

    data architecture too.
  59. @GanbaroDigital Event playback is too slow, too expensive to use

    all the time.
  60. @GanbaroDigital Speed up operations by caching current state in a

    datastore.
  61. @GanbaroDigital Event Validation UI Event Store API Event Playback

  62. @GanbaroDigital Event Validation UI API Event Playback Projection Cache Event

    Store Cache Lookups
  63. @GanbaroDigital “ Event-Sourcing guarantees that you can build any state

    at any time through event playback.
  64. @GanbaroDigital If the Projection Cache is lost, it can be

    rebuilt by playing back the events from the Event Store.
  65. @GanbaroDigital Event Validation UI API Event Playback Projection Cache Event

    Store Cache Lookups
  66. @GanbaroDigital Event Validation UI API Event Playback Projection Cache Event

    Store Cache Lookups ✗
  67. @GanbaroDigital The Projection Cache is built using code that changes

    over time.
  68. @GanbaroDigital Event Validation UI API Event Playback Projection Cache Event

    Store Cache Lookups
  69. @GanbaroDigital Event Validation UI API Event Playback Projection Cache Event

    Store Cache Lookups
  70. @GanbaroDigital “ Event-Sourcing guarantees that you can build any state

    at any time through event playback.
  71. @GanbaroDigital To rebuild state for any moment in time, you

    need to know which version of the code was applied to each event.
  72. @GanbaroDigital Snapshots of the Projection Cache are used to reduce

    this burden.
  73. @GanbaroDigital Snapshots contain projections at a point in time.

  74. @GanbaroDigital tl;dr

  75. @GanbaroDigital An ES-CQRS system stores events as its primary data,

    not state.
  76. @GanbaroDigital An ES-CQRS system builds state by playing back events.

  77. @GanbaroDigital Some of these events will hold personal data.

  78. @GanbaroDigital In ES-CQRS, personal data is stored in: - Event

    Store - Projection Cache - Snapshots
  79. @GanbaroDigital Event Stores may be immutable.

  80. @GanbaroDigital The Projection Cache is built using code that changes

    over time.
  81. @GanbaroDigital Snapshots contain projections at a point in time.

  82. @GanbaroDigital GDPR

  83. @GanbaroDigital What Is GDPR?

  84. @GanbaroDigital General
 Data Protection Regulation

  85. @GanbaroDigital General
 Data Protection Regulation

  86. @GanbaroDigital General
 Data Protection Regulation

  87. @GanbaroDigital General
 Data Protection Regulation

  88. @GanbaroDigital It came into effect May 25th 2018.

  89. @GanbaroDigital In the UK, it was enshrined in law by

    the Data Protection Act 2018.
  90. @GanbaroDigital In-scope: the personal data of all European Union citizens

    WORLDWIDE
  91. @GanbaroDigital For many developers, GDPR will be the first time

    they have worked in a regulated environment.
  92. @GanbaroDigital “ GDPR is the beginning of the end of

    the wild, wild west of unregulated software development.
  93. @GanbaroDigital GDPR applies to free / open-source software too. You

    can't get around that in your LICENSE.md file.
  94. @GanbaroDigital https://github.com/webdevlaw/open-source-privacy- standards

  95. @GanbaroDigital Here are just some* of the requirements that GDPR

    and DPA 2018 place on data processing. * IANAL etc etc
  96. @GanbaroDigital https://ico.org.uk/for-organisations/guide-to-data-protection/ guide-to-the-general-data-protection-regulation-gdpr/

  97. @GanbaroDigital Breaking Down GDPR • Obligations on Organisations • Rights

    of Individuals
  98. @GanbaroDigital Breaking Down GDPR • Obligations on Organisations • Rights

    of Individuals
  99. @GanbaroDigital Obligations on Organisations

  100. @GanbaroDigital It is illegal to hold personal data without a

    lawful basis.
  101. @GanbaroDigital Identify the lawful basis for each piece of personal

    data.
  102. @GanbaroDigital Maintain records of personal data.

  103. @GanbaroDigital Maintain records of processing activities.

  104. @GanbaroDigital Use personal data in a way that is fair.

  105. @GanbaroDigital Consent is one lawful basis for storing personal data.

  106. @GanbaroDigital Use personal data only for what you have explicit

    consent for.
  107. @GanbaroDigital Obtain new consent if you want to use personal

    data for new purposes.
  108. @GanbaroDigital Only collect personal data that you need for the

    processing you have consent for.
  109. @GanbaroDigital Correct personal data that is factually inaccurate or misleading.

    Or delete it.
  110. @GanbaroDigital You must not keep personal data any longer than

    required.
  111. @GanbaroDigital Delete all personal data that you no longer need.

  112. @GanbaroDigital The personal data must be erased from backups and

    archives too.
  113. @GanbaroDigital Inform all third-parties that you have deleted personal data

    that you have passed to them. And tell the individual about those third-parties.
  114. @GanbaroDigital Take appropriate security measures to protect personal data.

  115. @GanbaroDigital Have evidence to demonstrate your compliance with GDPR.

  116. @GanbaroDigital Rights of Individuals

  117. @GanbaroDigital • Right to be informed • Right of access

    • Right to rectification • Right to erasure • Right to restrict processing • Right to data portability • Right to object • Rights related to automated processing Individual Rights
  118. @GanbaroDigital Provide individuals with privacy information at the point of

    collection.
  119. @GanbaroDigital If you obtain personal data from third-party sources, you

    must* provide individuals with your privacy information within 1 month.
  120. @GanbaroDigital Provide subject access to personal data within 1 month

    of a request.
  121. @GanbaroDigital Make sure a subject access request does not disclose

    personal data about anyone else.
  122. @GanbaroDigital Correct factually inaccurate personal data within 1 month of

    a rectification request.
  123. @GanbaroDigital Erase all personal data you can no longer hold

    within 1 month of an erasure request.
  124. @GanbaroDigital The 'right to be forgotten' has stronger obligations if

    the personal data is about children.
  125. @GanbaroDigital Do not use personal data that is subject to

    a processing restriction request. But you can still store it.
  126. @GanbaroDigital Provide personal data in commonly-used machine-readable formats*.

  127. @GanbaroDigital *but only when lawful basis is consent or by

    contract, and only when personal data is processed by automated means.
  128. @GanbaroDigital We'll look at the Right to Object in a

    moment.
  129. @GanbaroDigital Provide individuals with information about solely-automated decision making.

  130. @GanbaroDigital Provide individuals with the means to request human intervention.

  131. @GanbaroDigital Provide individuals with the means to challenge solely-automated decisions.

  132. @GanbaroDigital Perform regular checks to ensure solely-automated decisions are working

    as intended.
  133. @GanbaroDigital Exemptions

  134. @GanbaroDigital Individuals have the right to object about the data

    held and how it is being used.
  135. @GanbaroDigital https://ico.org.uk/for-organisations/guide-to-data-protection/ guide-to-the-general-data-protection-regulation-gdpr/ individual-rights/right-to-object/

  136. @GanbaroDigital But wait, there's more!

  137. @GanbaroDigital In-scope: the personal data of all European citizens WORLDWIDE

  138. @GanbaroDigital In-scope: the personal data* of all European citizens WORLDWIDE

  139. @GanbaroDigital * there are exemptions

  140. @GanbaroDigital The list of exemptions was defined by the Data

    Protection Act 2018.
  141. @GanbaroDigital https://ico.org.uk/for-organisations/guide-to-data-protection/ guide-to-the-general-data-protection-regulation-gdpr/ exemptions/

  142. @GanbaroDigital Those are just the UK's exemptions. Each EU28 nation

    will have its own list.
  143. @GanbaroDigital “ GDPR is a complex regulatory framework. Obtain, and

    follow, qualified advice.
  144. @GanbaroDigital We've looked at Event Sourcing. We've looked at GDPR

    (and the DPA 2018).
  145. @GanbaroDigital ?? ?? What happens when immutability meets the reality

    of personal data regulation?
  146. @GanbaroDigital How GDPR Impacts ES-CQRS

  147. @GanbaroDigital Required Capabilities

  148. @GanbaroDigital ?? ?? What are the high-level requirements for GDPR

    compliance?
  149. @GanbaroDigital Here's my current list.

  150. @GanbaroDigital 'When' / 'if' is a separate topic for you

    and your legal advice.
  151. @GanbaroDigital I am sure that this is not an exhaustive

    list!
  152. @GanbaroDigital GDPR enforcement will identify gaps in the requirements and

    clarify acceptable practices.
  153. @GanbaroDigital You should also assume that future legislation will change

    the requirements too.
  154. @GanbaroDigital Requirement: You must store all personal data securely.

  155. @GanbaroDigital Requirement: You must be able to trace all personal

    data back to a lawful purpose.
  156. @GanbaroDigital Requirement: You must be able to trace all personal

    data back to a lawful purpose for each processing use.
  157. @GanbaroDigital Implies: You may need to track which items of

    personal data have been used for each piece of processing.
  158. @GanbaroDigital Requirement: You must be able to retrieve all personal

    data about any individual.
  159. @GanbaroDigital Requirement: You must be able to update any piece

    of personal data.
  160. @GanbaroDigital Requirement: You must be able to drop any piece

    of personal data. ... as if you never held it in the first place.
  161. @GanbaroDigital Requirement: You must be able to remove personal data

    from everywhere (inc backups and archives).
  162. @GanbaroDigital Requirement: You must be able to avoid processing personal

    data that you already have.
  163. @GanbaroDigital Requirement: You must be able to review any solely-automated

    decision.
  164. @GanbaroDigital Implies: You may need to track which items of

    personal data have been used for each piece of processing.
  165. @GanbaroDigital Requirement: You must be able to override any solely-automated

    decision.
  166. @GanbaroDigital None of these requirements are unique to ES-CQRS systems.

  167. @GanbaroDigital ES-CQRS systems have unique challenges to overcome.

  168. @GanbaroDigital ?? ?? What happens when immutability meets the reality

    of personal data regulation?
  169. @GanbaroDigital The Problem is State

  170. @GanbaroDigital In a traditional system, many of these GDPR requirements

    are met by amending state.
  171. @GanbaroDigital An ES-CQRS system stores events as its primary data,

    not state.
  172. @GanbaroDigital An ES-CQRS system builds state by playing back events.

  173. @GanbaroDigital Some of these events will hold personal data.

  174. @GanbaroDigital The interaction of events can be complex.

  175. @GanbaroDigital In ES-CQRS, personal data is stored in: - Event

    Store - Projection Cache - Snapshots
  176. @GanbaroDigital Here's the GDRP requirements that uniquely challenge ES-CQRS, and

    some questions to consider when adopting.
  177. @GanbaroDigital Requirement: You must store all personal data securely.

  178. @GanbaroDigital ?? ?? Does a specialist Event Store meet this

    requirement?
  179. @GanbaroDigital ?? ?? Do the Projection Cache and Snapshot storage

    meet this requirement?
  180. @GanbaroDigital Requirement: You must be able to update any piece

    of personal data.
  181. @GanbaroDigital ?? ?? How do you correct data in an

    append-only system?
  182. @GanbaroDigital ?? ?? If you still have the incorrect data,

    does that meet this requirement?
  183. @GanbaroDigital Requirement: You must be able to drop any piece

    of personal data. ... as if you never held it in the first place.
  184. @GanbaroDigital ?? ?? Can you hard-delete personal data from your

    Event Store?
  185. @GanbaroDigital ?? ?? Can you purge any piece of personal

    data from the Projection Cache and any snapshots?
  186. @GanbaroDigital ?? ?? If you do so by rebuilding the

    Projection Cache, are you sure you won't change anyone else's personal data?
  187. @GanbaroDigital Some Event Stores encrypt personal data, and "delete it"

    by throwing away the encryption keys.
  188. @GanbaroDigital ?? ?? If you still have the data, but

    cannot read it today, does that prevent it being read in the future?
  189. @GanbaroDigital ... and don't forget ...

  190. @GanbaroDigital Requirement: You must be able to remove personal data

    from everywhere (inc backups and archives).
  191. @GanbaroDigital Requirement: You must be able to avoid processing personal

    data that you already have.
  192. @GanbaroDigital ?? ?? If you rebuild state for an earlier

    time, how do you honour processing restrictions?
  193. @GanbaroDigital ?? ?? How do you ensure processing restrictions do

    not change anyone else's personal data after a projection rebuild?
  194. @GanbaroDigital Implies: You may need to track which items of

    personal data have been used for each piece of processing.
  195. @GanbaroDigital ?? ?? Can you reproduce the state used at

    any point in time? With 100% accuracy?
  196. @GanbaroDigital ?? ?? Do you need to archive state whenever

    it is used?
  197. @GanbaroDigital ?? ?? Can you have the benefits of Event

    Sourcing and be GDPR-compliant? And is it worth it?
  198. @GanbaroDigital Summing Up

  199. @GanbaroDigital “GDPR is foundational. Compliance touches every aspect of how

    your business works.
  200. @GanbaroDigital “ GDPR is a complex regulatory framework. Obtain, and

    follow, qualified advice.
  201. @GanbaroDigital We're in the early stage of GDPR enforcement. Enforcement

    actions (or inaction!) will shape future advice.
  202. @GanbaroDigital GDPR and immutability appear to be FUNDAMENTALLY incompatible.

  203. @GanbaroDigital As a CTO, I would not adopt any framework

    / approach that relies on immutability if it will store personal data.
  204. @GanbaroDigital If you are going to adopt Event Sourcing ...

  205. @GanbaroDigital When evaluating an ES framework, ask the question: where

    are the docs on how to achieve GDPR compliance?
  206. @GanbaroDigital When evaluating an ES framework, ask the question: where

    is the legal advice that it is GDPR compliant?
  207. @GanbaroDigital In an ES workshop, ask the question: how do

    you achieve GDPR compliance using what is being taught?
  208. @GanbaroDigital In an ES workshop, ask the question: where is

    the legal advice that the approach being taught achieves GDPR compliance?
  209. @GanbaroDigital Your organisation is legally liable for GDPR compliance, not

    the ES framework, not the ES consultant.
  210. @GanbaroDigital

  211. Thank You How Can We Help You? A presentation by

    @stuherbert
 for @GanbaroDigital