The Dark Side of Event Sourcing: Managing Data Conversion Michiel Overeem, Marten Spoor, Slinger Jansen SANER 2017

Event sourcing explained accountId Balance owner 1234567 0 Michiel Current state Event stream Event BankAccountCreated (accountId: 1234567, balance: 0, owner: Michiel); DepositPerformed (accountId : 1234567, amount: 100, balance: 100); WithdrawalPerformed (accountId: 1234567, amount: 50, balance: 50); OwnerChanged (accountId: 1234567, newOwner: Marten); accountId Balance owner 1234567 50 Marten [Fowler, 2005] http://martinfowler.com/eaaDev/EventSourcing.html

Command Query Responsibility Segregation (CQRS) explained The command side is only responsible for data modifications. It does not return data. The query side only reads and returns data, it does not modify it. Events make sure that data modifications are propogated to the query-side. [Young, 2010] http://codebetter.com/gregyoung/2010/02/13/cqrs-and-event-sourcing

Why event sourcing In era of cloud platforms and microservices, events are the way of communication. Storing those events improves the 1. anticipation on new business requirements, and 2. continuation of the operation of the platform.

Challenges for upgrades of event sourced systems • More data than traditional current view. • No standardized language (like SQL) that helps in expressing transformations. And in a cloud platform: • Zero-downtime is needed for data transformations

Shining light on the dark side Practitioners often just follow CQRS and Event Sourcing are trending, and sometimes even seen as silver bullets. Many practitioners start out without reviewing the consequences. The community often gives simple answers without taking the context into account.

Shining light on the dark side Practitioners often just follow CQRS and Event Sourcing are trending, and sometimes even seen as silver bullets. Many practitioners start out without reviewing the consequences. The community often gives simple answers without taking the context into account. RQ: How can an event sourced system be upgraded efficiently when the (implicit) event schema changes?

Research approach  We have created an inventory of the operations that can be used to transform the event store,  We mapped the operations onto techniques that can execute those,  The techniques are linked to strategies, that in turn can be used to deploy the data transformations. The framework is validated with three renowned experts. Part of it is implemented as an exploration. Combined with Deployed with Executed by Executed by Data upgrade Application upgrade Lazy transformation Upcasting In place transfor- mation Multiple versions Copy and transfor- mation Basic & Complex Event Basic & Complex Stream Basic Store Complex Store Big Flip Rolling Upgrade Blue-Green Expand- Contract Blue- Green Deployed with

Basic Event Operations Complex Event Operations Basic Stream Operations Complex Stream Operations Basic Store Operations Complex Store Operations Store Backaccount Stream #1234567 BankAccountCreated DepositPerformed WithdrawalPerformed Operations

Basic Event Operations Complex Event Operations Basic Stream Operations Complex Stream Operations Basic Store Operations Complex Store Operations Store Backaccount Stream #1234567 BankAccountCreated DepositPerformed WithdrawalPerformed Those break the independence- property and could harm scalability! Operations

Runtime Non-runtime Techniques

Versioning Evolution Techniques

Strategies for zero-downtime upgrades Non-runtime techniques need a strategy for data conversion  Expand-Contract  Blue-Green All techniques need at least a strategy for application upgrade  Big-Flip  Rolling upgrade  Blue-Green

Framework  An upfront design and decision support tool, to design the upgrade system of your platform Combined with Deployed with Executed by Executed by Data upgrade Application upgrade Lazy transformation Upcasting In place transfor- mation Multiple versions Copy and transfor- mation Basic & Complex Event Basic & Complex Stream Basic Store Complex Store Big Flip Rolling Upgrade Blue-Green Expand- Contract Blue- Green Deployed with

Expert interviews Three dutch experts (all have +6 years experience, and are part of the international community) were interviewed. • Naming issues came up • Frequency of operations was discussed • Technical versus functional immutability “You are maybe the only one, who has such an overview and also thought about edge cases, which I hope never to encounter. ” “Event sourcing needs a lot of upfront thought, which is hard to do with agile development. ”

Framework  By implementing two lanes, an automated upgrade system can be built. We have implemented the right lane of the framework in an exploration. Combined with Deployed with Executed by Executed by Data upgrade Application upgrade Upcasting Copy and transfor- mation Basic & Complex Event Basic & Complex Stream Basic Store Complex Store Big Flip Rolling Upgrade Blue-Green Blue- Green Deployed with

Future work Can we say anything about the frequency  of the operations,  and the application of the techniques and strategies? Is there any correlation between the type of application and the used technique and strategy? How can we upgrade the query-side in the same upgrade process?

Reach us at [email protected] We are the first to give a comprehensive framework of upgrades for event sourced systems. Event sourcing is the pattern for SaaS platforms, because it does not lose data that can be useful later on. Should we not avoid schema evolution at all cost, because it is inheritly destructive?