SANER 2017 - The Dark Side of Event Sourcing: Managing Data Conversion

Evolving software systems includes data schema
changes, and because of those schema changes data has to be
converted. Converting data between two different schemas while
continuing the operation of the system is a challenge when that
system is expected to be available always. Data conversion in
event sourced systems introduces new challenges, because of
the relative novelty of the event sourcing architectural pattern,
because of the lack of standardized tools for data conversion, and
because of the large amount of data that is stored in typical event
stores. This paper addresses the challenge of schema evolution
and the resulting data conversion for event sourced systems. First
of all a set of event store upgrade operations is proposed that
can be used to convert data between two versions of a data
schema. Second, a set of techniques and strategies that execute
the data conversion while continuing the operation of the system
is discussed. The final contribution is an event store upgrade
framework that identifies which techniques and strategies can
be combined to execute the event store upgrade operations
while continuing operation of the system. Two utilizations of
the framework are given, the first being as decision support in
upfront design of an upgrade system for event sourced systems.
The framework can also be utilized as the description of an
automated upgrade system that can be used for continuous
deployment. The event store upgrade framework is evaluated in
interviews with three renowned experts in the domain and has
been found to be a comprehensive overview that can be utilized
in the design and implementation of an upgrade system. The
automated upgrade system has been implemented partially and
applied in experiments.

Transcript

1. The Dark Side of Event Sourcing: Managing Data Conversion Michiel Overeem, Marten Spoor, Slinger Jansen SANER 2017
2. Event sourcing explained accountId Balance owner 1234567 0 Michiel Current state Event stream Event BankAccountCreated (accountId: 1234567, balance: 0, owner: Michiel); DepositPerformed (accountId : 1234567, amount: 100, balance: 100); WithdrawalPerformed (accountId: 1234567, amount: 50, balance: 50); OwnerChanged (accountId: 1234567, newOwner: Marten); accountId Balance owner 1234567 50 Marten [Fowler, 2005] http://martinfowler.com/eaaDev/EventSourcing.html
3. Command Query Responsibility Segregation (CQRS) explained The command side is only responsible for data modifications. It does not return data. The query side only reads and returns data, it does not modify it. Events make sure that data modifications are propogated to the query-side. [Young, 2010] http://codebetter.com/gregyoung/2010/02/13/cqrs-and-event-sourcing
4. Why event sourcing In era of cloud platforms and microservices, events are the way of communication. Storing those events improves the 1. anticipation on new business requirements, and 2. continuation of the operation of the platform.
5. Challenges for upgrades of event sourced systems • More data than traditional current view. • No standardized language (like SQL) that helps in expressing transformations. And in a cloud platform: • Zero-downtime is needed for data transformations
6. Shining light on the dark side Practitioners often just follow CQRS and Event Sourcing are trending, and sometimes even seen as silver bullets. Many practitioners start out without reviewing the consequences. The community often gives simple answers without taking the context into account.
7. Shining light on the dark side Practitioners often just follow CQRS and Event Sourcing are trending, and sometimes even seen as silver bullets. Many practitioners start out without reviewing the consequences. The community often gives simple answers without taking the context into account. RQ: How can an event sourced system be upgraded efficiently when the (implicit) event schema changes?
8. Research approach  We have created an inventory of the operations that can be used to transform the event store,  We mapped the operations onto techniques that can execute those,  The techniques are linked to strategies, that in turn can be used to deploy the data transformations. The framework is validated with three renowned experts. Part of it is implemented as an exploration. Combined with Deployed with Executed by Executed by Data upgrade Application upgrade Lazy transformation Upcasting In place transfor- mation Multiple versions Copy and transfor- mation Basic & Complex Event Basic & Complex Stream Basic Store Complex Store Big Flip Rolling Upgrade Blue-Green Expand- Contract Blue- Green Deployed with
9. Basic Event Operations Complex Event Operations Basic Stream Operations Complex Stream Operations Basic Store Operations Complex Store Operations Store Backaccount Stream #1234567 BankAccountCreated DepositPerformed WithdrawalPerformed Operations
10. Basic Event Operations Complex Event Operations Basic Stream Operations Complex Stream Operations Basic Store Operations Complex Store Operations Store Backaccount Stream #1234567 BankAccountCreated DepositPerformed WithdrawalPerformed Those break the independence- property and could harm scalability! Operations
11. Runtime Non-runtime Techniques
12. Versioning Evolution Techniques
13. Strategies for zero-downtime upgrades Non-runtime techniques need a strategy for data conversion  Expand-Contract  Blue-Green All techniques need at least a strategy for application upgrade  Big-Flip  Rolling upgrade  Blue-Green
14. Framework  An upfront design and decision support tool, to design the upgrade system of your platform Combined with Deployed with Executed by Executed by Data upgrade Application upgrade Lazy transformation Upcasting In place transfor- mation Multiple versions Copy and transfor- mation Basic & Complex Event Basic & Complex Stream Basic Store Complex Store Big Flip Rolling Upgrade Blue-Green Expand- Contract Blue- Green Deployed with
15. Expert interviews Three dutch experts (all have +6 years experience, and are part of the international community) were interviewed. • Naming issues came up • Frequency of operations was discussed • Technical versus functional immutability “You are maybe the only one, who has such an overview and also thought about edge cases, which I hope never to encounter. ” “Event sourcing needs a lot of upfront thought, which is hard to do with agile development. ”
16. Framework  By implementing two lanes, an automated upgrade system can be built. We have implemented the right lane of the framework in an exploration. Combined with Deployed with Executed by Executed by Data upgrade Application upgrade Upcasting Copy and transfor- mation Basic & Complex Event Basic & Complex Stream Basic Store Complex Store Big Flip Rolling Upgrade Blue-Green Blue- Green Deployed with
17. Future work Can we say anything about the frequency  of the operations,  and the application of the techniques and strategies? Is there any correlation between the type of application and the used technique and strategy? How can we upgrade the query-side in the same upgrade process?
18. Reach us at m.overeem@afas.nl We are the first to give a comprehensive framework of upgrades for event sourced systems. Event sourcing is the pattern for SaaS platforms, because it does not lose data that can be useful later on. Should we not avoid schema evolution at all cost, because it is inheritly destructive?