Slide 1

Slide 1 text

The Dark Side of Event Sourcing: Managing Data Conversion Michiel Overeem, Marten Spoor, Slinger Jansen M. Overeem, M. Spoor, and S. Jansen, “The Dark Side of Event Sourcing: Managing Data Conversion,” in IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER), 2017, pp. 193–204.

Slide 2

Slide 2 text

Event sourcing 101 accountId balance owner 1234567 0 Michiel Current state Event sourcing Event BankAccountCreated (accountId: 1234567, balance: 0, owner: Michiel) M. Fowler, “Event sourcing,” http://martinfowler.com/eaaDev/EventSourcing.html, 2005.

Slide 3

Slide 3 text

Event sourcing 101 accountId balance owner 1234567 100 Michiel Current state Event sourcing Event BankAccountCreated (accountId: 1234567, balance: 0, owner: Michiel) DepositPerformed (accountId : 1234567, amount: 100, balance: 100) M. Fowler, “Event sourcing,” http://martinfowler.com/eaaDev/EventSourcing.html, 2005.

Slide 4

Slide 4 text

Event sourcing 101 Event BankAccountCreated (accountId: 1234567, balance: 0, owner: Michiel) DepositPerformed (accountId : 1234567, amount: 100, balance: 100) WithdrawalPerformed (accountId: 1234567, amount: 50, balance: 50) accountId balance owner 1234567 50 Michiel Current state Event sourcing M. Fowler, “Event sourcing,” http://martinfowler.com/eaaDev/EventSourcing.html, 2005.

Slide 5

Slide 5 text

Event sourcing 101 Event BankAccountCreated (accountId: 1234567, balance: 0, owner: Michiel) DepositPerformed (accountId : 1234567, amount: 100, balance: 100) WithdrawalPerformed (accountId: 1234567, amount: 50, balance: 50) OwnerChanged (accountId: 1234567, newOwner: Marten) accountId balance owner 1234567 50 Marten Current state Event sourcing M. Fowler, “Event sourcing,” http://martinfowler.com/eaaDev/EventSourcing.html, 2005.

Slide 6

Slide 6 text

Sure, but this is normal for transactions …

Slide 7

Slide 7 text

Event sourcing 101 Event ShoppingCartCreated (cartid: 1234567, user: Michiel) cartid articles user 1234567 0 Michiel Current state Event sourcing M. Fowler, “Event sourcing,” http://martinfowler.com/eaaDev/EventSourcing.html, 2005. cartid article

Slide 8

Slide 8 text

Event sourcing 101 Event ShoppingCartCreated (cartid: 1234567, user: Michiel) ArticleAddedToCart(cartid: 1234567, article: Lumia 950) cartid articles user 1234567 1 Michiel Current state Event sourcing cartid article 1234567 Lumia 950 M. Fowler, “Event sourcing,” http://martinfowler.com/eaaDev/EventSourcing.html, 2005.

Slide 9

Slide 9 text

Event sourcing 101 cartid articles user 1234567 0 Michiel Current state Event sourcing cartid article Event ShoppingCartCreated (cartid: 1234567, user: Michiel) ArticleAddedToCart(cartid: 1234567, article: Lumia 950) ArticleRemovedFromCart(cartid: 1234567, article: Lumia 950) M. Fowler, “Event sourcing,” http://martinfowler.com/eaaDev/EventSourcing.html, 2005.

Slide 10

Slide 10 text

Event sourcing 101 cartid articles user 1234567 1 Michiel Current state Event sourcing cartid article 1234567 iPhone X Event ShoppingCartCreated (cartid: 1234567, user: Michiel) ArticleAddedToCart(cartid: 1234567, article: Lumia 950) ArticleRemovedFromCart(cartid: 1234567, article: Lumia 950) ArticleAddedToCart(cartid: 1234567, article: iPhone X) M. Fowler, “Event sourcing,” http://martinfowler.com/eaaDev/EventSourcing.html, 2005.

Slide 11

Slide 11 text

Event sourcing 101 cartid articles user 1234567 1 Michiel Current state Event sourcing cartid article 1234567 iPhone X Event ShoppingCartCreated (cartid: 1234567, user: Michiel) ArticleAddedToCart(cartid: 1234567, article: Lumia 950) ArticleRemovedFromCart(cartid: 1234567, article: Lumia 950) ArticleAddedToCart(cartid: 1234567, article: iPhone X) M. Fowler, “Event sourcing,” http://martinfowler.com/eaaDev/EventSourcing.html, 2005.

Slide 12

Slide 12 text

Command Query Responsibility Segregation (CQRS) 101 The command side is only responsible for data modifications. It does not return data. The query side only reads and returns data, it does not modify it. Events make sure that data modifications are propogated to the query-side. G. Young, “CQRS and Event Sourcing,” http://codebetter.com/gregyoung/2010/02/13/cqrs-and-event-sourcing, 2010.

Slide 13

Slide 13 text

Command Query Responsibility Segregation (CQRS) 101 These events could be communicated through an servicebus. But they could also be stored as an append-only log. G. Young, “CQRS and Event Sourcing,” http://codebetter.com/gregyoung/2010/02/13/cqrs-and-event-sourcing, 2010.

Slide 14

Slide 14 text

Why event sourcing In the era of cloud platforms and microservices, events are the way of communication. Storing those events improves the 1. anticipation on new business requirements, and 2. continuation of the operation of the platform.

Slide 15

Slide 15 text

Shining light on the dark side Practitioners often just follow ▪CQRS and Event Sourcing are trending, and sometimes even seen as silver bullets. ▪Many practitioners start out without reviewing the consequences. ▪The community often gives simple answers without taking the context into account. RQ: How can an event sourced system be upgraded efficiently when the (implicit) event schema changes?

Slide 16

Slide 16 text

Challenges for upgrades of event sourced systems • More data than traditional current view. • No standardized language (like SQL) that helps in expressing transformations. And in a cloud platform: • Zero-downtime is needed for data transformations

Slide 17

Slide 17 text

Research approach ▪ We have created an inventory of the operations that can be used to transform the event store, ▪ We mapped the operations onto techniques that can execute those, ▪ The techniques are linked to strategies, that in turn can be used to deploy the data transformations. The framework is validated with three renowned experts. Part of it is implemented as an exploration. Combined with Deployed with Executed by Executed by Data upgrade Application upgrade Lazy transformation Upcasting In place transfor- mation Multiple versions Copy and transfor- mation Basic & Complex Event Basic & Complex Stream Basic Store Complex Store Big Flip Rolling Upgrade Blue-Green Expand- Contract Blue- Green Deployed with

Slide 18

Slide 18 text

Basic Event Operations Complex Event Operations Basic Stream Operations Complex Stream Operations Basic Store Operations Complex Store Operations Store Backaccount Stream #1234567 BankAccountCreated DepositPerformed WithdrawalPerformed Operations

Slide 19

Slide 19 text

The framework Combined with Deployed with Executed by Executed by Data upgrade Application upgrade Lazy transformation Upcasting In place transfor- mation Multiple versions Copy and transfor- mation Basic & Complex Event Basic & Complex Stream Basic Store Complex Store Big Flip Rolling Upgrade Blue-Green Expand- Contract Blue- Green Deployed with At the top an inventory of operations.

Slide 20

Slide 20 text

Multiple versions Techniques Multiple versions of an event type are supported throughout the application. D. Betts, J. Dominguez, G. Melnik, F. Simonazzi, and M. Subramanian, Exploring CQRS and Event Sourcing: A journey into high scalability, availability, and maintainability with Windows Azure. Microsoft patterns & practices, 2013.

Slide 21

Slide 21 text

Multiple versions Techniques Multiple versions of an event type are supported throughout the application. Axon Framework, “Reference Guide Axon Framework reference guide - Event Upcasting.” 2016. Upcasting Introduce a central component that transforms an event before offering it to the application.

Slide 22

Slide 22 text

Multiple versions Techniques Multiple versions of an event type are supported throughout the application. P. J. Sadalage and M. Fowler, NoSQL distilled: a brief guide to the emerging world of polyglot persistence. Addison-Wesley, 2012. Upcasting Introduce a central component that transforms an event before offering it to the application. Lazy transformation Store the result of the transformation permanently in the event store.

Slide 23

Slide 23 text

Multiple versions Techniques Multiple versions of an event type are supported throughout the application. S. Scherzinger, M. Klettke, and U. Störl, “Cleager: Eager Schema Evolution in NoSQL Document Stores,” Datenbanksysteme f{ü}r Business, Technol. und Web (BTW), 16. Fachtagung des GI-Fachbereichs “Datenbanken und Informationssysteme” (DBIS), Upcasting Introduce a central component that transforms an event before offering it to the application. Lazy transformation Store the result of the transformation permanently in the event store. In place transformation A batch job that eagerly reads the data, transforms it, and writes the updated data back to the database.

Slide 24

Slide 24 text

Multiple versions Techniques Multiple versions of an event type are supported throughout the application. M. Callaghan, “Facebook - Online Schema Change for MySQL,” 2010. https://www.facebook.com/notes/mysql-at-facebook/online-schema- change-for-mysql/430801045932/. Upcasting Introduce a central component that transforms an event before offering it to the application. Lazy transformation Store the result of the transformation permanently in the event store. In place transformation A batch job that eagerly reads the data, transforms it, and writes the updated data back to the database. Copy and transformation Copies and transforms every event to a new store.

Slide 25

Slide 25 text

Multiple versions Techniques Multiple versions of an event type are supported throughout the application. M. Callaghan, “Facebook - Online Schema Change for MySQL,” 2010. https://www.facebook.com/notes/mysql-at-facebook/online-schema- change-for-mysql/430801045932/. Upcasting Introduce a central component that transforms an event before offering it to the application. Lazy transformation Store the result of the transformation permanently in the event store. In place transformation A batch job that eagerly reads the data, transforms it, and writes the updated data back to the database. Copy and transformation Copies and transforms every event to a new store.

Slide 26

Slide 26 text

The framework Combined with Deployed with Executed by Executed by Data upgrade Application upgrade Lazy transformation Upcasting In place transfor- mation Multiple versions Copy and transfor- mation Basic & Complex Event Basic & Complex Stream Basic Store Complex Store Big Flip Rolling Upgrade Blue-Green Expand- Contract Blue- Green Deployed with Techniques that execute the identified operations: ▪ at run-time or at deploy-time, ▪ through schema evolution or schema versioning.

Slide 27

Slide 27 text

Expand-Contract Strategies Three phase change: first expand, then update all concumers, then contract. D. Sato, “ParallelChange,” 2014. http://martinfowler.com/bliki/ParallelChange.html.

Slide 28

Slide 28 text

Expand-Contract Strategies Three phase change: first expand, then update all concumers, then contract. J. Humble and D. Farley, Continuous delivery: reliable software releases through build, test, and deployment automation. Addison-Wesley Professional, 2010. Blue-Green Use two slots for deployment and toggle between them.

Slide 29

Slide 29 text

Expand-Contract Strategies Three phase change: first expand, then update all concumers, then contract. T. Dumitras, P. Narasimhan, and E. Tilevich, “To Upgrade or Not to Upgrade Impact of Online Upgrades across Multiple Administrative Domains,” ACM SIGPLAN Not., vol. 45, no. 10, pp. 865–876, 2010. Blue-Green Use two slots for deployment and toggle between them. Rolling upgrade Upgrade machine for machine.

Slide 30

Slide 30 text

Expand-Contract Strategies Three phase change: first expand, then update all concumers, then contract. E. A. Brewer, “Lessons from Giant-Scale Services,” {IEEE} Internet Comput., vol. 5, no. 4, pp. 46–55, 2001. Blue-Green Use two slots for deployment and toggle between them. Rolling upgrade Upgrade machine for machine. Big Flip Flip the switch after updating half the machines.

Slide 31

Slide 31 text

Framework ▪ An upfront design and decision support tool, to design the upgrade system of your platform Combined with Deployed with Executed by Executed by Data upgrade Application upgrade Lazy transformation Upcasting In place transfor- mation Multiple versions Copy and transfor- mation Basic & Complex Event Basic & Complex Stream Basic Store Complex Store Big Flip Rolling Upgrade Blue-Green Expand- Contract Blue- Green Deployed with

Slide 32

Slide 32 text

Expert interviews Three dutch experts (all have +6 years experience, and are part of the international community) were interviewed. • Naming issues came up • Frequency of operations was discussed • Technical versus functional immutability “You are maybe the only one, who has such an overview and also thought about edge cases, which I hope never to encounter. ” “Event sourcing needs a lot of upfront thought, which is hard to do with agile development. ”

Slide 33

Slide 33 text

Framework ▪ By implementing two lanes, an automated upgrade system can be built. Combined with Deployed with Executed by Executed by Data upgrade Application upgrade Upcasting Copy and transfor- mation Basic & Complex Event Basic & Complex Stream Basic Store Complex Store Big Flip Rolling Upgrade Blue-Green Blue- Green Deployed with

Slide 34

Slide 34 text

Reach us at [email protected] Event sourcing is the pattern for SaaS platforms, because it does not lose data that is found to be useful later on. But more research is needed to formulate effective strategies on implementing event sourcing.