Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Migrating to Microservice Databases: From Relat...

Edson Yanaga
February 23, 2017

Migrating to Microservice Databases: From Relational Monolith to Distributed Data

In a distributed system with multiple moving parts, which is the case of Microservices, we can’t allow that a single complement downtime breaks down the entire system. Dealing with stateless code is easy, but it gets much harder when we have to deal with persistent state. In this scenario, zero downtime migrations are paramount to guarantee integrity and consistency.

Within all the Microservices characteristics, undoubtedly the one that creates more perplexity is the “one database per Microservice”. However, very few teams have the privilege of starting something from scratch: most of the times they have a legacy database that will survive any new implementation.

In legacy systems you traditionally have a model that adopts transactions, strong consistency, and CRUD. In order to guarantee integrity and consistency with zero downtime, we must reassess some of these concepts. In this talk we’ll discuss strong and eventual consistency, CRUD and CQRS, Event Sourcing, and how these techniques relate to each other in many different integration and evolution strategies for relational databases. We’ll explore Views, Materialized Views, Mirror Tables, Event Sourcing/Streaming, Data Virtualization, Change Data Capture, and how these strategies enable you to build up a Microservices architecture from a legacy monolithic relational database.

Edson Yanaga

February 23, 2017
Tweet

More Decks by Edson Yanaga

Other Decks in Technology

Transcript

  1. Migrating to Microservice Databases: From Relational Monolith to Distributed Data

    Edson Yanaga Director of Developer Experience @yanaga
  2. Join developers.redhat.com ALTER TABLE customers ADD COLUMN correct VARCHAR(20); UPDATE

    customers SET correct = wrong WHERE id BETWEEN 1 AND 100; UPDATE customers SET correct = wrong WHERE id BETWEEN 101 AND 200; ALTER TABLE customers DELETE COLUMN wrong; 27
  3. Join developers.redhat.com 28 Scenarios Add a Column Rename a Column

    Change Type/Format of a Column Delete a Column
  4. Join developers.redhat.com 29 Add a Column 1 ADD COLUMN 2

    Code computes the read value and writes to new column 3 Update data using shards 4 Code reads and writes from the new column
  5. Join developers.redhat.com 30 Rename a Column 1 ADD COLUMN 2

    Code reads from the old column and writes to both 3 Copy data using small shards 4 Code reads from the new column and writes to both 5 Code reads and writes from the new column 6 Delete the old column (later)
  6. Join developers.redhat.com 31 Change Type/Format of a Column 1 ADD

    COLUMN 2 Code reads from the old column and writes to both 3 Copy data using small shards 4 Code reads from the new column and writes to both 5 Code reads and writes from the new column 6 Delete the old column (later)
  7. Join developers.redhat.com 32 Delete a Column 1 DON’T 2 Stop

    using the read value but keep writing to the column 3 Delete the column
  8. • Componentization via Services • Organized around Business Capabilities •

    Products not Projects • Smart endpoints and dumb pipes • Decentralized Governance • Decentralized Data Management • Infrastructure Automation • Design for failure • Evolutionary Design
  9. Join developers.redhat.com 49 Scenarios Shared Tables Database View Database Materialized

    View Mirror Table using Trigger Mirror Table using Transactional Code Mirror Table using ETL Mirror Table using Data Virtualization Event Sourcing Change Data Capture
  10. Join developers.redhat.com 51 Database View Easiest one to implement Largest

    support from DBMS vendors Possible performance issues Strong Consistency One database must be reachable by the other Updatable depending on DBMS support
  11. Join developers.redhat.com 52 Database Materialized View Better performance Strong or

    Eventual Consistency One database must be reachable by the other Updatable depending on DBMS support
  12. Join developers.redhat.com 53 Database Trigger Depends on DBMS Support Strong

    Consistency One database must be reachable by the other
  13. Join developers.redhat.com 54 Transactional Code Any code: usually Stored Procedures

    or Distributed Transactions Strong Consistency Possible cohesion/coupling issues Possible performance issues Updatable depending on how it is implemented
  14. Join developers.redhat.com 55 ETL Tools Lots of available tools Requires

    external trigger (usually time-based) Can aggregate from multiple datasources Eventual Consistency Read only integration
  15. Join developers.redhat.com 56 Data Virtualization Real Time Access Strong Consistency

    Can aggregate from multiple datasources Updatable depending on Data Virtualization Platform
  16. Join developers.redhat.com 58 Event Sourcing State of data is a

    stream of events Eases auditing Eventual Consistency Usually combined with a Message Bus High scalability
  17. Join developers.redhat.com 59 Change Data Capture Read datasource is updated

    through a stream of events Eventual Consistency Usually combined with a Message Bus High scalability