Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Migrating to Microservice Databases: From Relational Monolith to Distributed Data

Edson Yanaga
February 23, 2017

Migrating to Microservice Databases: From Relational Monolith to Distributed Data

In a distributed system with multiple moving parts, which is the case of Microservices, we can’t allow that a single complement downtime breaks down the entire system. Dealing with stateless code is easy, but it gets much harder when we have to deal with persistent state. In this scenario, zero downtime migrations are paramount to guarantee integrity and consistency.

Within all the Microservices characteristics, undoubtedly the one that creates more perplexity is the “one database per Microservice”. However, very few teams have the privilege of starting something from scratch: most of the times they have a legacy database that will survive any new implementation.

In legacy systems you traditionally have a model that adopts transactions, strong consistency, and CRUD. In order to guarantee integrity and consistency with zero downtime, we must reassess some of these concepts. In this talk we’ll discuss strong and eventual consistency, CRUD and CQRS, Event Sourcing, and how these techniques relate to each other in many different integration and evolution strategies for relational databases. We’ll explore Views, Materialized Views, Mirror Tables, Event Sourcing/Streaming, Data Virtualization, Change Data Capture, and how these strategies enable you to build up a Microservices architecture from a legacy monolithic relational database.

Edson Yanaga

February 23, 2017
Tweet

More Decks by Edson Yanaga

Other Decks in Technology

Transcript

  1. Migrating to Microservice Databases:
    From Relational Monolith to Distributed Data
    Edson Yanaga
    Director of Developer Experience
    @yanaga

    View Slide

  2. Join developers.redhat.com
    2
    Java Champion Microsoft MVP

    View Slide

  3. http://developers.redhat.com/promotions/
    migrating-to-microservice-databases

    View Slide

  4. “Now, every company is a software company”
    — Forbes

    View Slide

  5. Join developers.redhat.com
    5

    View Slide

  6. DevOps &
    Microservices

    View Slide

  7. Feedback
    Loop

    View Slide

  8. Batch Size

    View Slide

  9. Join developers.redhat.com
    9
    Maintenance
    Window

    View Slide

  10. Zero Downtime

    View Slide

  11. Join developers.redhat.com
    11
    Blue Green
    Deployments

    View Slide

  12. Join developers.redhat.com
    12
    Deployment

    View Slide

  13. Join developers.redhat.com
    13
    Deployment
    Proxy

    View Slide

  14. Join developers.redhat.com
    14
    Blue
    Proxy
    Green

    View Slide

  15. Join developers.redhat.com
    15
    Blue
    Proxy
    Green

    View Slide

  16. Join developers.redhat.com
    16
    Blue
    Proxy
    Green

    View Slide

  17. Join developers.redhat.com
    17
    Blue
    Proxy
    Green

    View Slide

  18. Code is easy,
    state is hard

    View Slide

  19. What about
    my
    relational
    database?

    View Slide

  20. Join developers.redhat.com
    20

    View Slide

  21. Zero Downtime
    Migrations

    View Slide

  22. Back and Forward
    Compatibility

    View Slide

  23. Baby Steps = Smallest
    Possible Batch Size

    View Slide

  24. Too many rows = Long Locks

    View Slide

  25. Shard your
    updates

    View Slide

  26. Join developers.redhat.com
    ALTER TABLE customers RENAME COLUMN wrong TO correct;
    26

    View Slide

  27. Join developers.redhat.com
    ALTER TABLE customers ADD COLUMN correct VARCHAR(20);
    UPDATE customers SET correct = wrong
    WHERE id BETWEEN 1 AND 100;
    UPDATE customers SET correct = wrong
    WHERE id BETWEEN 101 AND 200;
    ALTER TABLE customers DELETE COLUMN wrong;
    27

    View Slide

  28. Join developers.redhat.com
    28
    Scenarios
    Add a Column
    Rename a Column
    Change Type/Format of a Column
    Delete a Column

    View Slide

  29. Join developers.redhat.com
    29
    Add a Column
    1 ADD COLUMN
    2 Code computes the read value and writes to new column
    3 Update data using shards
    4 Code reads and writes from the new column

    View Slide

  30. Join developers.redhat.com
    30
    Rename a Column
    1 ADD COLUMN
    2 Code reads from the old column and writes to both
    3 Copy data using small shards
    4 Code reads from the new column and writes to both
    5 Code reads and writes from the new column
    6 Delete the old column (later)

    View Slide

  31. Join developers.redhat.com
    31
    Change Type/Format of a Column
    1 ADD COLUMN
    2 Code reads from the old column and writes to both
    3 Copy data using small shards
    4 Code reads from the new column and writes to both
    5 Code reads and writes from the new column
    6 Delete the old column (later)

    View Slide

  32. Join developers.redhat.com
    32
    Delete a Column
    1 DON’T
    2 Stop using the read value but keep writing to the column
    3 Delete the column

    View Slide

  33. What about referential integrity
    constraints?

    View Slide

  34. Drop them and recreate when
    migration is done

    View Slide

  35. Microservices
    Characteristics
    https://martinfowler.com/microservices/

    View Slide

  36. • Componentization via Services
    • Organized around Business Capabilities
    • Products not Projects
    • Smart endpoints and dumb pipes
    • Decentralized Governance
    • Decentralized Data Management
    • Infrastructure Automation
    • Design for failure
    • Evolutionary Design

    View Slide

  37. Extracting your Microservice
    database

    View Slide

  38. Join developers.redhat.com
    38
    One database per Microservice

    View Slide

  39. Join developers.redhat.com
    39
    But I have a monolithic database!

    View Slide

  40. Join developers.redhat.com
    40

    View Slide

  41. Splitting is not easy,
    but how do I integrate later?

    View Slide

  42. Consistency Models

    View Slide

  43. Strong Consistency
    Eventual Consistency

    View Slide

  44. CRUD & CQRS

    View Slide

  45. Join developers.redhat.com
    45
    CRUD (Create, Read, Update, Delete)

    View Slide

  46. Join developers.redhat.com
    46
    CQRS (Command Query Responsibility Segregation)

    View Slide

  47. Join developers.redhat.com
    47
    CQRS with separate data stores

    View Slide

  48. CQRS & Event Sourcing

    View Slide

  49. Join developers.redhat.com
    49
    Scenarios
    Shared Tables
    Database View
    Database Materialized View
    Mirror Table using Trigger
    Mirror Table using Transactional Code
    Mirror Table using ETL
    Mirror Table using Data Virtualization
    Event Sourcing
    Change Data Capture

    View Slide

  50. Join developers.redhat.com
    50
    Shared Tables
    Fastest Data Integration
    Strong Consistency
    Low cohesion and high coupling

    View Slide

  51. Join developers.redhat.com
    51
    Database View
    Easiest one to implement
    Largest support from DBMS vendors
    Possible performance issues
    Strong Consistency
    One database must be reachable by the other
    Updatable depending on DBMS support

    View Slide

  52. Join developers.redhat.com
    52
    Database Materialized View
    Better performance
    Strong or Eventual Consistency
    One database must be reachable by the other
    Updatable depending on DBMS support

    View Slide

  53. Join developers.redhat.com
    53
    Database Trigger
    Depends on DBMS Support
    Strong Consistency
    One database must be reachable by the other

    View Slide

  54. Join developers.redhat.com
    54
    Transactional Code
    Any code: usually Stored Procedures or Distributed
    Transactions
    Strong Consistency
    Possible cohesion/coupling issues
    Possible performance issues
    Updatable depending on how it is implemented

    View Slide

  55. Join developers.redhat.com
    55
    ETL Tools
    Lots of available tools
    Requires external trigger (usually time-based)
    Can aggregate from multiple datasources
    Eventual Consistency
    Read only integration

    View Slide

  56. Join developers.redhat.com
    56
    Data Virtualization
    Real Time Access
    Strong Consistency
    Can aggregate from multiple datasources
    Updatable depending on Data Virtualization Platform

    View Slide

  57. Join developers.redhat.com
    57

    View Slide

  58. Join developers.redhat.com
    58
    Event Sourcing
    State of data is a stream of events
    Eases auditing
    Eventual Consistency
    Usually combined with a Message Bus
    High scalability

    View Slide

  59. Join developers.redhat.com
    59
    Change Data Capture
    Read datasource is updated through a stream of events
    Eventual Consistency
    Usually combined with a Message Bus
    High scalability

    View Slide

  60. http://debezium.io

    View Slide

  61. Join
    developers.redhat.com
    Feedback welcome!
    @yanaga

    View Slide

  62. plus.google.com/+RedHat
    linkedin.com/company/red-hat
    youtube.com/user/RedHatVideos
    facebook.com/redhatinc
    twitter.com/RedHatNews
    Thank you!

    View Slide