Migrating to Microservice Databases: From Relational Monolith to Distributed Data

In a distributed system with multiple moving parts, which is the case of Microservices, we can’t allow that a single complement downtime breaks down the entire system. Dealing with stateless code is easy, but it gets much harder when we have to deal with persistent state. In this scenario, zero downtime migrations are paramount to guarantee integrity and consistency.

Within all the Microservices characteristics, undoubtedly the one that creates more perplexity is the “one database per Microservice”. However, very few teams have the privilege of starting something from scratch: most of the times they have a legacy database that will survive any new implementation.

In legacy systems you traditionally have a model that adopts transactions, strong consistency, and CRUD. In order to guarantee integrity and consistency with zero downtime, we must reassess some of these concepts. In this talk we’ll discuss strong and eventual consistency, CRUD and CQRS, Event Sourcing, and how these techniques relate to each other in many different integration and evolution strategies for relational databases. We’ll explore Views, Materialized Views, Mirror Tables, Event Sourcing/Streaming, Data Virtualization, Change Data Capture, and how these strategies enable you to build up a Microservices architecture from a legacy monolithic relational database.

Transcript

1. Migrating to Microservice Databases: From Relational Monolith to Distributed Data Edson Yanaga Director of Developer Experience @yanaga
2. Join developers.redhat.com 2 Java Champion Microsoft MVP
3. http://developers.redhat.com/promotions/ migrating-to-microservice-databases
4. “Now, every company is a software company” — Forbes
5. Join developers.redhat.com 5
6. DevOps & Microservices
7. Feedback Loop
8. Batch Size
9. Join developers.redhat.com 9 Maintenance Window
10. Zero Downtime
11. Join developers.redhat.com 11 Blue Green Deployments
12. Join developers.redhat.com 12 Deployment
13. Join developers.redhat.com 13 Deployment Proxy
14. Join developers.redhat.com 14 Blue Proxy Green
15. Join developers.redhat.com 15 Blue Proxy Green
16. Join developers.redhat.com 16 Blue Proxy Green
17. Join developers.redhat.com 17 Blue Proxy Green
18. Code is easy, state is hard
19. What about my relational database?
20. Join developers.redhat.com 20
21. Zero Downtime Migrations
22. Back and Forward Compatibility
23. Baby Steps = Smallest Possible Batch Size
24. Too many rows = Long Locks
25. Shard your updates
26. Join developers.redhat.com ALTER TABLE customers RENAME COLUMN wrong TO correct; 26
27. Join developers.redhat.com ALTER TABLE customers ADD COLUMN correct VARCHAR(20); UPDATE customers SET correct = wrong WHERE id BETWEEN 1 AND 100; UPDATE customers SET correct = wrong WHERE id BETWEEN 101 AND 200; ALTER TABLE customers DELETE COLUMN wrong; 27
28. Join developers.redhat.com 28 Scenarios Add a Column Rename a Column Change Type/Format of a Column Delete a Column
29. Join developers.redhat.com 29 Add a Column 1 ADD COLUMN 2 Code computes the read value and writes to new column 3 Update data using shards 4 Code reads and writes from the new column
30. Join developers.redhat.com 30 Rename a Column 1 ADD COLUMN 2 Code reads from the old column and writes to both 3 Copy data using small shards 4 Code reads from the new column and writes to both 5 Code reads and writes from the new column 6 Delete the old column (later)
31. Join developers.redhat.com 31 Change Type/Format of a Column 1 ADD COLUMN 2 Code reads from the old column and writes to both 3 Copy data using small shards 4 Code reads from the new column and writes to both 5 Code reads and writes from the new column 6 Delete the old column (later)
32. Join developers.redhat.com 32 Delete a Column 1 DON’T 2 Stop using the read value but keep writing to the column 3 Delete the column
33. What about referential integrity constraints?
34. Drop them and recreate when migration is done
35. Microservices Characteristics https://martinfowler.com/microservices/
36. • Componentization via Services • Organized around Business Capabilities • Products not Projects • Smart endpoints and dumb pipes • Decentralized Governance • Decentralized Data Management • Infrastructure Automation • Design for failure • Evolutionary Design
37. Extracting your Microservice database
38. Join developers.redhat.com 38 One database per Microservice
39. Join developers.redhat.com 39 But I have a monolithic database!
40. Join developers.redhat.com 40
41. Splitting is not easy, but how do I integrate later?
42. Consistency Models
43. Strong Consistency Eventual Consistency
44. CRUD & CQRS
45. Join developers.redhat.com 45 CRUD (Create, Read, Update, Delete)
46. Join developers.redhat.com 46 CQRS (Command Query Responsibility Segregation)
47. Join developers.redhat.com 47 CQRS with separate data stores
48. CQRS & Event Sourcing
49. Join developers.redhat.com 49 Scenarios Shared Tables Database View Database Materialized View Mirror Table using Trigger Mirror Table using Transactional Code Mirror Table using ETL Mirror Table using Data Virtualization Event Sourcing Change Data Capture
50. Join developers.redhat.com 50 Shared Tables Fastest Data Integration Strong Consistency Low cohesion and high coupling
51. Join developers.redhat.com 51 Database View Easiest one to implement Largest support from DBMS vendors Possible performance issues Strong Consistency One database must be reachable by the other Updatable depending on DBMS support
52. Join developers.redhat.com 52 Database Materialized View Better performance Strong or Eventual Consistency One database must be reachable by the other Updatable depending on DBMS support
53. Join developers.redhat.com 53 Database Trigger Depends on DBMS Support Strong Consistency One database must be reachable by the other
54. Join developers.redhat.com 54 Transactional Code Any code: usually Stored Procedures or Distributed Transactions Strong Consistency Possible cohesion/coupling issues Possible performance issues Updatable depending on how it is implemented
55. Join developers.redhat.com 55 ETL Tools Lots of available tools Requires external trigger (usually time-based) Can aggregate from multiple datasources Eventual Consistency Read only integration
56. Join developers.redhat.com 56 Data Virtualization Real Time Access Strong Consistency Can aggregate from multiple datasources Updatable depending on Data Virtualization Platform
57. Join developers.redhat.com 57
58. Join developers.redhat.com 58 Event Sourcing State of data is a stream of events Eases auditing Eventual Consistency Usually combined with a Message Bus High scalability
59. Join developers.redhat.com 59 Change Data Capture Read datasource is updated through a stream of events Eventual Consistency Usually combined with a Message Bus High scalability
60. http://debezium.io
61. Join developers.redhat.com Feedback welcome! @yanaga
62. plus.google.com/+RedHat linkedin.com/company/red-hat youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHatNews Thank you!