Database Migrations: the Missing Link to Continuous Delivery

Slides of the speech I (@filippobosi), Luca Acquaviva (@lacquaviva) and Stefano Monti (@ste_monti) presented at Devoxx Poland 2018 in Krakow, about how to address correctly automated testing (both unit and regression) in a delivery pipeline where several applications are sharing a single database.

Transcript

1. @lacquaviva @filippobosi @ste_monti #DevoxxPL Platinum Sponsor: Database migrations: the missing link to continuous delivery Filippo Bosi @filippobosi fbosi@imolainformatica.it Luca Acquaviva @lacquaviva lacquaviva@imolainformatica.it Stefano Monti @ste_monti smonti@imolainformatica.it http://www.imolainformatica.it
2. @lacquaviva @filippobosi @ste_monti #DevoxxPL
3. @lacquaviva @filippobosi @ste_monti #DevoxxPL …that’s nothing new
4. @lacquaviva @filippobosi @ste_monti #DevoxxPL …that’s nothing new Continuous Delivery
5. @lacquaviva @filippobosi @ste_monti #DevoxxPL Continuous Delivery with DATABASES & NO DOWNTIME
6. @lacquaviva @filippobosi @ste_monti #DevoxxPL …that’s nothing new Continuous Delivery with DATABASES & NO DOWNTIME
7. @lacquaviva @filippobosi @ste_monti #DevoxxPL Principles & Application lifecycle Patterns Tools Continuous Delivery with Zero Downtime
8. @lacquaviva @filippobosi @ste_monti #DevoxxPL Simple Scenario “Single application with dedicated database” VS Complex Scenarios “Multiple applications with shared database” E.g. Monolith-to-microservices migrations Continuous Delivery with Zero Downtime
9. @lacquaviva @filippobosi @ste_monti #DevoxxPL Principles & Application lifecycle - Avoid breaking database changes à Maintain backward compatibility - Treat your database as code (Versioning & lifecycle) - Automate DB Testing Patterns Tools Single application with dedicated database
10. @lacquaviva @filippobosi @ste_monti #DevoxxPL Treat your database as code • Database changeset format (plain SQL vs XML/Liquibase) • Database objects: DDL vs data vs stored procedures • Version-control code AND database • Relate application code and database via version numbers and explicit dependencies Single application with dedicated database https://www.slideshare.net/axelfontaine/continuous-delivery-and-zero-downtime
11. @lacquaviva @filippobosi @ste_monti #DevoxxPL Tips • Single project for Database and Code • Database Code (DDL+Logic) and application share the same lifecycle • Release and deploy Application and Database Dedicated database - lifecycle one team one project one repository one pipeline one deploy
12. @lacquaviva @filippobosi @ste_monti #DevoxxPL Continuous Integration Deployment Env Artifact Repo Deploy Continuous Integration Deployment Env Artifact Repo Deploy
13. @lacquaviva @filippobosi @ste_monti #DevoxxPL Principles & Application lifecycle Patterns - Expand & Contract - Blue/Green Deploy (or Feature Toggle) - Adopt Delivery Pipelines Tools Single application with dedicated database
14. @lacquaviva @filippobosi @ste_monti #DevoxxPL Expand and contract https://dotnetvibes.com/2018/04/14/handling-rollback-of-database-deployments-with-a-single-click/ Make sure that database changes are backward compatibile E.g. Renaming Column SurName to LastName Name SurName LastName Expand: - add a new column LastName - add Trigger or Logic for dat a alignment Transition: - adapt application logic to use the new column - test extensively Contract: - remove old column and trigger Name LastName Name SurName LastName
15. @lacquaviva @filippobosi @ste_monti #DevoxxPL Blue/green deploy https://martinfowler.com/bliki/BlueGreenDeployment.html
16. @lacquaviva @filippobosi @ste_monti #DevoxxPL if (flag){ //access v1.1 ... } else{ //access v1.0 ... } Feature toggle https://martinfowler.com/articles/feature-toggles.html Name Surname LastName Name Surname LastName v1.0 v1.1
17. @lacquaviva @filippobosi @ste_monti #DevoxxPL Name Surname V 1.0 V 1.0 Blue/Green+Expand&Contract (1/4)
18. @lacquaviva @filippobosi @ste_monti #DevoxxPL V 1.1 Name Surname LastName V 1.1 V 2.0 #1 Deploy - Expand Blue/Green+Expand&Contract (2/4)
19. @lacquaviva @filippobosi @ste_monti #DevoxxPL Blue/Green+Expand&Contract (3/4) Name Surname LastName V 1.1 V 2.0 V 1.1 #2 Transition
20. @lacquaviva @filippobosi @ste_monti #DevoxxPL Blue/Green+Expand&Contract (4/4) Name LastName V 2.0 V 2.0 V 1.1 #3 Deploy -Contract
21. @lacquaviva @filippobosi @ste_monti #DevoxxPL Principles & Application lifecycle Patterns Tools - Database automation, e.g., Liquibase vs Flyway - Continuous Integration/Delivery , e.g., Jenkins Continuous Delivery with Zero downtime
22. @lacquaviva @filippobosi @ste_monti #DevoxxPL Active Migrations Tools
23. @lacquaviva @filippobosi @ste_monti #DevoxxPL Tools are required but not enough
24. @lacquaviva @filippobosi @ste_monti #DevoxxPL Principles, Patterns and tools mitigate problems, however… …migrations can fail! YOU NEED TO PLAN FOR (AUTOMATED) DATABASE ROLLBACKS! Open points – #1 migration failures
25. @lacquaviva @filippobosi @ste_monti #DevoxxPL WHAT DDL (provided backward compatibility is guaranteed) Stored Procedures/Logic (provided backward compatibility is guaranteed) DATA: WARNING STATIC/DOMAIN DATA (no env-dependent data!) LIVE/BUSINESS DATA (only fake data for testing purposes) Open points – #2 migration logic
26. @lacquaviva @filippobosi @ste_monti #DevoxxPL Continuous Delivery with Zero downtime Simple Scenario “Single application with dedicated database” VS Complex Scenarios “Multiple applications with shared database” E.g. Monolith-to-microservices migrations
27. @lacquaviva @filippobosi @ste_monti #DevoxxPL A typical monolith-to-microservices roadmap… Monolith-to-microservices migration 2. Modular Application with Single Database 1. Monolith 3. Microservices
28. @lacquaviva @filippobosi @ste_monti #DevoxxPL Monolith-to-microservices migration: an example router t0 router router router t1 t2 t3 t4 router
29. @lacquaviva @filippobosi @ste_monti #DevoxxPL • Many applications/modules depend on the same database àTreat database as a dependency: database artifact with version àMore complex and extensive automated database testing • Many teams need to modify the database àMore complex application lifecycles and delivery pipelines Monolith-to-microservices migration: issues Worst case
30. @lacquaviva @filippobosi @ste_monti #DevoxxPL Testing strategies – App modules UNIT TEST END 2 END TEST ACCEPTANCE ENV
31. @lacquaviva @filippobosi @ste_monti #DevoxxPL Testing strategies – App modules https://martinfowler.com/articles/microservice-testing/ UNIT TEST CONTRACT TEST END 2 END TEST LOCAL/SANDBOXED ENV ACCEPTANCE ENV
32. @lacquaviva @filippobosi @ste_monti #DevoxxPL Testing strategies – DB module UNIT TEST CONTRACT TEST M1 LOCAL/SANDBOXED ENV CONTRACT TEST M2 ACCEPTANCE ENV
33. @lacquaviva @filippobosi @ste_monti #DevoxxPL multiple distinct repo Tips • Dedicated project for Database and Code • Database code (DDL+Logic) and applications code have different lifecycle • Consider Database like another dependency Shared database many teams multiple projects multiple repositories multiple pipelines multiple deployments multiple distinct project multiple distinct pipeline depends
34. ACCEPTANCE PRO D END TO END TESTING CONTINUOUS DB INTEGRATION PIPELINE CONTINUOUS DB DELIVERY PIPELINE depends
35. @lacquaviva @filippobosi @ste_monti #DevoxxPL Demo
36. @lacquaviva @filippobosi @ste_monti #DevoxxPL • Development complexity increases • Delivery Governance complexity increases Open points
37. @lacquaviva @filippobosi @ste_monti #DevoxxPL THANK YOU !