CI/CD non-breaking changes exercise - Cork Software Crafters - February 2024

https://www.meetup.com/cork-software-crafters/events/298859591/

Transcript

1. Cork Software Crafters Follow us on Twitter: @CorkSwCraft Meetup: www.meetup.com/cork-software-crafters Slack: softwarecrafters.slack.com/messages/cork Continuous Delivery non-breaking changes exercise 8th February 2024 facilitated by Paulo Clavijo Esteban
2. Software Crafters Communities Worldwide softwarecrafters.org
3. follow Cork Software Crafters @CorkSwCraft
4. We are always looking for new ideas, new talks and mostly new hands-on sessions for our meetups. Contact us if you want to collaborate!
5. 6:15pm - Welcome. 6:30pm - Introduction. 6:40pm - Hands-on time. We will work in groups on the exercise. 7:50pm - Retrospective. Paulo Clavijo (@pclavijo) Agenda Cork Software Crafters
6. Introduction
7. Non-breaking changes exercise We favour a Continuous Integration/Continuous Delivery approach, integrating changes with the trunk branch and deploying all the way to production multiple times per day. Our pull requests should always take in consideration a blue/green deployment. But there is a multitude of "breaking changes" that can make our life complicated :-(. Paulo Clavijo @pclavijo Cork Software Crafters In this group exercise, we will go through different scenarios for breaking code changes (UI changes, API changes, Database changes, Message Queue changes, Performance improvements ...) and discuss together what could be the right steps to integrate changes in a non-breaking way.
8. Continous Delivery “Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time… You achieve continuous delivery by continuously integrating the software done by the development team, building executables, and running automated tests on those executables to detect problems… You push those executables into increasingly production-like environments to ensure that the software will work in production.” - Martin Fowler Paulo Clavijo @pclavijo Cork Software Crafters
9. Trunk-Based Development Paulo Clavijo @pclavijo Cork Software Crafters http://trunkbaseddevelopment.com
10. Blue-green deployment Paulo Clavijo @pclavijo Users Router Green environment Blue environment create a new green environment and switch from Blue (old) to Green (new) example.com new-example.com code v1.0 code v1.1 Cork Software Crafters
11. Practice Time
12. We are here to learn . Cork Software Crafters Paulo Clavijo (@pclavijo) Focus on doing it right . We need to slow down . Collaboration . Coding Dojo Mindset
13. Collaboration • We will work in groups of 4 to 5 people • 5 min for scenario • 3 min to share ideas and solutions • Swap groups Paulo Clavijo @pclavijo
14. Keep in mind • Service is available continuously • Deployment requires parallel versions (blue/green) • Deploy the smallest changes you can • We do not deploy multiple services at once (avoid coupled changes) Paulo Clavijo @pclavijo Cork Software Crafters
15. UI changes
16. Scenario UI-1 (example) Replacing and old home page by a new and redesigned one on a SPA UI. - The current home page view has /my-dashboard as route. This route is also the default route in the app, users are redirected to it when they just use the base url www.app.com. - The new page will have multiple components and actions, it will take around 2 week to develop. - Users can not see the new page until it is completely finished. Paulo Clavijo @pclavijo Cork Software Crafters
17. Scenario UI-2 New feature to allow product reviews. - Customers should be able to see reviews for a product on the produt detail view. - Logged-in customers can add their own review for a product. - This new feature will take 1 week to develop. - Users can not see the new feature until it is completely finished. Paulo Clavijo @pclavijo Cork Software Crafters
18. Hide new functionality until it is finished “Shipping semicompleted functionality along with the rest of your application is a good practice because it means you're always integrating and testing the entire system as it exists at any time. This makes planning and delivering the entire application much easier…” - Jez Humble and David Farley Continuous Delivery (p348) Paulo Clavijo @pclavijo Cork Software Crafters
19. Feature Toggles “A feature toggle is a technique in software development that attempts to provide an alternative to maintaining multiple source-code branches (known as feature branches), such that a feature can be tested even before it is completed and ready for release. Feature toggle is used to hide, enable or disable the feature during run time. For example, during the development process, a developer can enable the feature for testing and disable it for other users.” - Wikipedia Paulo Clavijo @pclavijo
20. API changes
21. Scenario API-1 Make the paramater category required in an existing /products endpoint. - The parameter allows filtering products GET /products?category=home Paulo Clavijo @pclavijo Cork Software Crafters
22. Scenario API-2 Rename the request query parameter pcod to productId in a existing /products endpoint. - The parameter is used for retrieving a single product GET /products?pcod=12345 fun getProduct(@RequestParam pcod: String): Product { return productsService.getProduct(pcod) } Paulo Clavijo @pclavijo Cork Software Crafters
23. Advice and considerations API breaking changes: • adding a new or modifying an existing validation to an existing resource • requiring a parameter that wasn’t required before • changing existing error response codes/messages • modifying the expected payload • changing the data type of an existing field • changing supported filtering on existing endpoints • renaming a field or endpoint • changing the URL structure of an existing endpoint … Paulo Clavijo @pclavijo Cork Software Crafters
24. Parallel Change “Parallel change, also known as expand and contract, is a pattern to implement backward-incompatible changes to an interface in a safe manner, by breaking the change into three distinct phases: expand, migrate, and contract. ” - Martin Fowler Paulo Clavijo @pclavijo Cork Software Crafters
25. Parallel Change Paulo Clavijo @pclavijo Cork Software Crafters
26. Parallel Change Paulo Clavijo @pclavijo Cork Software Crafters
27. Parallel Change Paulo Clavijo @pclavijo Cork Software Crafters
29. DB changes
30. Scenario DB-0 (example) Adding a new property category to the existing Tag entity. - We are using a table on a relational database. - Attribute not related to other tables. - Change in code and table column. Paulo Clavijo @pclavijo Cork Software Crafters
31. Scenario DB-0 (example) Adding a new property category to the existing Tag entity. - We are using a table on a relational database. - Attribute not related to other tables. - Change in code and table column. Possible Solution: A single deploy: + ALTER TABLE tag + ADD COLUMN IF NOT EXISTS category varchar(20) NOT NULL DEFAULT 'Other'; @Entity class Tag( @Id val id: String, val name: String, + @Enumerated(EnumType.STRING) + val category: TagCategory = TagCategory.Other, Paulo Clavijo @pclavijo
32. Scenario DB-1 Rename the attribute informationCompliance to regulatoryRequirements in the Application entity. - We are using a table on a relational database. - Attribute not related to other tables. - Change in code and table column. Paulo Clavijo @pclavijo Cork Software Crafters
33. Scenario DB-2 Transform the attribute name to firstname and lastname in the Customer entity. - We are using a table on a relational database. - Attribute not related to other tables. - Change in code and table column. - Data already exist in the table. - No other service connects to this database. Paulo Clavijo @pclavijo Cork Software Crafters
34. Advice and considerations Database changes that can cause downtime or errors: • removing a column • adding a column with a default value • renaming a column • changing the type of a column • setting NOT NULL on an existing column • renaming a table • adding a foreign key • backfilling data • adding an index non-concurrently ... Paulo Clavijo @pclavijo Cork Software Crafters
35. Advice and considerations • Read chapter 5 (Evolutionary Data) from Building Evolutionary Architectures book. • Take a look at github.com/ankane/strong_migrations#dangerous-operations Paulo Clavijo @pclavijo Cork Software Crafters
37. Large Scale changes
38. Scenario LS-1 Change the persistence technology from Relational Database to a NoSql store. We will start with the Customer entity. - Other entities only refer to Customer using its ID. - Customer is not used in joins. Paulo Clavijo @pclavijo Cork Software Crafters
39. Branch by Abstraction “Branch by Abstraction is a technique for making a large-scale change to a software system in gradual way that allows you to release the system regularly while the change is still in-progress.” - Martin Fowler Paulo Clavijo @pclavijo
40. Branch by Abstraction “Branch by Abstraction is a technique for making a large-scale change to a software system in gradual way that allows you to release the system regularly while the change is still in-progress.” - Martin Fowler Paulo Clavijo @pclavijo
41. Message queue changes
42. Scenario Q-1 - Server A sends a message OrderPlaced with a field old_field using a queue to Server B and Server C. - We need to change old_field to new_field. Server A queue Message Server B Server C Paulo Clavijo @pclavijo Cork Software Crafters
43. Postel’s Law and the Tolerant Reader pattern “When communicating between systems, Postel’s Law and the Tolerant Reader pattern state that an implementation should be liberal when receiving data and conservative when sending data. This makes system integration less painful because changes to data fields ignored by the receiving system becomes seamless to the overall integration.” - Johnsson, Deogun and Sawano Secure By Design Paulo Clavijo @pclavijo Cork Software Crafters
44. Performance changes
45. Scenario P-1 We have a service that calculates daily delivery routes for postmen. It uses a CPU and memory inefficient route calculation logic, we want to reimplement the calculation with a new algorithm that improves performance. - Calculation is complex with lots of corner cases. - We expect the same outcome from old and new logic. - It will take around 2 week to develop. - This service is highly important for the business. - Paulo Clavijo @pclavijo Cork Software Crafters
46. Advice and considerations • Consider Canary Release • Consider Branch By Abstraction with verification (involves verifying that the two implementations return the same results to requests) • Take a look at Scienties github.com/github/scientist (a library for carefully refactoring critical paths) Paulo Clavijo @pclavijo Cork Software Crafters
47. Verify Branch By Abstraction “... a run-time configuration toggle delegates to a verifying implementation that calls both components with the same inputs and fails fast if they do not produce the same output. This reduces the feedback loop for old and new component incompatibilities, and increases confidence in our evolving architecture.” - Steve Smith Paulo Clavijo @pclavijo
48. Key Takeaway: Limit WIP Think in ways to limit work-in-progress (WIP) at any level. • Limit the time you are in “Red”. • Limit change size and and time on your commits, branches, MRs, Deployments, Releases. Paulo Clavijo @pclavijo Cork Software Crafters
49. References - Taller Parallel Changes, Eduardo Ferro. Software Crafters Barcelona 2018. - BlueGreenDeployment, Martin Fowler - BranchByAbstraction, Martin Fowler - Parallel Change, Danilo Sato - Parallel Change Refactoring, Oleksii Fedorov - Parallel Change (Parallel Design), Jakub Holy - Effective Design (slides 8 and 9), Kent Beck - The Limited Red Society, Joshua Kerievsky - StranglerFigApplication, Martin Fowler - CanaryRelease, Danilo Sato Paulo Clavijo (@pclavijo) Cork Software Crafters
50. Recommended Books - Continuous Delivery by Jez Humble, David Farley. - Building Evolutionary Architectures: Support Constant Change by Neal Ford, Rebecca Parsons and Patrick Kua. - The DevOps Handbook by Gene Kim, Jez Humble, Patrick Debois, John Willis (chapter 12). - The Mikado Method by Ola Ellnestam. - Working Effectively with Legacy Code by Michael Feathers. - Refactoring by Martin Fowler. Paulo Clavijo (@pclavijo) Cork Software Crafters
51. Interesting Tools - Strong Migrations. https://github.com/ankane/strong_migrations (a libray to catch unsafe migrations in development) - Scientist https://github.com/github/scientist (a library for carefully refactoring critical paths) Paulo Clavijo (@pclavijo) Cork Software Crafters
52. Retrospective