Continuous Delivery for Legacy Code (v2) 🇬🇧 @Build Stuff 2019

This is based on a true story.

My day job is software archeology. I find joy in recovering and analyzing code bones and culture as well as making the skeleton walk again. A short time ago, however, I was confronted with the most horrible code base I have ever seen. This talk is about how we managed to save it and achieve bi-weekly deployments with a high level of confidence.

Five million lines of code in multiple languages (Classic ASP, .NET, VBScript, VBA, JavaScript, T-SQL, PL-SQL) in one monolith. The business logic stretched from the UI (WebForms, Scripting, SQL Queries) down to the database (Stored Procedures), there was no test coverage and an enormous amount of hidden coupling. A version control system was not used, we had no test environment, deployments required developers to copy their local compilation to production and multiple customer installations are supported by uncommenting and commenting code. Together we will explore what to do.

https://buildstuff2019.sched.com/event/UzgO/richard-gross-continuous-delivery-for-legacy-code

Transcript

1. Continuous Delivery for Legacy Code 13.11.19 Audit Software Archeology Develop Small Icons by Fontawesome Many Icons by Franziska Haaf binarykitten.de richargh.de/ speakerdeck.com/richargh @arghrich Richard Gross pronoun.is/he
2. Dylan Beattie @dylanbeattie Legacy Code is code that’s too scary to update and too profitable to delete. Michael Feathers @mfeathers Legacy Code is code without tests. 2 Slides by @arghrich
3. Continuous Delivery Legacy Code is code that’s too scary to update and too profitable to delete. If somebody thinks of a good idea, how do we [make it feasable to] deliver it to users as quickly as possibly? 3 Dylan Beattie @dylanbeattie Slides by @arghrich
4. 4 Photo by Anne-sophie Parent from Pexels Spring Slides by @arghrich
5. 5 COTS „Apocalypto“ Please Audit Icons by Franziska Haaf binarykitten.de Slides by @arghrich Spring: Audit
6. 6 Excel Webservice Website Fat Client Jobs Database T-SQL JavaScript VBA C# Asp.NET + C# Classic Asp + VBScript WinForms + C# MIA No Code delivered Slides by @arghrich Spring: Audit 5M 4M LoC Delivery
7. Replicated Databases 7 Excel Webservice Website Fat Client Jobs DB1 DB2 Replication Dependency Slides by @arghrich Spring: Audit
8. 8 Icons by Franziska Haaf binarykitten.de Doesn‘t Compile BL & DB Everywhere Interesting Deployment 15 years old No Tests Every Duplication Slides by @arghrich Spring: Audit
9. COTS „Apocalypto“ In-house „Fenix“ 9 Buy I‘ll Icons by Franziska Haaf binarykitten.de Slides by @arghrich Spring: Audit
10. 10 In-house „Fenix“ Please Fix Icons by Franziska Haaf binarykitten.de Slides by @arghrich Spring: Audit
11. Oh Boy
12. We could just … 12 Icons by Franziska Haaf binarykitten.de Slides by @arghrich Spring: Audit
13. 13 Photo by Levi Damasceno from Pexels Slides by @arghrich Spring: Audit
14. The True Cost of Rewrites Features Releases over time Catch Up Missing Features Sub-Par Parity Enhancements Planned Rewrite Actual Features Undiscovered Scope Old App Adoption Original Article by Doug Bradbury The True Cost of Rewrites Slides by @arghrich
15. 15 Photo by Pixabay from Pexels Slides by @arghrich Early Summer
16. How did they get here? 16 Slides by @arghrich Early Summer
17. GIGO 17 Ideas Product Process (Team) Spirit Predicts Slides by @arghrich Early Summer
18. GIGO 18 Ideas Product Process (Team) Spirit Fix Predicts Slides by @arghrich Early Summer
19. How did we not get here? 19 Slides by @arghrich Early Summer
20. 20 We needed a significant change in team and culture Slides by @arghrich Early Summer: Spirit
21. 21 Henrik Kniberg @henrikkniberg Culture is the sum of everyone's attitude and actions, so model the behaviour you want to see! Observe behavior: • Who does the management promote? • Is there fear of failure? • Is quality appreciated or story points or …? • What is the delivery „process“? • Is there some kind of learning exchange? Slides by @arghrich Early Summer: Spirit
22. 4 Laws of behavior change Make it … Obvious • Define Principles together • Shared purpose Attractive • Reduce (time) pressure • (Refactoring) score 22 Atomic Habits Easy • Tech-coach • Reduce friction Satisfying • Pair up • Never break the chain (twice) Slides by @arghrich Early Summer: Spirit
23. 23 We built a new Co-located Cross-functional Team with shared DEV TEST OPS Key Accounter & principles purpose Slides by @arghrich Early Summer: Spirit
24. Our Legacy CD Principles • Build quality in • Focus on current tree, not on the forest • Tackle fear head on • Focus on reducing batch size • Decouple Deployment and Release • Computers perform repetitive tasks, people solve problems Four Principles of Low-Risk Software Releases by Jez Humble CD Principles by Jez Humble The Why 24 Slides by @arghrich Early Summer: Spirit
25. 25 Principles provide adaptability Slides by @arghrich Early Summer: Spirit
26. 26 Icons by Franziska Haaf binarykitten.de In-house „Fenix“ Please Cloud Slides by @arghrich Early Summer: Spirit
27. Oh Boy
28. Fix 28 Product Process (Team) Spirit Predicts Slides by @arghrich Summer
29. Local Env Source Repository Artifact Repository Commit Stage Acceptance Manual Test Env … Staging Env Prod Env Self-Service Automatic Feedback (Tag Artifact) Dave Farley - Optimising Continuous Delivery Slides by @arghrich Our CD Pipe
30. • Don‘t Branch • Don‘t Branch • Don‘t Branch • Never Branch to Refactor • Branches indicate Ci =Continuous Isolation Strategy by Dave Farley Our Branching Strategy 30 Slides by @arghrich Summer: Process
31. • Code • Config • Deployment scripts • Infrastructure scripts • Docs • … 31 We versioned and tested Everything Slides by @arghrich Summer: Process
32. They used File Version Anti-Practice 32 File tree from actual project Slides by @arghrich Summer: Process
33. 33 Photo by Daniel Frank from Pexels Slides by @arghrich Autumn
34. Fix 34 Product Process (Team) Spirit Slides by @arghrich Autumn
35. Where do we start? 35 Slides by @arghrich Autumn
36. Art by Zaufishan 36 Slides by @arghrich Autumn: Product
37. github.com/MaibornWolff/codecharta 37 Shameless Slides by @arghrich Autumn: Product
38. Visualize metrics with buildings 38 AbstractService.java Lines of code Complexity number of authors Icons by Fontawesome Very Shameless Slides by @arghrich Autumn: Product
39. Find potential hotspots 39 Still Shameless Slides by @arghrich Autumn: Product
40. No tool to rule them all 40 Slides by @arghrich Continuously Periodically Structure101 SonarQube CodeCharta CodeScene ArchUnit TSArch SonarArchitect Quality Views by Colin Breck Autumn: Product Baseline Quality Views Shamelessly putting CodeCharta first
41. 41 Photo by Pixabay from Pexels Slides by @arghrich Autumn: Product Baseline Counting Fixed Windows
42. 42 vX - redesigned v1 - legacy Input for UI/API/DB Output Output Continuous Compare Slides by @arghrich Autumn: Product Baseline Golden Master Characterizes Behavior before Refactoring
43. 43 Slides by @arghrich Autumn: Product Baseline Golden Master Cheatsheet Input • Have Lots & Check coverage • Capture from Production • Generate • Try Testrecorder Output • Log messages • Datalake • Database Hidden Inputs • Date • Randoms When • Before refactorings • Generate new output when needed
44. How do we refactor what we don‘t understand? 44 Slides by @arghrich Autumn: Product Baseline
45. 45 Slides by @arghrich Autumn: Product Baseline Emily Bache @emilybache „Nopefactoring“ The No-thinking refactoring“ Advanced Testing & Refactoring Techniques • Lift-up conditional • Split to Classes Llewellyn Falco @LlewellynFalco Cutting Code Quickly
46. 46 Icons by Franziska Haaf binarykitten.de In-house „Fenix“ Add Feature Slides by @arghrich Autumn
47. Fuuuu 47 Photo by Bartek Wojtas from Pexels Slides by @arghrich Autumn
48. • Write categorization, if possible function Tests • Safely verify it can handle production • Divide code into stable and unstable • Understand Code Domain 48 Slides by @arghrich Autumn: Conquer the Fear Conquer the fear
49. Learn Domain with Reverse Object Mother • Start application with empty database and without domain model • Click through a UseCase • Analyse exceptions and errors „App needs at least an object A with this field“ • Expand Domain Modell with your finding • Create required state in the DB with your model • Document finding as categorization test • Repeat 49 Slides by @arghrich Autumn: Conquer the Fear
50. 50 Photo by Emmet from Pexels Slides by @arghrich Autumn: Conquer the Fear
51. • Direct Read/Write à Facades • Classes à Interfaces + Dependency Injection • Fire&Forget calls à Events • Obsolete/duplicate tables à Table Views • Dynamic GET/POST à Explicit constants 51 Photo by Krizjohn Rosales from Pexels Isolate Stable from Unstable Working Effectively with Legacy Code Slides by @arghrich Autumn: Conquer the Fear
52. Sprout quality code 52 Photo by Alex Fu from Pexels Working Effectively with Legacy Code Table View Interface Facade Event Slides by @arghrich Autumn: Conquer the Fear
53. Use Strangler Application for bigger features Picture by Paul Hammant Great talk Strangling the Monolith With a Data-Driven Approach 53 Slides by @arghrich Autumn: Conquer the Fear
54. 54 Photo by Riccardo Bresciani from Pexels Slides by @arghrich Winter
55. Slides by @arghrich vX - redesigned 55 Icons by Fontawesome Load Balancer v1 - legacy Continuous Compare Same number of: • 200? • 404? • 5xx? • … Winter: Launch confidence Dark Launch to gain confidence
56. Remove any obsolete code in Tech || Business 56 Photo by Stas Knop from Pexels Slides by @arghrich Winter: Remove waste Don‘t fix what doesn‘t need fixing
57. Finding obsolete code • Add Analytics / Activity-Log for your dark launch pages • User surveys • IDE suggestions • grep FROM, JOIN, INSERT, UPDATE, DELETE in .asp and check if that code is reachable • Legacy Toggles • Forget to deploy parts of your application 57 Slides by @arghrich Winter: Remove waste
58. 58 Photo by Irina Iriser from Pexels Slides by @arghrich Next Spring: Summary
59. Legacy Delivery 59 Photo by pixabay from Pexels Before • 2 Weeks for a feature • 5 months until it works After • 4 Weeks for a feature • Then it works Slides by @arghrich Next Spring: Summary
60. Was it worth it? • Users had capitulated à then we started fixing bugs • More Stable then before • We couldn‘t have reimplemented this from scratch • à None can describe what „same as before“ means • à Even if, it would have taken more than a year • Seamless migration from COTS to in-house à min. feature freeze 60 Slides by @arghrich Next Spring: Summary
61. Would I do it again? 61 Slides by @arghrich Next Spring: Summary
62. Hell Yes We had • Autonomy • Mastery • Purpose 62 Model from „Drive“ by Daniel Pink Slides by @arghrich Next Spring: Summary
63. 63 (Team) Spirit • Conquer the fear • CD Principles • Build quality in • Shared purpose Product • Isolate stable from unstable • Golden Master • Strangler • Feathers Process • CD Pipeline • Infrastructure Automation Slides by @arghrich Next Spring: Summary
64. Ken Mugrage @kmugrage It‘s not CD if you can‘t deploy to production right now 64 Slides by @arghrich Next Spring: Summary
65. Done Questions, Remarks, Sympathy? J 30 Mio 510 Employees 9th year 23 Nations 300 Weeks Munich, Augsburg, Berlin, Frankfurt @arghrich speakerdeck.com/richargh richargh.de/ Most of our projects aren‘t like this J Richard Gross pronoun.is/he
66. Reading • https://trello.com/b/Fdd876S8/continuous-delivery-checklist- template • I did not think I would show this slide 66 Slides by @arghrich