Continuous Delivery for Legacy Code - NDC London

Presented at NDC London 2024

------------------------------

This is based on a true story.

My day job is software archeology. I find joy in recovering and analyzing code bones and culture as well as making the skeleton walk again. A short time ago, however, I was confronted with the most horrible code base I have ever seen. This talk is about how we managed to save it and achieve bi-weekly deployments with a high level of confidence.

Five million lines of code in multiple languages (Classic ASP, .NET, VBScript, VBA, JavaScript, T-SQL, PL-SQL) in one monolith. The business logic stretched from the UI (WebForms, Scripting, SQL Queries) down to the database (Stored Procedures), there was no test coverage and an enormous amount of hidden coupling. A version control system was not used, we had no test environment, deployments required developers to copy their local compilation to production and multiple customer installations are supported by uncommenting and commenting code.

Together we will explore what to do when you inherit such a thing: how to identify hotspots, find hidden coupling, explore how connascence can help you, ways to test as well as refactor and how to achieve a regular deployment schedule.

Transcript

1. Continuous Delivery for Legacy Code 01.02.24 Richard Gross (he/him) Developer Archaeologist Auditor richargh.de/ speakerdeck.com/richargh @arghrich
2. Slides by @arghrich What are the properties of low- quality software? 2
3. Slides by @arghrich 3 Dave Farely, author of the best-selling book “Continuous Delivery” The Quality of a System is Defined by Our Ability to Change it! Dave Farley @davefarley77
4. Slides by @arghrich Bad (legacy) software is software where we no longer have any options 4
5. Slides by @arghrich A story of the worst software I have ever and you will never see. 5 And we still got to Continuous Delivery J
6. Slides by @arghrich Along comes a customer … 6 Audit Icons by Franziska Haaf binarykitten.de COTS „Apocalypto“ Please Audit Customer Me
7. Slides by @arghrich 4M LoC Delivery 7 Audit Excel Webservic e Website Fat Client Jobs Database T-SQL JavaScript VBA C# Asp.NET + C# Classic Asp + VBScript WinForms + C# MIA 5M No Code delivered
8. Slides by @arghrich Copy-Based Version ControlTM with production server as origin 8 Audit File tree from actual project
9. Slides by @arghrich Cycle Time of 6 months 9 Audit weeks Dev Machine Cp from Prod Develop Compile Don‘t Test Release Cp to Prod 1 hour Bugfixing Return to Start 5 months 6 months
10. Slides by @arghrich Too much to fix 10 Audit Icon by Franziska Haaf binarykitten.de „Interesting“ Deployment 15 years of Uncontrolled growth Horrible Cycle Time No Tests Conflated Logic, UI & SQL Not Versioned & Doesn‘t Compile
11. Slides by @arghrich 11 Icons by Franziska Haaf binarykitten.de COTS „Apocalypto“ In-house „Fenix“ Buy I‘ll Customer Money
12. Slides by @arghrich 12 Icons by Franziska Haaf binarykitten.de In-house „Fenix“ Please Fix Me Customer
13. Oh Boy
14. Ken Mugrage @kmugrage Legacy Code is code that’s too scary to update and too profitable to delete. It‘s not CD if you can‘t deploy to production right now Dylan Beattie @dylanbeattie Slides by @arghrich 14
15. Slides by @arghrich Can we deploy this to production right now? Nope 15
16. 16 Slides by @arghrich Not Versioned & Doesn‘t Compile Icon by Franziska Haaf binarykitten.de 16
17. • Code • Configuration • Deployment scripts • Infrastructure Provisioning • Alerting • Monitoring • Documentation • Onboarding.sh Scripts • … 17 Put everything into version control Slides by @arghrich 17 Git
18. Slides by @arghrich Establish a Pipeline 18 Commit Stage Compile Git
19. Slides by @arghrich Versioned & Automated Compile „Interesting“ Deployment 19
20. Slides by @arghrich Scripts to Provision Infrastructure Same but parameterized Scripts for TST, STG and PRD • Machine • Web Server • Application Server • Database • Email Server • FTP Server • Pipeline Server 20 Phoenix Infra: With one click we can (re)create any infrastructure Gives us
21. Slides by @arghrich Deploy via Pipeline to Test Env 21 Commit Stage Compile Git Deploy To TST (Cont.) STG (Click) Smoke Test
22. Slides by @arghrich Can we deploy this to production right now? We have a button to do it, but do we want to? 22
23. Automated Deployment Slides by @arghrich Versioned & Automated Compile 15 years of Uncontrolled growth
24. Slides by @arghrich Don’t “Fix everything at once, mindset”TM 24
25. Slides by @arghrich Do the “Hotspot Mindset”TM 25
26. Slides by @arghrich Finding Hotspots 26
27. Art by Zaufishan Slides by @arghrich 27
28. Visualize metrics with CodeCharta buildings SomeService.cs Lines of code Complexity number of authors Icons by Fontawesome Open-Source Tool CodeCharta: https://maibornwolff.github.io/codecharta/ Slides by @arghrich 28
29. Find potential hotspots Open-Source Tool CodeCharta: https://maibornwolff.github.io/codecharta/ Slides by @arghrich 29 Lot‘s of code, very complex, and everyone has to change it
30. Slides by @arghrich No tool to rule them all 30 Shamelessly putting CodeCharta first 1 Quality Views by Colin Breck Continuously Periodically Structure101 SonarQube CodeCharta CodeScene ArchUnit TSArch SonarArchitec t Quality Views 1
31. Slides by @arghrich 31 1 By Matthias Verraes 2 Graphana Graphs + Push Gateway Photo by Pixabay from Pexels Count Fixed Windows 2 Three Post-it‘s To Refactor 1 Slides by @arghrich
32. Versioned & Automated Compile Automated Deployment We know what to focus on No Tests Conflated Logic, UI & SQL
33. Slides by @arghrich Start with tests for critical use cases and hotspots • ~10 E2E Tests For critical use cases • DB-Integration Tests For direct-db-access hotspots that you will refactor • Unit tests (should give 80% confidence) For refactored hotspots For no-direct-db-access hotspots For new code 33
34. Slides by @arghrich Categorization test1: Write Tests for code that you don’t understand 34 1 Also called approval or golden master test. 2 Put system in a known (database) state is also an input. 3 Log messages you add are also an output. vX - redesigned v1 - legacy Input for UI/API/DB 2 Output 3 Output 3 Continuous Compare
35. Slides by @arghrich Understand Coupling and Cohesion High Coupling Low Cohesion Low Coupling High Cohesion 35 Unknown source
36. Slides by @arghrich Uncouple your Code 36 Delete Obsolete (unreachable or unused) Code Categorization Test, then Refactor Code Sprout new tested Code Break Dependencies (Seams) Strangler Fig Unfixable code Feature Flag Legacy Code Write ArchUnit Tests
37. Slides by @arghrich Can we deploy this to production right now? Hmm, it’s somewhat tested but can we really be sure nothing breaks in production? 37
38. 38 Automated Deployment Slides by @arghrich Versioned & Automated Compile Seperated Logic, UI & SQL In Hotspots Critical Use Case Tests Growing Unit Tests Unit tested new code We know what to focus on Horrible Cycle Time
39. Slides by @arghrich Do Monitor in Production 39
40. Slides by @arghrich RED Metrics 40 1 https://thenewstack.io/monitoring-microservices-red-method/ Community Dashboard https://community.grafana.com/t/how-can-i-configure-red-dashboard/69885 R -> Request Rate E -> Errors / second D -> Duration per request
41. Slides by @arghrich New Cycle Time, without bug-cycle 41 Slide adapted from Dave Farley Commit Stage Compile Unit Test Lint Security Docker Release Git Monitor Logs Events Metrics Alerts Automated Performance testing 20 min Deploy To TST (Cont.) STG (Click) Smoke Test <1 min Manual Testing 30 min Automated Acceptance Testing User Acceptance Test DB-Integration Test 20 min 5 min 60 min
42. Slides by @arghrich Can we deploy this to production right now? Yes. Depending on where we make the change, even within 30min. 43
43. Slides by @arghrich But there is still a problem… 44
44. Slides by @arghrich How did the previous developers get here? 45
45. Slides by @arghrich Healthy Software 46 Ideas Product Process Mindset Predicts
46. Slides by @arghrich GIGO 47 Ideas Product Process Mindset Predicts
47. Slides by @arghrich A Feature Factory helps no one 48 https://cutlefish.substack.com/p/tbm-4952-your-calendar-your-priorities
48. Slides by @arghrich 49 Henrik Kniberg @henrikkniberg Culture is the sum of everyone's attitude and actions, so model the behaviour you want to see! Observe behavior: • Who does the management promote? • Is there fear of failure? • Is quality appreciated or story points or …? • What is the delivery „process“? • Is there some kind of learning exchange?
49. Slides by @arghrich Make the desired behavior … 50 Obvious • Define Principles together • Shared purpose Attractive • Reduce (time) pressure • (Refactoring) score Atomic Habits 4 Laws of behavior change Easy • Tech-coach • Reduce friction Satisfying • Pair up • Never break the chain (twice)
50. Slides by @arghrich CD is a change in mindset 51
51. Slides by @arghrich CD = Working in a way that software is always in a releasable state It’s not about the tools, even though they are getting better all the time 52
52. Slides by @arghrich Thanks 53 Developer Richard Gross (he/him) Archaeologist Auditor richargh.de/ speakerdeck.com/richargh @arghrich Works for maibornwolff.de/ Tailored Software by People who Care. DE TN ES
53. Slides by @arghrich Backup 55
54. Slides by @arghrich 56
55. Slides by @arghrich 10 customer installations 57 Frobozz Co. Oceanic Airlines … Umbrella Corp Soylent Corp
56. Slides by @arghrich And also 10 separate codebases 58 … Oceanic Airlines Soylent Corp Frobozz Co. Umbrella Corp Similar C/P Copy/ Paste Feature Copy/Paste
57. Slides by @arghrich 10 codebases And 10 database schemata 59 Similar Similar
58. Slides by @arghrich Our Legacy CD Principles • Build quality in • Focus on current tree, not on the forest • Tackle fear head on • Focus on reducing batch size • Decouple Deployment and Release • Computers perform repetitive tasks, people solve problems 60 CD Principles by Jez Humble Four Principles of Low-Risk Software Releases by Jez Humble
59. Slides by @arghrich Our branching strategy • Don‘t Branch • Don‘t Even Branch • Branch and you’re screwed • Never Branch to Refactor • Branches indicate Ci =Continuous Isolation 61
60. Slides by @arghrich The True Cost of Rewrites 62 Original Article by Doug Bradbury The True Cost of Rewrites Features Releases over time Catch Up Missing Features Sub-Par Parity Enhancements Planned Rewrite Actual Features Undiscovered Scope Old App Adoption