Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Continuous Delivery for Legacy Code - DevOpsCon Winter 2022

December 06, 2022

Continuous Delivery for Legacy Code - DevOpsCon Winter 2022

This is based on a true story.

My day job is software archeology. I find joy in recovering and analyzing code bones and culture as well as making the skeleton walk again. A short time ago, however, I was confronted with the most horrible code base I have ever seen. This talk is about how we managed to save it and achieve bi-weekly deployments with a high level of confidence.

Five million lines of code in multiple languages (Classic ASP, .NET, VBScript, VBA, JavaScript, T-SQL, PL-SQL) in one monolith. The business logic stretched from the UI (WebForms, Scripting, SQL Queries) down to the database (Stored Procedures), there was no test coverage and an enormous amount of hidden coupling. A version control system was not used, we had no test environment, deployments required developers to copy their local compilation to production and multiple customer installations are supported by uncommenting and commenting code.

Together we will explore what to do when you inherit such a thing: how to identify hotspots, find hidden coupling, explore how connascence can help you, ways to test as well as refactor and how to achieve a regular deployment schedule.


December 06, 2022

More Decks by Richard

Other Decks in Programming


  1. Continuous Delivery for Legacy Code 06.12.22 Richard Gross (he/him) IT-Tailor

    Archaeologist Auditor richargh.de/ speakerdeck.com/richargh @arghrich
  2. Slides by @arghrich 3 Dave Farely, author of the best-selling

    book “Continuous Delivery” The Quality of a System is Defined by Our Ability to Change it! Dave Farley @davefarley77
  3. Slides by @arghrich A story of the worst software I

    have ever and you will never see. 5 And we still got to Continuous Delivery J
  4. Slides by @arghrich Along comes a customer … 6 Audit

    Icons by Franziska Haaf binarykitten.de COTS „Apocalypto“ Please Audit Customer Me
  5. Slides by @arghrich 4M LoC Delivery 7 Audit Excel Webservice

    Website Fat Client Jobs Database T-SQL JavaScript VBA C# Asp.NET + C# Classic Asp + VBScript WinForms + C# MIA 5M No Code delivered
  6. Slides by @arghrich Cycle Time of 6 months 9 Audit

    weeks Dev Machine Cp from Prod Develop Compile Don‘t Test Release Cp to Prod 1 hour Bugfixing Return to Start 5 months 6 months
  7. Slides by @arghrich „Interesting“ Deployment 15 years of Uncontrolled growth

    Horrible Cycle Time No Tests Conflated Logic, UI & SQL Too much to fix Not Versioned & Doesn‘t Compile Icon by Franziska Haaf binarykitten.de 10
  8. Slides by @arghrich 11 Icons by Franziska Haaf binarykitten.de COTS

    „Apocalypto“ In-house „Fenix“ Buy I‘ll Customer Money
  9. Ken Mugrage @kmugrage Legacy Code is code that’s too scary

    to update and too profitable to delete. It‘s not CD if you can‘t deploy to production right now Dylan Beattie @dylanbeattie Slides by @arghrich 14
  10. • Code • Configuration • Deployment scripts • Infrastructure Provisioning

    • Alerting • Monitoring • Documentation • Onboarding.sh Scripts • … 17 Put everything into version control Slides by @arghrich 17 Git
  11. Slides by @arghrich Scripts to Provision Infrastructure Same but parameterized

    Scripts for TST, STG and PRD • Machine • Web Server • Application Server • Database • Email Server • FTP Server • Pipeline Server 20 Phoenix Infra: With one click we can (re)create any infrastructure Gives us
  12. Slides by @arghrich Deploy via Pipeline to Test Env 21

    Commit Stage Compile Git Deploy To TST (Cont.) STG (Click) Smoke Test Deploy
  13. Slides by @arghrich Can we deploy this to production right

    now? We have a button to do it, but do we want to? 22
  14. Visualize metrics with CodeCharta buildings SomeService.cs Lines of code Complexity

    number of authors Icons by Fontawesome Open-Source Tool CodeCharta: https://maibornwolff.github.io/codecharta/ Slides by @arghrich 28
  15. Slides by @arghrich No tool to rule them all 30

    Shamelessly putting CodeCharta first 1 Quality Views by Colin Breck Continuously Periodically Structure101 SonarQube CodeCharta CodeScene ArchUnit TSArch SonarArchitect Quality Views 1
  16. Slides by @arghrich 31 1 By Matthias Verraes 2 Graphana

    Graphs + Push Gateway Photo by Pixabay from Pexels Count Fixed Windows 2 Three Post-it‘s To Refactor 1 Slides by @arghrich
  17. Versioned & Automated Compile Automated Deployment We know what to

    focus on No Tests Conflated Logic, UI & SQL
  18. Slides by @arghrich Start with tests for critical use cases

    and hotspots • ~10 E2E Tests For critical use cases • DB-Integration Tests For direct-db-access hotspots that you will refactor • Unit tests (should give 80% confidence) For refactored hotspots For no-direct-db-access hotspots For new code 33
  19. Slides by @arghrich Categorization test1: Write Tests for code that

    you don’t understand 34 1 Also called approval or golden master test. 2 Put system in a known (database) state is also an input. 3 Log messages you add are also an output. vX - redesigned v1 - legacy Input for UI/API/DB 2 Output 3 Output 3 Continuous Compare
  20. Slides by @arghrich Understand Coupling and Cohesion High Coupling Low

    Cohesion Low Coupling High Cohesion 35 Unknown source
  21. Slides by @arghrich Uncouple your Code 36 Delete Obsolete (unreachable

    or unused) Code Categorization Test, then Refactor Code Sprout new tested Code Break Dependencies (Seams) Strangler Fig Unfixable code Feature Flag Legacy Code Write ArchUnit Tests
  22. Slides by @arghrich Can we deploy this to production right

    now? Hmm, it’s somewhat tested but can we really be sure nothing breaks in production? 37
  23. 38 Automated Deployment Slides by @arghrich Versioned & Automated Compile

    Seperated Logic, UI & SQL In Hotspots Critical Use Case Tests Growing Unit Tests Unit tested new code We know what to focus on Horrible Cycle Time
  24. Slides by @arghrich RED Metrics 40 1 https://thenewstack.io/monitoring-microservices-red-method/ Community Dashboard

    https://community.grafana.com/t/how-can-i-configure-red-dashboard/69885 R -> Request Rate E -> Errors / second D -> Duration per request
  25. Slides by @arghrich New Cycle Time, without bug-cycle 41 Slide

    adapted from Dave Farley Commit Stage Compile Unit Test Lint Security Docker Release Git Monitor Logs Events Metrics Alerts Automated Performance testing 20 min Deploy To TST (Cont.) STG (Click) Smoke Test <1 min Manual Testing 30 min Automated Acceptance Testing User Acceptance Test DB-Integration Test 20 min 5 min 60 min
  26. Slides by @arghrich Can we deploy this to production right

    now? Yes. Depending on where we make the change, even within 30min. 43
  27. Slides by @arghrich A Feature Factory helps no one 48

  28. Slides by @arghrich 49 Henrik Kniberg @henrikkniberg Culture is the

    sum of everyone's attitude and actions, so model the behaviour you want to see! Observe behavior: • Who does the management promote? • Is there fear of failure? • Is quality appreciated or story points or …? • What is the delivery „process“? • Is there some kind of learning exchange?
  29. Slides by @arghrich Make the desired behavior … 50 Obvious

    • Define Principles together • Shared purpose Attractive • Reduce (time) pressure • (Refactoring) score Atomic Habits 4 Laws of behavior change Easy • Tech-coach • Reduce friction Satisfying • Pair up • Never break the chain (twice)
  30. Slides by @arghrich CD = Working in a way that

    software is always in a releasable state It’s not about the tools, even though they are getting better all the time 52
  31. Slides by @arghrich Thanks 53 IT-Tailor Richard Gross (he/him) Archaeologist

    Auditor richargh.de/ speakerdeck.com/richargh @arghrich Works at maibornwolff.de/ Tailored Software by People who Care. DE TN ES
  32. Slides by @arghrich And also 10 separate codebases 60 …

    Oceanic Airlines Soylent Corp Frobozz Co. Umbrella Corp Similar C/P Copy/ Paste Feature Copy/Paste