Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Continuous Delivery for Legacy Code - DevOpsCon Winter 2022

Richard
December 06, 2022

Continuous Delivery for Legacy Code - DevOpsCon Winter 2022

This is based on a true story.

My day job is software archeology. I find joy in recovering and analyzing code bones and culture as well as making the skeleton walk again. A short time ago, however, I was confronted with the most horrible code base I have ever seen. This talk is about how we managed to save it and achieve bi-weekly deployments with a high level of confidence.

Five million lines of code in multiple languages (Classic ASP, .NET, VBScript, VBA, JavaScript, T-SQL, PL-SQL) in one monolith. The business logic stretched from the UI (WebForms, Scripting, SQL Queries) down to the database (Stored Procedures), there was no test coverage and an enormous amount of hidden coupling. A version control system was not used, we had no test environment, deployments required developers to copy their local compilation to production and multiple customer installations are supported by uncommenting and commenting code.

Together we will explore what to do when you inherit such a thing: how to identify hotspots, find hidden coupling, explore how connascence can help you, ways to test as well as refactor and how to achieve a regular deployment schedule.

Richard

December 06, 2022
Tweet

More Decks by Richard

Other Decks in Programming

Transcript

  1. Continuous Delivery for Legacy Code 06.12.22 Richard Gross (he/him) IT-Tailor

    Archaeologist Auditor richargh.de/ speakerdeck.com/richargh @arghrich
  2. Slides by @arghrich What are the properties of low- quality

    software? 2
  3. Slides by @arghrich 3 Dave Farely, author of the best-selling

    book “Continuous Delivery” The Quality of a System is Defined by Our Ability to Change it! Dave Farley @davefarley77
  4. Slides by @arghrich Bad (legacy) software is software where we

    no longer have any options 4
  5. Slides by @arghrich A story of the worst software I

    have ever and you will never see. 5 And we still got to Continuous Delivery J
  6. Slides by @arghrich Along comes a customer … 6 Audit

    Icons by Franziska Haaf binarykitten.de COTS „Apocalypto“ Please Audit Customer Me
  7. Slides by @arghrich 4M LoC Delivery 7 Audit Excel Webservice

    Website Fat Client Jobs Database T-SQL JavaScript VBA C# Asp.NET + C# Classic Asp + VBScript WinForms + C# MIA 5M No Code delivered
  8. Slides by @arghrich Copy-Based Version ControlTM with production server as

    origin 8 Audit File tree from actual project
  9. Slides by @arghrich Cycle Time of 6 months 9 Audit

    weeks Dev Machine Cp from Prod Develop Compile Don‘t Test Release Cp to Prod 1 hour Bugfixing Return to Start 5 months 6 months
  10. Slides by @arghrich „Interesting“ Deployment 15 years of Uncontrolled growth

    Horrible Cycle Time No Tests Conflated Logic, UI & SQL Too much to fix Not Versioned & Doesn‘t Compile Icon by Franziska Haaf binarykitten.de 10
  11. Slides by @arghrich 11 Icons by Franziska Haaf binarykitten.de COTS

    „Apocalypto“ In-house „Fenix“ Buy I‘ll Customer Money
  12. Slides by @arghrich 12 Icons by Franziska Haaf binarykitten.de In-house

    „Fenix“ Please Fix Me Customer
  13. Oh Boy

  14. Ken Mugrage @kmugrage Legacy Code is code that’s too scary

    to update and too profitable to delete. It‘s not CD if you can‘t deploy to production right now Dylan Beattie @dylanbeattie Slides by @arghrich 14
  15. Slides by @arghrich Can we deploy this to production right

    now? Nope 15
  16. 16 Slides by @arghrich Not Versioned & Doesn‘t Compile Icon

    by Franziska Haaf binarykitten.de 16
  17. • Code • Configuration • Deployment scripts • Infrastructure Provisioning

    • Alerting • Monitoring • Documentation • Onboarding.sh Scripts • … 17 Put everything into version control Slides by @arghrich 17 Git
  18. Slides by @arghrich Establish a Pipeline 18 Commit Stage Compile

    Git
  19. Slides by @arghrich Versioned & Automated Compile „Interesting“ Deployment 19

  20. Slides by @arghrich Scripts to Provision Infrastructure Same but parameterized

    Scripts for TST, STG and PRD • Machine • Web Server • Application Server • Database • Email Server • FTP Server • Pipeline Server 20 Phoenix Infra: With one click we can (re)create any infrastructure Gives us
  21. Slides by @arghrich Deploy via Pipeline to Test Env 21

    Commit Stage Compile Git Deploy To TST (Cont.) STG (Click) Smoke Test Deploy
  22. Slides by @arghrich Can we deploy this to production right

    now? We have a button to do it, but do we want to? 22
  23. Automated Deployment Slides by @arghrich Versioned & Automated Compile 15

    years of Uncontrolled growth
  24. Slides by @arghrich Don’t “Fix everything at once, mindset”TM 24

  25. Slides by @arghrich Do the “Hotspot Mindset”TM 25

  26. Slides by @arghrich Finding Hotspots 26

  27. Art by Zaufishan Slides by @arghrich 27

  28. Visualize metrics with CodeCharta buildings SomeService.cs Lines of code Complexity

    number of authors Icons by Fontawesome Open-Source Tool CodeCharta: https://maibornwolff.github.io/codecharta/ Slides by @arghrich 28
  29. Find potential hotspots Open-Source Tool CodeCharta: https://maibornwolff.github.io/codecharta/ Slides by @arghrich

    29 Lot‘s of code, very complex, and everyone has to change it
  30. Slides by @arghrich No tool to rule them all 30

    Shamelessly putting CodeCharta first 1 Quality Views by Colin Breck Continuously Periodically Structure101 SonarQube CodeCharta CodeScene ArchUnit TSArch SonarArchitect Quality Views 1
  31. Slides by @arghrich 31 1 By Matthias Verraes 2 Graphana

    Graphs + Push Gateway Photo by Pixabay from Pexels Count Fixed Windows 2 Three Post-it‘s To Refactor 1 Slides by @arghrich
  32. Versioned & Automated Compile Automated Deployment We know what to

    focus on No Tests Conflated Logic, UI & SQL
  33. Slides by @arghrich Start with tests for critical use cases

    and hotspots • ~10 E2E Tests For critical use cases • DB-Integration Tests For direct-db-access hotspots that you will refactor • Unit tests (should give 80% confidence) For refactored hotspots For no-direct-db-access hotspots For new code 33
  34. Slides by @arghrich Categorization test1: Write Tests for code that

    you don’t understand 34 1 Also called approval or golden master test. 2 Put system in a known (database) state is also an input. 3 Log messages you add are also an output. vX - redesigned v1 - legacy Input for UI/API/DB 2 Output 3 Output 3 Continuous Compare
  35. Slides by @arghrich Understand Coupling and Cohesion High Coupling Low

    Cohesion Low Coupling High Cohesion 35 Unknown source
  36. Slides by @arghrich Uncouple your Code 36 Delete Obsolete (unreachable

    or unused) Code Categorization Test, then Refactor Code Sprout new tested Code Break Dependencies (Seams) Strangler Fig Unfixable code Feature Flag Legacy Code Write ArchUnit Tests
  37. Slides by @arghrich Can we deploy this to production right

    now? Hmm, it’s somewhat tested but can we really be sure nothing breaks in production? 37
  38. 38 Automated Deployment Slides by @arghrich Versioned & Automated Compile

    Seperated Logic, UI & SQL In Hotspots Critical Use Case Tests Growing Unit Tests Unit tested new code We know what to focus on Horrible Cycle Time
  39. Slides by @arghrich Do Monitor in Production 39

  40. Slides by @arghrich RED Metrics 40 1 https://thenewstack.io/monitoring-microservices-red-method/ Community Dashboard

    https://community.grafana.com/t/how-can-i-configure-red-dashboard/69885 R -> Request Rate E -> Errors / second D -> Duration per request
  41. Slides by @arghrich New Cycle Time, without bug-cycle 41 Slide

    adapted from Dave Farley Commit Stage Compile Unit Test Lint Security Docker Release Git Monitor Logs Events Metrics Alerts Automated Performance testing 20 min Deploy To TST (Cont.) STG (Click) Smoke Test <1 min Manual Testing 30 min Automated Acceptance Testing User Acceptance Test DB-Integration Test 20 min 5 min 60 min
  42. Slides by @arghrich Can we deploy this to production right

    now? Yes. Depending on where we make the change, even within 30min. 43
  43. Slides by @arghrich But there is still a problem… 44

  44. Slides by @arghrich How did the previous developers get here?

    45
  45. Slides by @arghrich Healthy Software 46 Ideas Product Process Mindset

    Predicts
  46. Slides by @arghrich GIGO 47 Ideas Product Process Mindset Predicts

  47. Slides by @arghrich A Feature Factory helps no one 48

    https://cutlefish.substack.com/p/tbm-4952-your-calendar-your-priorities
  48. Slides by @arghrich 49 Henrik Kniberg @henrikkniberg Culture is the

    sum of everyone's attitude and actions, so model the behaviour you want to see! Observe behavior: • Who does the management promote? • Is there fear of failure? • Is quality appreciated or story points or …? • What is the delivery „process“? • Is there some kind of learning exchange?
  49. Slides by @arghrich Make the desired behavior … 50 Obvious

    • Define Principles together • Shared purpose Attractive • Reduce (time) pressure • (Refactoring) score Atomic Habits 4 Laws of behavior change Easy • Tech-coach • Reduce friction Satisfying • Pair up • Never break the chain (twice)
  50. Slides by @arghrich CD is a change in mindset 51

  51. Slides by @arghrich CD = Working in a way that

    software is always in a releasable state It’s not about the tools, even though they are getting better all the time 52
  52. Slides by @arghrich Thanks 53 IT-Tailor Richard Gross (he/him) Archaeologist

    Auditor richargh.de/ speakerdeck.com/richargh @arghrich Works at maibornwolff.de/ Tailored Software by People who Care. DE TN ES
  53. Slides by @arghrich Backup 56

  54. Slides by @arghrich 57

  55. Slides by @arghrich 58

  56. Slides by @arghrich 10 customer installations 59 Frobozz Co. Oceanic

    Airlines … Umbrella Corp Soylent Corp
  57. Slides by @arghrich And also 10 separate codebases 60 …

    Oceanic Airlines Soylent Corp Frobozz Co. Umbrella Corp Similar C/P Copy/ Paste Feature Copy/Paste
  58. Slides by @arghrich 10 codebases And 10 database schemata 61

    Similar Similar