Upgrade to Pro — share decks privately, control downloads, hide ads and more …

CD for Legacy Code

October 16, 2019

CD for Legacy Code

This is based on a true story.

My day job is software archeology. I find joy in recovering and analyzing code bones and culture as well as making the skeleton walk again. A short time ago, however, I was confronted with the most horrible code base I have ever seen. This talk is about how we managed to save it and achieve bi-weekly deployments with a high level of confidence.

Five million lines of code in multiple languages (Classic ASP, .NET, VBScript, VBA, JavaScript, T-SQL, PL-SQL) in one monolith. The business logic stretched from the UI (WebForms, Scripting, SQL Queries) down to the database (Stored Procedures), there was no test coverage and an enormous amount of hidden coupling. A version control system was not used, we had no test environment, deployments required developers to copy their local compilation to production and multiple customer installations are supported by uncommenting and commenting code. Together we will explore what to do.



October 16, 2019

More Decks by Richard

Other Decks in Technology


  1. Continuous Delivery for Legacy Code Slides by Richard Gross @arghrich

    16.10.19 Audit Software Archeology Develop Many Icons by Franziska Haaf binarykitten.de
  2. Dylan Beattie @dylanbeattie Legacy Code is code that’s too scary

    to update and too profitable to delete. Michael Feathers @mfeathers Legacy Code is code without tests. Slide by @arghrich 2
  3. Legacy Code is code that’s too scary to update and

    too profitable to delete. Continuous Delivery If somebody thinks of a good idea, how do we [make it feasable to] deliver it to users as quickly as possibly? 3 Dylan Beattie @dylanbeattie Slide by @arghrich
  4. Slide by @arghrich 4 Photo by Anne-sophie Parent from Pexels

  5. Slide by @arghrich 5 Spring: Audit Icons by Franziska Haaf

    binarykitten.de COTS „Apocalypto“ Please Audit
  6. 4M LoC Delivery 6 Excel Webservice Website Fat Client Jobs

    Database Spring: Audit Slide by @arghrich T-SQL JavaScript VBA C# Asp.NET + C# Classic Asp + VBScript WinForms + C# MIA No Code delivered 5M 4M LoC Delivery
  7. Replicated Databases 7 Excel Webservice Website Fat Client Jobs DB1

    DB2 Spring: Audit Slide by @arghrich Replication Dependency
  8. Slide by @arghrich 8 Icons by Franziska Haaf binarykitten.de Doesn‘t

    Compile BL & DB Everywhere Interesting Deployment 15 years old No Tests Every Duplication Spring: Audit
  9. COTS „Apocalypto“ In-house „Fenix“ Slide by @arghrich 9 Spring: Audit

    Icons by Franziska Haaf binarykitten.de Buy I‘ll
  10. Slide by @arghrich 10 Spring: Audit Icons by Franziska Haaf

    binarykitten.de In-house „Fenix“ Please Fix
  11. Oh Boy

  12. Slide by @arghrich 12 We could just …

  13. Slide by @arghrich 13 Photo by Levi Damasceno from Pexels

  14. The True Cost of Rewrites Features Releases over time Catch

    Up Missing Features Sub-Par Parity Enhancements Planned Rewrite Actual Features Undiscovered Scope Old App Adoption Slide Adaption by @arghrich Original Article by Doug Bradbury The True Cost of Rewrites
  15. Slide by @arghrich 15 Early Summer Photo by Pixabay

  16. Slide by @arghrich 16 How did they get here? Early

  17. GIGO 17 Ideas Product Process (Team) Spirit Early Summer Slide

    by @arghrich Predicts
  18. GIGO 18 Ideas Product Process (Team) Spirit Early Summer Slide

    by @arghrich Predicts Fix
  19. Slide by @arghrich 19 How did we not get here?

    Early Summer
  20. We started a significant change in team and culture 20

    Early Summer: Spirit Slide by @arghrich
  21. 21 Henrik Kniberg @henrikkniberg Culture is the sum of everyone's

    attitude and actions, so model the behaviour you want to see! Observe behavior: • Who does the management promote? • Is there fear of failure? • What is the delivery „process“? • Is there some kind of learning exchange? • Is quality appreciated or story points? Early Summer: Spirit Slide by @arghrich
  22. 4 Laws of behavior change Make it … Obvious •

    Define Principles together • Shared purpose Attractive • Reduce (time) pressure • (Refactoring) score Slide by @arghrich 22 Atomic Habits Early Summer: Spirit Easy • Tech-coach • Reduce friction Satisfying • Pair up • Never break the chain (twice)
  23. We built a new Co-located Cross-functional Team with shared 23

    DEV TEST OPS Key Accounter Early Summer: Spirit & principles purpose
  24. Our Legacy CD Principles • Build quality in • Focus

    on current tree, not on the forest • Tackle fear head on • Focus on reducing batch size • Low-Risk releases are incremental • Decouple Deployment and Release • Computers perform repetitive tasks, people solve problems Slide Adaption by @arghrich Four Principles of Low-Risk Software Releases by Jez Humble CD Principles by Jez Humble Early Summer: Spirit 24 The Why
  25. Slide by @arghrich 25 Early Summer: Spirit Principles provide adaptability

  26. Slide by @arghrich 26 Summer Icons by Franziska Haaf binarykitten.de

    In-house „Fenix“ Please Cloud
  27. Oh Boy!

  28. Fix 28 Product Process (Team) Spirit Summer Slide by @arghrich

  29. Local Env Source Repository Artifact Repository Commit Stage Acceptance Manual

    Test Env … Staging Env Prod Env Self-Service Automatic Our CD Pipe Feedback (Tag Artifact) Slide Adaption by @arghrich Dave Farley - Optimising Continuous Delivery
  30. Our Branching Strategy • Don‘t Branch • Don‘t Branch •

    Don‘t Branch • Never Branch to Refactor •Branches indicate Ci =Continuous Isolation Slide by @arghrich 30 Strategy by Dave Farley Summer: Process
  31. We versioned and tested Everything • Code • Config •

    Deployment scripts • Infrastructure scripts • Docs • … 31 Slide by @arghrich Summer: Process
  32. They used File Version Anti-Practice 32 Slide by @arghrich File

    tree from actual project Summer: Process
  33. Slide by @arghrich 33 Autumn Photo by Daniel Frank from

  34. Fix 34 Product Process (Team) Spirit Autumn Slide by @arghrich

  35. Slide by @arghrich 35 Where do we start? Autumn

  36. Slide by @arghrich Art by Zaufishan 36 Autumn: Product

  37. github.com/MaibornWolff/codecharta Slide by @arghrich 37 Shameless Autumn: Product

  38. Visualize metrics with buildings Slide by @arghrich 38 AbstractService.java Lines

    of code Complexity number of authors Autumn: Product Icons by Fontawesome Very Shameless
  39. Find potential hotspots Slide by @arghrich 39 Still Shameless Autumn:

  40. Establish baseline with tools Then track progress • CodeCharta https://github.com/

    MaibornWolff/codecharta • SonarQube https://www.sonarqube.org/ • Linter • Profiler • CodeMaat https://github.com/ adamtornhill/code-maat • Structure101 http://structure101.com/ Slide by @arghrich 40 Autumn: Product Baseline
  41. Golden Master Baseline 41 vX - redesigned v1 - legacy

    Input for UI/API/DB Output Output Continuous Compare Slide by @arghrich Autumn: Product Baseline
  42. Slide by @arghrich 42 Autumn Icons by Franziska Haaf binarykitten.de

    In-house „Fenix“ Add Feature
  43. Slide by @arghrich 43 Photo by Bartek Wojtas from Pexels

    Autumn Fuuuu
  44. Conquer the fear • Write categorization/function Tests • Safely verify

    it can handle production • Divide code into stable and unstable Slide by @arghrich 44 Autumn: Conquer the Fear
  45. Slide by @arghrich 45 Photo by Emmet from Pexels Autumn:

    Conquer the Fear
  46. Slide by @arghrich 46 Photo by Krizjohn Rosales from Pexels

    Autumn: Conquer the Fear Isolate Stable from Unstable Working Effectively with Legacy Code • Direct Read/Write à Facades • Classes à Interfaces + Dependency Injection • Fire&Forget calls à Events • Obsolete/duplicate tables à Table Views
  47. Slide by @arghrich 47 Photo by Alex Fu from Pexels

    Autumn: Conquer the Fear Sprout quality code Working Effectively with Legacy Code Table View Interface Facade Event
  48. Use Strangler Application for bigger features 48 Slide by @arghrich

    Autumn: Conquer the Fear Picture by Paul Hammant
  49. Slide by @arghrich 49 Winter Photo by Riccardo Bresciani from

  50. vX - redesigned Dark Launch to gain confidence Slide by

    @arghrich 50 Icons by Fontawesome Load Balancer v1 - legacy Continuous Compare Same number of: • 200? • 404? • 5xx? • … Winter: Launch confidence
  51. Remove any obsolete code in Tech || Business 51 Slide

    by @arghrich Winter: Remove waste Don‘t fix what doesn‘t need fixing Slide by @arghrich Photo by Stas Knop from Pexels
  52. Finding obsolete code • Add Analytics / Activity-Log for your

    dark pages • User surveys • IDE suggestions • grep FROM, JOIN, INSERT, UPDATE, DELETE and check if that code is reachable • Legacy Toggles • Forget to deploy parts of your application 52 Slide by @arghrich Winter: Remove waste
  53. Slide by @arghrich 53 Next Spring: Summary Photo by Irina

    Iriser from Pexels
  54. Legacy Delivery 54 Slide by @arghrich Next Spring: Summary Photo

    by pixabay from Pexels Before • 2 Weeks for a feature • 5 months until it works After • 4 Weeks for a feature • Then it works
  55. Was it worth it? • Users had capitulated à then

    we started fixing bugs • More Stable then before • We couldn‘t have reimplemented this from scratch • à None can describe what „same as before“ means • à Even if, it would have taken more than a year • Seamless migration from COTS to in-house à min. feature freeze 55 Slide by @arghrich Next Spring: Summary
  56. Slide by @arghrich 56 Would I do it again? Next

    Spring: Summary
  57. Hell Yes We had • Autonomy • Mastery • Purpose

    57 Slide by @arghrich Next Spring: Summary Model from „Drive“ by Daniel Pink
  58. 58 Next Spring: Summary Slide by @arghrich (Team) Spirit •

    Conquer the fear • CD Principles • Build quality in • Shared purpose Product • Isolate stable from unstable • Golden Master • Strangler • Feathers Process • CD Pipeline • Infrastructure Automation
  59. Ken Mugrage @kmugrage It‘s not CD if you can‘t deploy

    to production right now 59 Slide by @arghrich Next Spring: Summary
  60. Done Questions, Remarks, Sympathy? J 30 Mio 410 Employees 9th

    year 23 Nations 300 Weeks Munich, Augsburg, Berlin, Frankfurt
  61. Learn Domain with Reverse Object Mother • Start application with

    empty database and without domain model • Click through a UseCase • Analyse exceptions and errors „App needs at least an object A with this field“ • Expand Domain Modell with your finding • Create required state in the DB with your model • Document finding as categorization test • Repeat 61 Slide by @arghrich Autumn: Rake the forest
  62. Reading • https://trello.com/b/Fdd876S8/continuous- delivery-checklist-template • I did not think I

    would show this slide 62 Slide by @arghrich