Save 37% off PRO during our Black Friday Sale! »

Continuous Delivery for Legacy Code

0ba0746fe19c0679ba24018cce4c8b1a?s=47 Richard
November 13, 2019

Continuous Delivery for Legacy Code

This is based on a true story.

My day job is software archeology. I find joy in recovering and analyzing code bones and culture as well as making the skeleton walk again. A short time ago, however, I was confronted with the most horrible code base I have ever seen. This talk is about how we managed to save it and achieve bi-weekly deployments with a high level of confidence.

Five million lines of code in multiple languages (Classic ASP, .NET, VBScript, VBA, JavaScript, T-SQL, PL-SQL) in one monolith. The business logic stretched from the UI (WebForms, Scripting, SQL Queries) down to the database (Stored Procedures), there was no test coverage and an enormous amount of hidden coupling. A version control system was not used, we had no test environment, deployments required developers to copy their local compilation to production and multiple customer installations are supported by uncommenting and commenting code. Together we will explore what to do.

https://buildstuff2019.sched.com/event/UzgO/richard-gross-continuous-delivery-for-legacy-code

0ba0746fe19c0679ba24018cce4c8b1a?s=128

Richard

November 13, 2019
Tweet

Transcript

  1. Continuous Delivery for Legacy Code 13.11.19 Audit Software Archeology Develop

    Small Icons by Fontawesome Many Icons by Franziska Haaf binarykitten.de richargh.de/ speakerdeck.com/richargh @arghrich Richard Gross pronoun.is/he
  2. Dylan Beattie @dylanbeattie Legacy Code is code that’s too scary

    to update and too profitable to delete. Michael Feathers @mfeathers Legacy Code is code without tests. 2 Slides by @arghrich
  3. Continuous Delivery Legacy Code is code that’s too scary to

    update and too profitable to delete. If somebody thinks of a good idea, how do we [make it feasable to] deliver it to users as quickly as possibly? 3 Dylan Beattie @dylanbeattie Slides by @arghrich
  4. 4 Photo by Anne-sophie Parent from Pexels Spring Slides by

    @arghrich
  5. 5 COTS „Apocalypto“ Please Audit Icons by Franziska Haaf binarykitten.de

    Slides by @arghrich Spring: Audit
  6. 6 Excel Webservice Website Fat Client Jobs Database T-SQL JavaScript

    VBA C# Asp.NET + C# Classic Asp + VBScript WinForms + C# MIA No Code delivered Slides by @arghrich Spring: Audit 5M 4M LoC Delivery
  7. Replicated Databases 7 Excel Webservice Website Fat Client Jobs DB1

    DB2 Replication Dependency Slides by @arghrich Spring: Audit
  8. 8 Icons by Franziska Haaf binarykitten.de Doesn‘t Compile BL &

    DB Everywhere Interesting Deployment 15 years old No Tests Every Duplication Slides by @arghrich Spring: Audit
  9. COTS „Apocalypto“ In-house „Fenix“ 9 Buy I‘ll Icons by Franziska

    Haaf binarykitten.de Slides by @arghrich Spring: Audit
  10. 10 In-house „Fenix“ Please Fix Icons by Franziska Haaf binarykitten.de

    Slides by @arghrich Spring: Audit
  11. Oh Boy

  12. We could just … 12 Icons by Franziska Haaf binarykitten.de

    Slides by @arghrich Spring: Audit
  13. 13 Photo by Levi Damasceno from Pexels Slides by @arghrich

    Spring: Audit
  14. The True Cost of Rewrites Features Releases over time Catch

    Up Missing Features Sub-Par Parity Enhancements Planned Rewrite Actual Features Undiscovered Scope Old App Adoption Original Article by Doug Bradbury The True Cost of Rewrites Slides by @arghrich
  15. 15 Photo by Pixabay from Pexels Slides by @arghrich Early

    Summer
  16. How did they get here? 16 Slides by @arghrich Early

    Summer
  17. GIGO 17 Ideas Product Process (Team) Spirit Predicts Slides by

    @arghrich Early Summer
  18. GIGO 18 Ideas Product Process (Team) Spirit Fix Predicts Slides

    by @arghrich Early Summer
  19. How did we not get here? 19 Slides by @arghrich

    Early Summer
  20. 20 We needed a significant change in team and culture

    Slides by @arghrich Early Summer: Spirit
  21. 21 Henrik Kniberg @henrikkniberg Culture is the sum of everyone's

    attitude and actions, so model the behaviour you want to see! Observe behavior: • Who does the management promote? • Is there fear of failure? • Is quality appreciated or story points or …? • What is the delivery „process“? • Is there some kind of learning exchange? Slides by @arghrich Early Summer: Spirit
  22. 4 Laws of behavior change Make it … Obvious •

    Define Principles together • Shared purpose Attractive • Reduce (time) pressure • (Refactoring) score 22 Atomic Habits Easy • Tech-coach • Reduce friction Satisfying • Pair up • Never break the chain (twice) Slides by @arghrich Early Summer: Spirit
  23. 23 We built a new Co-located Cross-functional Team with shared

    DEV TEST OPS Key Accounter & principles purpose Slides by @arghrich Early Summer: Spirit
  24. Our Legacy CD Principles • Build quality in • Focus

    on current tree, not on the forest • Tackle fear head on • Focus on reducing batch size • Decouple Deployment and Release • Computers perform repetitive tasks, people solve problems Four Principles of Low-Risk Software Releases by Jez Humble CD Principles by Jez Humble The Why 24 Slides by @arghrich Early Summer: Spirit
  25. 25 Principles provide adaptability Slides by @arghrich Early Summer: Spirit

  26. 26 Icons by Franziska Haaf binarykitten.de In-house „Fenix“ Please Cloud

    Slides by @arghrich Early Summer: Spirit
  27. Oh Boy

  28. Fix 28 Product Process (Team) Spirit Predicts Slides by @arghrich

    Summer
  29. Local Env Source Repository Artifact Repository Commit Stage Acceptance Manual

    Test Env … Staging Env Prod Env Self-Service Automatic Feedback (Tag Artifact) Dave Farley - Optimising Continuous Delivery Slides by @arghrich Our CD Pipe
  30. • Don‘t Branch • Don‘t Branch • Don‘t Branch •

    Never Branch to Refactor • Branches indicate Ci =Continuous Isolation Strategy by Dave Farley Our Branching Strategy 30 Slides by @arghrich Summer: Process
  31. • Code • Config • Deployment scripts • Infrastructure scripts

    • Docs • … 31 We versioned and tested Everything Slides by @arghrich Summer: Process
  32. They used File Version Anti-Practice 32 File tree from actual

    project Slides by @arghrich Summer: Process
  33. 33 Photo by Daniel Frank from Pexels Slides by @arghrich

    Autumn
  34. Fix 34 Product Process (Team) Spirit Slides by @arghrich Autumn

  35. Where do we start? 35 Slides by @arghrich Autumn

  36. Art by Zaufishan 36 Slides by @arghrich Autumn: Product

  37. github.com/MaibornWolff/codecharta 37 Shameless Slides by @arghrich Autumn: Product

  38. Visualize metrics with buildings 38 AbstractService.java Lines of code Complexity

    number of authors Icons by Fontawesome Very Shameless Slides by @arghrich Autumn: Product
  39. Find potential hotspots 39 Still Shameless Slides by @arghrich Autumn:

    Product
  40. No tool to rule them all 40 Slides by @arghrich

    Continuously Periodically Structure101 SonarQube CodeCharta CodeScene ArchUnit TSArch SonarArchitect Quality Views by Colin Breck Autumn: Product Baseline Quality Views Shamelessly putting CodeCharta first
  41. 41 Photo by Pixabay from Pexels Slides by @arghrich Autumn:

    Product Baseline Counting Fixed Windows
  42. 42 vX - redesigned v1 - legacy Input for UI/API/DB

    Output Output Continuous Compare Slides by @arghrich Autumn: Product Baseline Golden Master Characterizes Behavior before Refactoring
  43. 43 Slides by @arghrich Autumn: Product Baseline Golden Master Cheatsheet

    Input • Have Lots & Check coverage • Capture from Production • Generate • Try Testrecorder Output • Log messages • Datalake • Database Hidden Inputs • Date • Randoms When • Before refactorings • Generate new output when needed
  44. How do we refactor what we don‘t understand? 44 Slides

    by @arghrich Autumn: Product Baseline
  45. 45 Slides by @arghrich Autumn: Product Baseline Emily Bache @emilybache

    „Nopefactoring“ The No-thinking refactoring“ Advanced Testing & Refactoring Techniques • Lift-up conditional • Split to Classes Llewellyn Falco @LlewellynFalco Cutting Code Quickly
  46. 46 Icons by Franziska Haaf binarykitten.de In-house „Fenix“ Add Feature

    Slides by @arghrich Autumn
  47. Fuuuu 47 Photo by Bartek Wojtas from Pexels Slides by

    @arghrich Autumn
  48. • Write categorization, if possible function Tests • Safely verify

    it can handle production • Divide code into stable and unstable • Understand Code Domain 48 Slides by @arghrich Autumn: Conquer the Fear Conquer the fear
  49. Learn Domain with Reverse Object Mother • Start application with

    empty database and without domain model • Click through a UseCase • Analyse exceptions and errors „App needs at least an object A with this field“ • Expand Domain Modell with your finding • Create required state in the DB with your model • Document finding as categorization test • Repeat 49 Slides by @arghrich Autumn: Conquer the Fear
  50. 50 Photo by Emmet from Pexels Slides by @arghrich Autumn:

    Conquer the Fear
  51. • Direct Read/Write à Facades • Classes à Interfaces +

    Dependency Injection • Fire&Forget calls à Events • Obsolete/duplicate tables à Table Views • Dynamic GET/POST à Explicit constants 51 Photo by Krizjohn Rosales from Pexels Isolate Stable from Unstable Working Effectively with Legacy Code Slides by @arghrich Autumn: Conquer the Fear
  52. Sprout quality code 52 Photo by Alex Fu from Pexels

    Working Effectively with Legacy Code Table View Interface Facade Event Slides by @arghrich Autumn: Conquer the Fear
  53. Use Strangler Application for bigger features Picture by Paul Hammant

    Great talk Strangling the Monolith With a Data-Driven Approach 53 Slides by @arghrich Autumn: Conquer the Fear
  54. 54 Photo by Riccardo Bresciani from Pexels Slides by @arghrich

    Winter
  55. Slides by @arghrich vX - redesigned 55 Icons by Fontawesome

    Load Balancer v1 - legacy Continuous Compare Same number of: • 200? • 404? • 5xx? • … Winter: Launch confidence Dark Launch to gain confidence
  56. Remove any obsolete code in Tech || Business 56 Photo

    by Stas Knop from Pexels Slides by @arghrich Winter: Remove waste Don‘t fix what doesn‘t need fixing
  57. Finding obsolete code • Add Analytics / Activity-Log for your

    dark launch pages • User surveys • IDE suggestions • grep FROM, JOIN, INSERT, UPDATE, DELETE in .asp and check if that code is reachable • Legacy Toggles • Forget to deploy parts of your application 57 Slides by @arghrich Winter: Remove waste
  58. 58 Photo by Irina Iriser from Pexels Slides by @arghrich

    Next Spring: Summary
  59. Legacy Delivery 59 Photo by pixabay from Pexels Before •

    2 Weeks for a feature • 5 months until it works After • 4 Weeks for a feature • Then it works Slides by @arghrich Next Spring: Summary
  60. Was it worth it? • Users had capitulated à then

    we started fixing bugs • More Stable then before • We couldn‘t have reimplemented this from scratch • à None can describe what „same as before“ means • à Even if, it would have taken more than a year • Seamless migration from COTS to in-house à min. feature freeze 60 Slides by @arghrich Next Spring: Summary
  61. Would I do it again? 61 Slides by @arghrich Next

    Spring: Summary
  62. Hell Yes We had • Autonomy • Mastery • Purpose

    62 Model from „Drive“ by Daniel Pink Slides by @arghrich Next Spring: Summary
  63. 63 (Team) Spirit • Conquer the fear • CD Principles

    • Build quality in • Shared purpose Product • Isolate stable from unstable • Golden Master • Strangler • Feathers Process • CD Pipeline • Infrastructure Automation Slides by @arghrich Next Spring: Summary
  64. Ken Mugrage @kmugrage It‘s not CD if you can‘t deploy

    to production right now 64 Slides by @arghrich Next Spring: Summary
  65. Done Questions, Remarks, Sympathy? J 30 Mio 510 Employees 9th

    year 23 Nations 300 Weeks Munich, Augsburg, Berlin, Frankfurt @arghrich speakerdeck.com/richargh richargh.de/ Most of our projects aren‘t like this J Richard Gross pronoun.is/he
  66. Reading • https://trello.com/b/Fdd876S8/continuous-delivery-checklist- template • I did not think I

    would show this slide 66 Slides by @arghrich