CD for Legacy Code

This is based on a true story.

My day job is software archeology. I find joy in recovering and analyzing code bones and culture as well as making the skeleton walk again. A short time ago, however, I was confronted with the most horrible code base I have ever seen. This talk is about how we managed to save it and achieve bi-weekly deployments with a high level of confidence.

Five million lines of code in multiple languages (Classic ASP, .NET, VBScript, VBA, JavaScript, T-SQL, PL-SQL) in one monolith. The business logic stretched from the UI (WebForms, Scripting, SQL Queries) down to the database (Stored Procedures), there was no test coverage and an enormous amount of hidden coupling. A version control system was not used, we had no test environment, deployments required developers to copy their local compilation to production and multiple customer installations are supported by uncommenting and commenting code. Together we will explore what to do.

https://www.the-architecture-gathering.de/programm/programm-details/1394/cd-for-legacy-code/

Transcript

1. Continuous Delivery
   for Legacy Code
   Slides by Richard Gross @arghrich
   16.10.19
   Audit
   Software
   Archeology
   Develop
   Many Icons by Franziska Haaf binarykitten.de
   

   View Slide

2. Dylan Beattie
   @dylanbeattie
   Legacy Code is
   code that’s too
   scary to update
   and too profitable
   to delete.
   Michael Feathers
   @mfeathers
   Legacy
   Code is code
   without tests.
   Slide by @arghrich
   2
   

   View Slide

3. Legacy Code is
   code that’s too
   scary to update
   and too profitable
   to delete.
   Continuous
   Delivery
   If somebody
   thinks of a good
   idea, how do we
   [make it feasable
   to] deliver it to
   users as quickly
   as possibly?
   3
   Dylan Beattie
   @dylanbeattie
   Slide by @arghrich
   

   View Slide

4. Slide by @arghrich
   4
   Photo by Anne-sophie Parent from Pexels
   Spring
   

   View Slide

5. Slide by @arghrich
   5 Spring: Audit
   Icons by Franziska Haaf binarykitten.de
   COTS
   „Apocalypto“
   Please Audit
   

   View Slide

6. 4M LoC Delivery
   6
   Excel
   Webservice
   Website
   Fat Client
   Jobs
   Database
   Spring: Audit Slide by @arghrich
   T-SQL
   JavaScript
   VBA
   C#
   Asp.NET
   + C#
   Classic Asp
   + VBScript
   WinForms
   + C#
   MIA
   No Code
   delivered
   5M 4M LoC Delivery
   

   View Slide

7. Replicated Databases
   7
   Excel
   Webservice
   Website
   Fat Client
   Jobs
   DB1 DB2
   Spring: Audit Slide by @arghrich
   Replication Dependency
   

   View Slide

8. Slide by @arghrich
   8
   Icons by Franziska Haaf binarykitten.de
   Doesn‘t
   Compile
   BL & DB
   Everywhere
   Interesting
   Deployment
   15 years
   old
   No Tests
   Every
   Duplication
   Spring: Audit
   

   View Slide

9. COTS
   „Apocalypto“
   In-house
   „Fenix“
   Slide by @arghrich
   9 Spring: Audit
   Icons by Franziska Haaf binarykitten.de
   Buy
   I‘ll
   

   View Slide

10. Slide by @arghrich
    10 Spring: Audit
    Icons by Franziska Haaf binarykitten.de
    In-house
    „Fenix“
    Please Fix
    

    View Slide

11. Oh Boy
    

    View Slide

12. Slide by @arghrich
    12
    We could just …
    

    View Slide

13. Slide by @arghrich
    13
    Photo by Levi Damasceno from Pexels
    

    View Slide

14. The True Cost of Rewrites
    Features
    Releases over time
    Catch Up
    Missing
    Features
    Sub-Par Parity
    Enhancements
    Planned
    Rewrite
    Actual Features
    Undiscovered Scope
    Old App
    Adoption
    Slide Adaption by @arghrich
    Original Article by Doug Bradbury The True Cost of Rewrites
    

    View Slide

15. Slide by @arghrich
    15 Early Summer
    Photo by Pixabay
    

    View Slide

16. Slide by @arghrich
    16
    How did they get here?
    Early Summer
    

    View Slide

17. GIGO
    17
    Ideas
    Product
    Process
    (Team)
    Spirit
    Early Summer Slide by @arghrich
    Predicts
    

    View Slide

18. GIGO
    18
    Ideas
    Product
    Process
    (Team)
    Spirit
    Early Summer Slide by @arghrich
    Predicts
    Fix
    

    View Slide

19. Slide by @arghrich
    19
    How did we not get here?
    Early Summer
    

    View Slide

20. We started a
    significant
    change
    in team and culture
    20 Early Summer: Spirit Slide by @arghrich
    

    View Slide

21. 21
    Henrik Kniberg
    @henrikkniberg
    Culture is the sum of everyone's
    attitude and actions, so model the
    behaviour you want to see!
    Observe behavior:
    • Who does the management promote?
    • Is there fear of failure?
    • What is the delivery „process“?
    • Is there some kind of learning exchange?
    • Is quality appreciated or story points?
    Early Summer: Spirit Slide by @arghrich
    

    View Slide

22. 4 Laws of behavior change
    Make it …
    Obvious
    • Define Principles
    together
    • Shared purpose
    Attractive
    • Reduce (time)
    pressure
    • (Refactoring) score
    Slide by @arghrich
    22
    Atomic Habits
    Early Summer: Spirit
    Easy
    • Tech-coach
    • Reduce friction
    Satisfying
    • Pair up
    • Never break the chain
    (twice)
    

    View Slide

23. We built a new Co-located
    Cross-functional Team
    with
    shared
    23
    DEV TEST OPS
    Key Accounter
    Early Summer: Spirit
    & principles
    purpose
    

    View Slide

24. Our Legacy CD Principles
    • Build quality in
    • Focus on current tree, not on the forest
    • Tackle fear head on
    • Focus on reducing batch size
    • Low-Risk releases are incremental
    • Decouple Deployment and Release
    • Computers perform repetitive tasks, people solve
    problems
    Slide Adaption by @arghrich
    Four Principles of Low-Risk Software Releases by Jez Humble
    CD Principles by Jez Humble
    Early Summer: Spirit
    24
    The Why
    

    View Slide

25. Slide by @arghrich
    25 Early Summer: Spirit
    Principles provide adaptability
    

    View Slide

26. Slide by @arghrich
    26 Summer
    Icons by Franziska Haaf binarykitten.de
    In-house
    „Fenix“
    Please Cloud
    

    View Slide

27. Oh Boy!
    

    View Slide

28. Fix
    28
    Product
    Process
    (Team)
    Spirit
    Summer Slide by @arghrich
    Predicts
    

    View Slide

29. Local
    Env Source
    Repository
    Artifact
    Repository
    Commit
    Stage
    Acceptance
    Manual
    Test
    Env
    …
    Staging
    Env
    Prod
    Env
    Self-Service
    Automatic
    Our CD Pipe
    Feedback (Tag Artifact) Slide Adaption by @arghrich
    Dave Farley - Optimising Continuous Delivery
    

    View Slide

30. Our Branching Strategy
    • Don‘t Branch
    • Don‘t Branch
    • Don‘t Branch
    • Never Branch to Refactor
    •Branches indicate
    Ci
    =Continuous Isolation
    Slide by @arghrich
    30
    Strategy by Dave Farley
    Summer: Process
    

    View Slide

31. We versioned and tested
    Everything
    • Code
    • Config
    • Deployment scripts
    • Infrastructure scripts
    • Docs
    • …
    31 Slide by @arghrich
    Summer: Process
    

    View Slide

32. They used File Version
    Anti-Practice
    32 Slide by @arghrich
    File tree from actual project
    Summer: Process
    

    View Slide

33. Slide by @arghrich
    33 Autumn
    Photo by Daniel Frank from Pexels
    

    View Slide

34. Fix
    34
    Product
    Process
    (Team)
    Spirit
    Autumn Slide by @arghrich
    

    View Slide

35. Slide by @arghrich
    35
    Where do we start?
    Autumn
    

    View Slide

36. Slide by @arghrich
    Art by Zaufishan
    36 Autumn: Product
    

    View Slide

37. github.com/MaibornWolff/codecharta
    Slide by @arghrich
    37
    Shameless
    Autumn: Product
    

    View Slide

38. Visualize metrics with buildings
    Slide by @arghrich
    38
    AbstractService.java
    Lines of code
    Complexity
    number of authors
    Autumn: Product
    Icons by Fontawesome
    Very Shameless
    

    View Slide

39. Find potential hotspots
    Slide by @arghrich
    39
    Still Shameless
    Autumn: Product
    

    View Slide

40. Establish baseline with tools
    Then track progress
    • CodeCharta
    https://github.com/
    MaibornWolff/codecharta
    • SonarQube
    https://www.sonarqube.org/
    • Linter
    • Profiler
    • CodeMaat
    https://github.com/
    adamtornhill/code-maat
    • Structure101
    http://structure101.com/
    Slide by @arghrich
    40 Autumn: Product Baseline
    

    View Slide

41. Golden Master Baseline
    41
    vX - redesigned
    v1 - legacy
    Input for UI/API/DB
    Output
    Output
    Continuous
    Compare
    Slide by @arghrich
    Autumn: Product Baseline
    

    View Slide

42. Slide by @arghrich
    42 Autumn
    Icons by Franziska Haaf binarykitten.de
    In-house
    „Fenix“
    Add Feature
    

    View Slide

43. Slide by @arghrich
    43
    Photo by Bartek Wojtas from Pexels
    Autumn
    Fuuuu
    

    View Slide

44. Conquer the fear
    • Write categorization/function Tests
    • Safely verify it can handle production
    • Divide code into stable and unstable
    Slide by @arghrich
    44 Autumn: Conquer the Fear
    

    View Slide

45. Slide by @arghrich
    45
    Photo by Emmet from Pexels
    Autumn: Conquer the Fear
    

    View Slide

46. Slide by @arghrich
    46
    Photo by Krizjohn Rosales from Pexels
    Autumn: Conquer the Fear
    Isolate
    Stable from Unstable
    Working
    Effectively with
    Legacy Code
    • Direct Read/Write à Facades
    • Classes à Interfaces + Dependency Injection
    • Fire&Forget calls à Events
    • Obsolete/duplicate tables à Table Views
    

    View Slide

47. Slide by @arghrich
    47
    Photo by Alex Fu from Pexels
    Autumn: Conquer the Fear
    Sprout quality code
    Working
    Effectively with
    Legacy Code
    Table View
    Interface Facade Event
    

    View Slide

48. Use Strangler Application
    for bigger features
    48 Slide by @arghrich
    Autumn: Conquer the Fear
    Picture by Paul Hammant
    

    View Slide

49. Slide by @arghrich
    49 Winter
    Photo by Riccardo Bresciani from Pexels
    

    View Slide

50. vX - redesigned
    Dark Launch to gain confidence
    Slide by @arghrich
    50
    Icons by Fontawesome
    Load Balancer
    v1 - legacy
    Continuous
    Compare
    Same number of:
    • 200?
    • 404?
    • 5xx?
    • …
    Winter: Launch confidence
    

    View Slide

51. Remove
    any
    obsolete code
    in
    Tech || Business
    51 Slide by @arghrich
    Winter: Remove waste
    Don‘t fix what doesn‘t need fixing
    Slide by @arghrich
    Photo by Stas Knop from Pexels
    

    View Slide

52. Finding obsolete code
    • Add Analytics / Activity-Log for your dark pages
    • User surveys
    • IDE suggestions
    • grep FROM, JOIN, INSERT, UPDATE, DELETE
    and check if that code is reachable
    • Legacy Toggles
    • Forget to deploy parts of your application
    52 Slide by @arghrich
    Winter: Remove waste
    

    View Slide

53. Slide by @arghrich
    53 Next Spring: Summary
    Photo by Irina Iriser from Pexels
    

    View Slide

54. Legacy Delivery
    54 Slide by @arghrich
    Next Spring: Summary
    Photo by pixabay from Pexels
    Before
    • 2 Weeks for a feature
    • 5 months until it works
    After
    • 4 Weeks for a feature
    • Then it works
    

    View Slide

55. Was it worth it?
    • Users had capitulated
    à then we started fixing bugs
    • More Stable then before
    • We couldn‘t have reimplemented this from scratch
    • à None can describe what „same as before“ means
    • à Even if, it would have taken more than a year
    • Seamless migration from COTS to in-house
    à min. feature freeze
    55 Slide by @arghrich
    Next Spring: Summary
    

    View Slide

56. Slide by @arghrich
    56
    Would I do it again?
    Next Spring: Summary
    

    View Slide

57. Hell Yes
    We had
    • Autonomy
    • Mastery
    • Purpose
    57 Slide by @arghrich
    Next Spring: Summary
    Model from „Drive“ by Daniel Pink
    

    View Slide

58. 58 Next Spring: Summary Slide by @arghrich
    (Team)
    Spirit
    • Conquer the
    fear
    • CD Principles
    • Build quality in
    • Shared
    purpose
    Product
    • Isolate stable
    from unstable
    • Golden Master
    • Strangler
    • Feathers
    Process
    • CD Pipeline
    • Infrastructure
    Automation
    

    View Slide

59. Ken Mugrage
    @kmugrage
    It‘s not CD if you can‘t deploy to
    production right now
    59 Slide by @arghrich
    Next Spring: Summary
    

    View Slide

60. Done
    Questions, Remarks, Sympathy? J
    30 Mio
    410
    Employees 9th year
    23
    Nations
    300
    Weeks
    Munich, Augsburg,
    Berlin, Frankfurt
    

    View Slide

61. Learn Domain with
    Reverse Object Mother
    • Start application with empty database and
    without domain model
    • Click through a UseCase
    • Analyse exceptions and errors
    „App needs at least an object A with this field“
    • Expand Domain Modell with your finding
    • Create required state in the DB with your model
    • Document finding as categorization test
    • Repeat
    61 Slide by @arghrich
    Autumn: Rake the forest
    

    View Slide

62. Reading
    • https://trello.com/b/Fdd876S8/continuous-
    delivery-checklist-template
    • I did not think I would show this slide
    62 Slide by @arghrich
    

    View Slide