Continuous Delivery for Legacy Code

This is based on a true story.

My day job is software archeology. I find joy in recovering and analyzing code bones and culture as well as making the skeleton walk again. A short time ago, however, I was confronted with the most horrible code base I have ever seen. This talk is about how we managed to save it and achieve bi-weekly deployments with a high level of confidence.

Five million lines of code in multiple languages (Classic ASP, .NET, VBScript, VBA, JavaScript, T-SQL, PL-SQL) in one monolith. The business logic stretched from the UI (WebForms, Scripting, SQL Queries) down to the database (Stored Procedures), there was no test coverage and an enormous amount of hidden coupling. A version control system was not used, we had no test environment, deployments required developers to copy their local compilation to production and multiple customer installations are supported by uncommenting and commenting code. Together we will explore what to do.

https://buildstuff2019.sched.com/event/UzgO/richard-gross-continuous-delivery-for-legacy-code

Transcript

1. Continuous Delivery
   for Legacy Code
   13.11.19
   Audit
   Software
   Archeology
   Develop
   Small Icons by Fontawesome
   Many Icons by Franziska Haaf binarykitten.de richargh.de/
   speakerdeck.com/richargh
   @arghrich
   Richard Gross pronoun.is/he
   

   View Slide

2. Dylan Beattie
   @dylanbeattie
   Legacy Code is
   code that’s too
   scary to update
   and too profitable
   to delete.
   Michael Feathers
   @mfeathers
   Legacy
   Code is code
   without tests.
   2 Slides by @arghrich
   

   View Slide

3. Continuous
   Delivery
   Legacy Code is
   code that’s too
   scary to update
   and too profitable
   to delete.
   If somebody
   thinks of a good
   idea, how do we
   [make it feasable
   to] deliver it to
   users as quickly
   as possibly?
   3
   Dylan Beattie
   @dylanbeattie
   Slides by @arghrich
   

   View Slide

4. 4
   Photo by Anne-sophie Parent from Pexels
   Spring Slides by @arghrich
   

   View Slide

5. 5
   COTS
   „Apocalypto“
   Please Audit
   Icons by Franziska Haaf binarykitten.de
   Slides by @arghrich
   Spring: Audit
   

   View Slide

6. 6
   Excel
   Webservice
   Website
   Fat Client
   Jobs
   Database T-SQL
   JavaScript
   VBA
   C#
   Asp.NET
   + C#
   Classic Asp
   + VBScript
   WinForms
   + C#
   MIA
   No Code
   delivered
   Slides by @arghrich
   Spring: Audit
   5M 4M LoC Delivery
   

   View Slide

7. Replicated Databases
   7
   Excel
   Webservice
   Website
   Fat Client
   Jobs
   DB1 DB2
   Replication Dependency
   Slides by @arghrich
   Spring: Audit
   

   View Slide

8. 8
   Icons by Franziska Haaf binarykitten.de
   Doesn‘t
   Compile
   BL & DB
   Everywhere
   Interesting
   Deployment
   15 years
   old
   No Tests
   Every
   Duplication
   Slides by @arghrich
   Spring: Audit
   

   View Slide

9. COTS
   „Apocalypto“
   In-house
   „Fenix“
   9
   Buy
   I‘ll
   Icons by Franziska Haaf binarykitten.de
   Slides by @arghrich
   Spring: Audit
   

   View Slide

10. 10
    In-house
    „Fenix“
    Please Fix
    Icons by Franziska Haaf binarykitten.de
    Slides by @arghrich
    Spring: Audit
    

    View Slide

11. Oh Boy
    

    View Slide

12. We could just …
    12
    Icons by Franziska Haaf binarykitten.de
    Slides by @arghrich
    Spring: Audit
    

    View Slide

13. 13
    Photo by Levi Damasceno from Pexels
    Slides by @arghrich
    Spring: Audit
    

    View Slide

14. The True Cost of Rewrites
    Features
    Releases over time
    Catch Up
    Missing
    Features
    Sub-Par Parity
    Enhancements
    Planned
    Rewrite
    Actual Features
    Undiscovered Scope
    Old App
    Adoption
    Original Article by Doug Bradbury The True Cost of Rewrites
    Slides by @arghrich
    

    View Slide

15. 15
    Photo by Pixabay from Pexels
    Slides by @arghrich
    Early Summer
    

    View Slide

16. How did they get here?
    16 Slides by @arghrich
    Early Summer
    

    View Slide

17. GIGO
    17
    Ideas
    Product
    Process
    (Team)
    Spirit
    Predicts
    Slides by @arghrich
    Early Summer
    

    View Slide

18. GIGO
    18
    Ideas
    Product
    Process
    (Team)
    Spirit
    Fix
    Predicts
    Slides by @arghrich
    Early Summer
    

    View Slide

19. How did we not get here?
    19 Slides by @arghrich
    Early Summer
    

    View Slide

20. 20
    We needed a
    significant
    change
    in team and culture
    Slides by @arghrich
    Early Summer: Spirit
    

    View Slide

21. 21
    Henrik Kniberg
    @henrikkniberg
    Culture is the sum of everyone's
    attitude and actions, so model the
    behaviour you want to see!
    Observe behavior:
    • Who does the management promote?
    • Is there fear of failure?
    • Is quality appreciated or story points or …?
    • What is the delivery „process“?
    • Is there some kind of learning exchange?
    Slides by @arghrich
    Early Summer: Spirit
    

    View Slide

22. 4 Laws of behavior change
    Make it …
    Obvious
    • Define Principles
    together
    • Shared purpose
    Attractive
    • Reduce (time) pressure
    • (Refactoring) score
    22
    Atomic Habits
    Easy
    • Tech-coach
    • Reduce friction
    Satisfying
    • Pair up
    • Never break the chain
    (twice)
    Slides by @arghrich
    Early Summer: Spirit
    

    View Slide

23. 23
    We built a new Co-located
    Cross-functional Team
    with
    shared
    DEV TEST OPS
    Key Accounter
    & principles
    purpose
    Slides by @arghrich
    Early Summer: Spirit
    

    View Slide

24. Our Legacy CD Principles
    • Build quality in
    • Focus on current tree, not on the forest
    • Tackle fear head on
    • Focus on reducing batch size
    • Decouple Deployment and Release
    • Computers perform repetitive tasks, people solve
    problems
    Four Principles of Low-Risk Software Releases by Jez Humble
    CD Principles by Jez Humble
    The Why
    24 Slides by @arghrich
    Early Summer: Spirit
    

    View Slide

25. 25
    Principles provide adaptability
    Slides by @arghrich
    Early Summer: Spirit
    

    View Slide

26. 26
    Icons by Franziska Haaf binarykitten.de
    In-house
    „Fenix“
    Please Cloud
    Slides by @arghrich
    Early Summer: Spirit
    

    View Slide

27. Oh Boy
    

    View Slide

28. Fix
    28
    Product
    Process
    (Team)
    Spirit
    Predicts
    Slides by @arghrich
    Summer
    

    View Slide

29. Local
    Env Source
    Repository
    Artifact
    Repository
    Commit
    Stage
    Acceptance
    Manual
    Test
    Env
    …
    Staging
    Env
    Prod
    Env
    Self-Service
    Automatic
    Feedback (Tag Artifact)
    Dave Farley - Optimising Continuous Delivery
    Slides by @arghrich
    Our CD Pipe
    

    View Slide

30. • Don‘t Branch
    • Don‘t Branch
    • Don‘t Branch
    • Never Branch to Refactor
    • Branches indicate
    Ci
    =Continuous Isolation
    Strategy by Dave Farley
    Our Branching Strategy
    30 Slides by @arghrich
    Summer: Process
    

    View Slide

31. • Code
    • Config
    • Deployment scripts
    • Infrastructure scripts
    • Docs
    • …
    31
    We versioned and tested Everything
    Slides by @arghrich
    Summer: Process
    

    View Slide

32. They used File Version
    Anti-Practice
    32
    File tree from actual project
    Slides by @arghrich
    Summer: Process
    

    View Slide

33. 33
    Photo by Daniel Frank from Pexels
    Slides by @arghrich
    Autumn
    

    View Slide

34. Fix
    34
    Product
    Process
    (Team)
    Spirit
    Slides by @arghrich
    Autumn
    

    View Slide

35. Where do we start?
    35 Slides by @arghrich
    Autumn
    

    View Slide

36. Art by Zaufishan
    36 Slides by @arghrich
    Autumn: Product
    

    View Slide

37. github.com/MaibornWolff/codecharta
    37
    Shameless
    Slides by @arghrich
    Autumn: Product
    

    View Slide

38. Visualize metrics with buildings
    38
    AbstractService.java
    Lines of code
    Complexity
    number of authors
    Icons by Fontawesome
    Very Shameless
    Slides by @arghrich
    Autumn: Product
    

    View Slide

39. Find potential hotspots
    39
    Still Shameless
    Slides by @arghrich
    Autumn: Product
    

    View Slide

40. No tool to rule them all
    40 Slides by @arghrich
    Continuously
    Periodically
    Structure101
    SonarQube
    CodeCharta
    CodeScene ArchUnit TSArch
    SonarArchitect
    Quality Views by Colin Breck
    Autumn: Product Baseline
    Quality Views
    Shamelessly putting CodeCharta first
    

    View Slide

41. 41
    Photo by Pixabay from Pexels
    Slides by @arghrich
    Autumn: Product Baseline
    Counting Fixed Windows
    

    View Slide

42. 42
    vX - redesigned
    v1 - legacy
    Input for UI/API/DB
    Output
    Output
    Continuous
    Compare
    Slides by @arghrich
    Autumn: Product Baseline
    Golden Master
    Characterizes Behavior before Refactoring
    

    View Slide

43. 43 Slides by @arghrich
    Autumn: Product Baseline
    Golden Master Cheatsheet
    Input
    • Have Lots & Check
    coverage
    • Capture from Production
    • Generate
    • Try Testrecorder
    Output
    • Log messages
    • Datalake
    • Database
    Hidden Inputs
    • Date
    • Randoms
    When
    • Before refactorings
    • Generate new output
    when needed
    

    View Slide

44. How do we refactor what
    we don‘t understand?
    44 Slides by @arghrich
    Autumn: Product Baseline
    

    View Slide

45. 45 Slides by @arghrich
    Autumn: Product Baseline
    Emily Bache
    @emilybache
    „Nopefactoring“
    The No-thinking refactoring“
    Advanced Testing & Refactoring Techniques
    • Lift-up conditional
    • Split to Classes
    Llewellyn Falco
    @LlewellynFalco
    Cutting Code Quickly
    

    View Slide

46. 46
    Icons by Franziska Haaf binarykitten.de
    In-house
    „Fenix“
    Add Feature
    Slides by @arghrich
    Autumn
    

    View Slide

47. Fuuuu
    47
    Photo by Bartek Wojtas from Pexels
    Slides by @arghrich
    Autumn
    

    View Slide

48. • Write categorization, if possible function Tests
    • Safely verify it can handle production
    • Divide code into stable and unstable
    • Understand Code Domain
    48 Slides by @arghrich
    Autumn: Conquer the Fear
    Conquer the fear
    

    View Slide

49. Learn Domain with
    Reverse Object Mother
    • Start application with empty database and without domain
    model
    • Click through a UseCase
    • Analyse exceptions and errors
    „App needs at least an object A with this field“
    • Expand Domain Modell with your finding
    • Create required state in the DB with your model
    • Document finding as categorization test
    • Repeat
    49 Slides by @arghrich
    Autumn: Conquer the Fear
    

    View Slide

50. 50
    Photo by Emmet from Pexels
    Slides by @arghrich
    Autumn: Conquer the Fear
    

    View Slide

51. • Direct Read/Write à Facades
    • Classes à Interfaces + Dependency Injection
    • Fire&Forget calls à Events
    • Obsolete/duplicate tables à Table Views
    • Dynamic GET/POST à Explicit constants
    51
    Photo by Krizjohn Rosales from Pexels
    Isolate
    Stable from Unstable
    Working
    Effectively with
    Legacy Code
    Slides by @arghrich
    Autumn: Conquer the Fear
    

    View Slide

52. Sprout quality code
    52
    Photo by Alex Fu from Pexels
    Working
    Effectively with
    Legacy Code
    Table View
    Interface Facade Event
    Slides by @arghrich
    Autumn: Conquer the Fear
    

    View Slide

53. Use Strangler Application
    for bigger features
    Picture by Paul Hammant
    Great talk Strangling the Monolith With a Data-Driven Approach
    53 Slides by @arghrich
    Autumn: Conquer the Fear
    

    View Slide

54. 54
    Photo by Riccardo Bresciani from Pexels
    Slides by @arghrich
    Winter
    

    View Slide

55. Slides by @arghrich
    vX - redesigned
    55
    Icons by Fontawesome
    Load Balancer
    v1 - legacy
    Continuous
    Compare
    Same number of:
    • 200?
    • 404?
    • 5xx?
    • …
    Winter: Launch confidence
    Dark Launch to gain confidence
    

    View Slide

56. Remove
    any
    obsolete code
    in
    Tech || Business
    56
    Photo by Stas Knop from Pexels
    Slides by @arghrich
    Winter: Remove waste
    Don‘t fix what doesn‘t need fixing
    

    View Slide

57. Finding obsolete code
    • Add Analytics / Activity-Log for your dark launch pages
    • User surveys
    • IDE suggestions
    • grep FROM, JOIN, INSERT, UPDATE, DELETE in .asp and
    check if that code is reachable
    • Legacy Toggles
    • Forget to deploy parts of your application
    57 Slides by @arghrich
    Winter: Remove waste
    

    View Slide

58. 58
    Photo by Irina Iriser from Pexels
    Slides by @arghrich
    Next Spring: Summary
    

    View Slide

59. Legacy Delivery
    59
    Photo by pixabay from Pexels
    Before
    • 2 Weeks for a feature
    • 5 months until it works
    After
    • 4 Weeks for a feature
    • Then it works
    Slides by @arghrich
    Next Spring: Summary
    

    View Slide

60. Was it worth it?
    • Users had capitulated
    à then we started fixing bugs
    • More Stable then before
    • We couldn‘t have reimplemented this from scratch
    • à None can describe what „same as before“ means
    • à Even if, it would have taken more than a year
    • Seamless migration from COTS to in-house
    à min. feature freeze
    60 Slides by @arghrich
    Next Spring: Summary
    

    View Slide

61. Would I do it again?
    61 Slides by @arghrich
    Next Spring: Summary
    

    View Slide

62. Hell Yes
    We had
    • Autonomy
    • Mastery
    • Purpose
    62
    Model from „Drive“ by Daniel Pink
    Slides by @arghrich
    Next Spring: Summary
    

    View Slide

63. 63
    (Team)
    Spirit
    • Conquer the
    fear
    • CD Principles
    • Build quality in
    • Shared
    purpose
    Product
    • Isolate stable
    from unstable
    • Golden Master
    • Strangler
    • Feathers
    Process
    • CD Pipeline
    • Infrastructure
    Automation
    Slides by @arghrich
    Next Spring: Summary
    

    View Slide

64. Ken Mugrage
    @kmugrage
    It‘s not CD if you can‘t deploy to
    production right now
    64 Slides by @arghrich
    Next Spring: Summary
    

    View Slide

65. Done
    Questions, Remarks, Sympathy? J
    30 Mio
    510
    Employees
    9th year
    23
    Nations
    300
    Weeks
    Munich, Augsburg,
    Berlin, Frankfurt
    @arghrich
    speakerdeck.com/richargh
    richargh.de/
    Most of our projects
    aren‘t like this J
    Richard Gross pronoun.is/he
    

    View Slide

66. Reading
    • https://trello.com/b/Fdd876S8/continuous-delivery-checklist-
    template
    • I did not think I would show this slide
    66 Slides by @arghrich
    

    View Slide