Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Continuous Delivery for Legacy Code - DevOpsCon Winter 2022

Richard
December 06, 2022

Continuous Delivery for Legacy Code - DevOpsCon Winter 2022

This is based on a true story.

My day job is software archeology. I find joy in recovering and analyzing code bones and culture as well as making the skeleton walk again. A short time ago, however, I was confronted with the most horrible code base I have ever seen. This talk is about how we managed to save it and achieve bi-weekly deployments with a high level of confidence.

Five million lines of code in multiple languages (Classic ASP, .NET, VBScript, VBA, JavaScript, T-SQL, PL-SQL) in one monolith. The business logic stretched from the UI (WebForms, Scripting, SQL Queries) down to the database (Stored Procedures), there was no test coverage and an enormous amount of hidden coupling. A version control system was not used, we had no test environment, deployments required developers to copy their local compilation to production and multiple customer installations are supported by uncommenting and commenting code.

Together we will explore what to do when you inherit such a thing: how to identify hotspots, find hidden coupling, explore how connascence can help you, ways to test as well as refactor and how to achieve a regular deployment schedule.

Richard

December 06, 2022
Tweet

More Decks by Richard

Other Decks in Programming

Transcript

  1. Continuous Delivery
    for Legacy Code
    06.12.22
    Richard Gross (he/him)
    IT-Tailor Archaeologist Auditor
    richargh.de/
    speakerdeck.com/richargh
    @arghrich

    View Slide

  2. Slides by @arghrich
    What are the properties of low-
    quality software?
    2

    View Slide

  3. Slides by @arghrich
    3
    Dave Farely, author of the best-selling book “Continuous Delivery”
    The Quality of a System
    is Defined by
    Our Ability to Change it! Dave Farley
    @davefarley77

    View Slide

  4. Slides by @arghrich
    Bad (legacy) software is
    software where we no longer
    have any options
    4

    View Slide

  5. Slides by @arghrich
    A story of the worst software
    I have ever and you will never
    see.
    5
    And we still got to Continuous Delivery J

    View Slide

  6. Slides by @arghrich
    Along comes a customer …
    6 Audit
    Icons by Franziska Haaf binarykitten.de
    COTS
    „Apocalypto“
    Please
    Audit
    Customer
    Me

    View Slide

  7. Slides by @arghrich
    4M LoC Delivery
    7 Audit
    Excel
    Webservice
    Website
    Fat Client
    Jobs
    Database T-SQL
    JavaScript
    VBA
    C#
    Asp.NET
    + C#
    Classic Asp
    + VBScript
    WinForms
    + C#
    MIA
    5M
    No Code
    delivered

    View Slide

  8. Slides by @arghrich
    Copy-Based Version ControlTM with
    production server as origin
    8 Audit
    File tree from actual project

    View Slide

  9. Slides by @arghrich
    Cycle Time of 6 months
    9 Audit
    weeks
    Dev Machine
    Cp from Prod
    Develop
    Compile
    Don‘t Test
    Release
    Cp to Prod
    1 hour
    Bugfixing
    Return to
    Start
    5 months
    6 months

    View Slide

  10. Slides by @arghrich
    „Interesting“
    Deployment
    15 years of
    Uncontrolled growth
    Horrible Cycle Time
    No Tests
    Conflated
    Logic, UI & SQL
    Too much to fix
    Not Versioned
    & Doesn‘t
    Compile
    Icon by Franziska Haaf binarykitten.de
    10

    View Slide

  11. Slides by @arghrich
    11
    Icons by Franziska Haaf binarykitten.de
    COTS
    „Apocalypto“
    In-house
    „Fenix“
    Buy
    I‘ll
    Customer Money

    View Slide

  12. Slides by @arghrich
    12
    Icons by Franziska Haaf binarykitten.de
    In-house
    „Fenix“
    Please
    Fix
    Me
    Customer

    View Slide

  13. Oh Boy

    View Slide

  14. Ken Mugrage
    @kmugrage
    Legacy Code is
    code that’s too
    scary to update
    and too profitable
    to delete.
    It‘s not CD if
    you can‘t deploy
    to production
    right now
    Dylan Beattie
    @dylanbeattie
    Slides by @arghrich
    14

    View Slide

  15. Slides by @arghrich
    Can we deploy this to
    production right now?
    Nope
    15

    View Slide

  16. 16
    Slides by @arghrich
    Not Versioned
    & Doesn‘t
    Compile
    Icon by Franziska Haaf binarykitten.de
    16

    View Slide

  17. • Code
    • Configuration
    • Deployment scripts
    • Infrastructure Provisioning
    • Alerting
    • Monitoring
    • Documentation
    • Onboarding.sh Scripts
    • …
    17
    Put everything into version control
    Slides by @arghrich
    17
    Git

    View Slide

  18. Slides by @arghrich
    Establish a Pipeline
    18
    Commit Stage
    Compile
    Git

    View Slide

  19. Slides by @arghrich
    Versioned &
    Automated Compile
    „Interesting“
    Deployment
    19

    View Slide

  20. Slides by @arghrich
    Scripts to Provision Infrastructure
    Same but parameterized Scripts for TST, STG and PRD
    • Machine
    • Web Server
    • Application Server
    • Database
    • Email Server
    • FTP Server
    • Pipeline Server
    20
    Phoenix Infra: With one click we can
    (re)create any infrastructure
    Gives us

    View Slide

  21. Slides by @arghrich
    Deploy via Pipeline to Test Env
    21
    Commit Stage
    Compile
    Git
    Deploy To
    TST (Cont.)
    STG (Click)
    Smoke Test Deploy

    View Slide

  22. Slides by @arghrich
    Can we deploy this to
    production right now?
    We have a button to do it, but do we want to?
    22

    View Slide

  23. Automated
    Deployment
    Slides by @arghrich
    Versioned &
    Automated Compile
    15 years of
    Uncontrolled growth

    View Slide

  24. Slides by @arghrich
    Don’t “Fix everything at once,
    mindset”TM
    24

    View Slide

  25. Slides by @arghrich
    Do the “Hotspot Mindset”TM
    25

    View Slide

  26. Slides by @arghrich
    Finding Hotspots
    26

    View Slide

  27. Art by Zaufishan
    Slides by @arghrich
    27

    View Slide

  28. Visualize metrics with CodeCharta buildings
    SomeService.cs
    Lines of code
    Complexity
    number of authors
    Icons by Fontawesome
    Open-Source Tool CodeCharta:
    https://maibornwolff.github.io/codecharta/
    Slides by @arghrich
    28

    View Slide

  29. Find potential hotspots
    Open-Source Tool CodeCharta:
    https://maibornwolff.github.io/codecharta/
    Slides by @arghrich
    29
    Lot‘s of code, very complex,
    and everyone has to change it

    View Slide

  30. Slides by @arghrich
    No tool to rule them all
    30
    Shamelessly putting CodeCharta first
    1 Quality Views by Colin Breck
    Continuously
    Periodically
    Structure101
    SonarQube
    CodeCharta
    CodeScene ArchUnit TSArch
    SonarArchitect
    Quality Views 1

    View Slide

  31. Slides by @arghrich
    31
    1 By Matthias Verraes
    2 Graphana Graphs + Push Gateway
    Photo by Pixabay from Pexels
    Count
    Fixed
    Windows 2
    Three
    Post-it‘s
    To
    Refactor 1
    Slides by @arghrich

    View Slide

  32. Versioned &
    Automated Compile
    Automated
    Deployment
    We know what to focus on
    No Tests
    Conflated
    Logic, UI & SQL

    View Slide

  33. Slides by @arghrich
    Start with tests for critical use cases and
    hotspots
    • ~10 E2E Tests
    For critical use cases
    • DB-Integration Tests
    For direct-db-access hotspots that you will refactor
    • Unit tests (should give 80% confidence)
    For refactored hotspots
    For no-direct-db-access hotspots
    For new code
    33

    View Slide

  34. Slides by @arghrich
    Categorization test1: Write Tests for code
    that you don’t understand
    34
    1 Also called approval or golden master test.
    2 Put system in a known (database) state is also an input.
    3 Log messages you add are also an output.
    vX - redesigned
    v1 - legacy
    Input for UI/API/DB 2
    Output 3
    Output 3
    Continuous
    Compare

    View Slide

  35. Slides by @arghrich
    Understand Coupling and Cohesion
    High Coupling
    Low Cohesion
    Low Coupling
    High Cohesion
    35
    Unknown source

    View Slide

  36. Slides by @arghrich
    Uncouple your Code
    36
    Delete Obsolete
    (unreachable or
    unused) Code
    Categorization
    Test, then
    Refactor Code
    Sprout new
    tested Code
    Break Dependencies
    (Seams)
    Strangler Fig
    Unfixable code
    Feature Flag
    Legacy Code
    Write
    ArchUnit Tests

    View Slide

  37. Slides by @arghrich
    Can we deploy this to
    production right now?
    Hmm, it’s somewhat tested but can we really be sure nothing breaks in
    production?
    37

    View Slide

  38. 38
    Automated
    Deployment
    Slides by @arghrich
    Versioned &
    Automated Compile
    Seperated
    Logic, UI & SQL
    In Hotspots
    Critical Use Case Tests
    Growing Unit Tests
    Unit tested new code
    We know what to focus on Horrible Cycle Time

    View Slide

  39. Slides by @arghrich
    Do Monitor in Production
    39

    View Slide

  40. Slides by @arghrich
    RED Metrics
    40
    1 https://thenewstack.io/monitoring-microservices-red-method/
    Community Dashboard https://community.grafana.com/t/how-can-i-configure-red-dashboard/69885
    R -> Request Rate E -> Errors / second
    D -> Duration per request

    View Slide

  41. Slides by @arghrich
    New Cycle Time, without bug-cycle
    41
    Slide adapted from Dave Farley
    Commit Stage
    Compile
    Unit Test
    Lint
    Security
    Docker
    Release
    Git
    Monitor
    Logs
    Events
    Metrics
    Alerts
    Automated
    Performance
    testing
    20 min
    Deploy To
    TST (Cont.)
    STG (Click)
    Smoke Test
    <1 min
    Manual
    Testing
    30 min
    Automated
    Acceptance
    Testing
    User Acceptance Test
    DB-Integration Test
    20 min
    5 min
    60 min

    View Slide

  42. Slides by @arghrich
    Can we deploy this to
    production right now?
    Yes. Depending on where we make the change, even within 30min.
    43

    View Slide

  43. Slides by @arghrich
    But there is still a problem…
    44

    View Slide

  44. Slides by @arghrich
    How did the previous
    developers get here?
    45

    View Slide

  45. Slides by @arghrich
    Healthy Software
    46
    Ideas
    Product
    Process
    Mindset
    Predicts

    View Slide

  46. Slides by @arghrich
    GIGO
    47
    Ideas
    Product
    Process
    Mindset
    Predicts

    View Slide

  47. Slides by @arghrich
    A Feature Factory helps no one
    48
    https://cutlefish.substack.com/p/tbm-4952-your-calendar-your-priorities

    View Slide

  48. Slides by @arghrich
    49
    Henrik Kniberg
    @henrikkniberg
    Culture is the sum of everyone's
    attitude and actions, so model the
    behaviour you want to see!
    Observe behavior:
    • Who does the management promote?
    • Is there fear of failure?
    • Is quality appreciated or story points or …?
    • What is the delivery „process“?
    • Is there some kind of learning exchange?

    View Slide

  49. Slides by @arghrich
    Make the desired behavior …
    50
    Obvious
    • Define Principles
    together
    • Shared purpose
    Attractive
    • Reduce (time)
    pressure
    • (Refactoring) score
    Atomic Habits
    4 Laws of
    behavior change
    Easy
    • Tech-coach
    • Reduce friction
    Satisfying
    • Pair up
    • Never break the chain
    (twice)

    View Slide

  50. Slides by @arghrich
    CD is a change in mindset
    51

    View Slide

  51. Slides by @arghrich
    CD = Working in a way that
    software is always in a
    releasable state
    It’s not about the tools, even though they are getting better all the time
    52

    View Slide

  52. Slides by @arghrich
    Thanks
    53
    IT-Tailor
    Richard Gross (he/him)
    Archaeologist Auditor
    richargh.de/
    speakerdeck.com/richargh
    @arghrich
    Works at maibornwolff.de/
    Tailored Software by People
    who Care.
    DE TN ES

    View Slide

  53. Slides by @arghrich
    Backup
    56

    View Slide

  54. Slides by @arghrich
    57

    View Slide

  55. Slides by @arghrich
    58

    View Slide

  56. Slides by @arghrich
    10 customer installations
    59
    Frobozz
    Co.
    Oceanic
    Airlines

    Umbrella
    Corp
    Soylent
    Corp

    View Slide

  57. Slides by @arghrich
    And also 10 separate codebases
    60

    Oceanic
    Airlines
    Soylent
    Corp
    Frobozz
    Co.
    Umbrella
    Corp
    Similar
    C/P
    Copy/
    Paste
    Feature
    Copy/Paste

    View Slide

  58. Slides by @arghrich
    10 codebases And 10 database schemata
    61
    Similar Similar

    View Slide