$30 off During Our Annual Pro Sale. View Details »

Gradually Modularizing your Monolith with Ruby Packs and Packwerk

Gradually Modularizing your Monolith with Ruby Packs and Packwerk

Are you struggling with large amounts of complexity in your application? A big org with lots of cross-team dependencies? Too much coupling, causing bugs and incidents? Difficult time ramping up new engineers? Or do you just want to improve product optionality by having more reusable building blocks to create new features for your customers?

Join us to apply the packs ecosystem and packwerk to gradually modularize your application. We'll dive into decomposing your application, moving files into packs, and coming up with a technical and managerial plan to successfully roll this out to and get buy-in from the rest of your organization.

shagemann

April 26, 2023
Tweet

More Decks by shagemann

Other Decks in Technology

Transcript

  1. Gradually Modularizing your
    Monolith with Ruby Packs and
    Packwerk
    Railsconf 2023 - Workshop
    Alex Evanczuk and Stephan Hagemann

    View Slide

  2. 󰗞
    I am Stephan
    Head of
    Product Infrastructure at Gusto
    https://ruby.social/@shageman
    stephanhagemann.com
    gradualmodularization.com
    󰗞
    I am Alex
    Product Infrastructure,
    Modularity at Gusto
    [email protected]

    View Slide

  3. cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh
    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh

    View Slide

  4. Hundreds of engineers
    Millions of lines of code
    Thousands of commits per month
    Packwerk in production dev since 2020
    Developing github.com/rubyatscale since 2022
    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh
    Gusto Context

    View Slide

  5. … like gems but with less overhead
    … and specifically built for working within large apps
    … and with an extensible ecosystem to gradually modularize over
    time
    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh
    Packages / Packs !?1!!

    View Slide

  6. ● Create a high-level structure within your application
    ● Understand and manage app-internal dependencies
    ● Reduce incidental complexity by removing dependency cycles
    ● Create better-defined, domain-specific internal APIs
    ● Reduce CI build time and cost
    ● Untangle the ball-of-mud
    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh
    Why packages?
    Why gradual modularization?

    View Slide

  7. leanpub.com/package-based-rails-applications/c/railsconf2023
    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh
    Ruby/Rails Modularity Slack
    Server:
    tinyurl.com/rubyslack

    View Slide

  8. Gradual Modularization:
    The Tools
    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh

    View Slide

  9. Standing on the shoulders of giants!
    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh
    Packwerk Rubocop Sorbet

    View Slide

  10. https://github.com/rubyatscale
    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh

    View Slide

  11. Packs Rails Package Folders
    Packwerk Dependency checker
    Rubocop Packs Packs/TypedPublicApis cop
    Packs/DocumentedPublicApis cop
    Packs/ClassMethodsAsPublicApis cop
    Packs/RootNamespaceIsPackName cop
    Packwerk extensions Privacy checker
    Visibility checker
    Architecture checker
    Code Ownership Package-oriented ownership
    Use Packs CLI tools to manage and move files to packs
    Pack Stats Statistics reporting for packs and packwerk/rubocop usage
    Gradual Modularization - Tools
    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh

    View Slide

  12. Where should you start?
    Gradual Modularization is a bit like
    choose your own adventure
    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh

    View Slide

  13. Packs Rails Package Folders
    Packwerk Dependency checker
    Rubocop Packs Packs/TypedPublicApis cop
    Packs/DocumentedPublicApis cop
    Packs/ClassMethodsAsPublicApis cop
    Packs/RootNamespaceIsPackName cop
    Packwerk extensions Privacy checker
    Visibility checker
    Architecture checker
    Code Ownership Package-oriented ownership
    Use Packs CLI tools to manage and move files to packs
    Pack Stats Statistics reporting for packs and packwerk/rubocop usage
    Gradual Modularization - Tools
    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh

    View Slide

  14. https://tinyurl.com/mod-tools-analysis


    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh

    View Slide

  15. Packs Rails Package Folders
    Packwerk Dependency checker
    Rubocop Packs Packs/TypedPublicApis cop
    Packs/DocumentedPublicApis cop
    Packs/ClassMethodsAsPublicApis cop
    Packs/RootNamespaceIsPackName cop
    Packwerk extensions Privacy checker
    Visibility checker
    Architecture checker
    Code Ownership Package-oriented ownership
    Use Packs CLI tools to manage and move files to packs
    Pack Stats Statistics reporting for packs and packwerk/rubocop usage
    Gradual Modularization - Tools
    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh

    View Slide

  16. Packs Rails Package Folders
    Packwerk Dependency checker
    Rubocop Packs Packs/TypedPublicApis cop
    Packs/DocumentedPublicApis cop
    Packs/ClassMethodsAsPublicApis cop
    Packs/RootNamespaceIsPackName cop
    Packwerk extensions Privacy checker
    Visibility checker
    Architecture checker
    Code Ownership Package-oriented ownership
    Use Packs CLI tools to manage and move files to packs
    Pack Stats Statistics reporting for packs and packwerk/rubocop usage
    Gradual Modularization - Tools
    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh

    View Slide

  17. Let’s get practical!
    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh

    View Slide

  18. Say hi to our helpers!
    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh

    View Slide

  19. If you get stuck, send a message to our
    slack channel:
    #ws-gradually-modularizing-your-monolith-with-ruby-packs-and-packwerk
    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh

    View Slide

  20. cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh
    Intro
    Install workshop
    Next Stage!
    (00 - 06)
    Understand the
    steps of the
    stage
    Execute all steps
    against
    mastodon
    Execute steps
    against your app
    Profit
    Good stage cadence >
    stage completion
    ● Best effort
    ● Didn’t finish? Continue
    during the next stage
    ● We’ll be around after, if
    you want to discuss more

    View Slide

  21. Step through script
    cd mastodon
    code ../railsconf-workshop/XX.sh
    # => run individual transforms manually
    Checkout the code for the stage
    cd mastodon
    git clean -fd && git checkout . && git checkout XX
    Run & read script
    cd mastodon
    code ../railsconf-workshop/XX.sh
    ../railsconf-workshop/XX.sh
    Options for getting through the workshop

    View Slide

  22. Goal of this workshop:
    Let’s save mastodon some money on CI!
    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh

    View Slide

  23. 00
    Getting set up
    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh

    View Slide

  24. Stage 00 - part 1

    echo "
    gem 'packs-rails'
    gem 'use_packs', group: %w(development test)
    gem 'visualize_packwerk', group: %w(development test)
    gem 'packwerk',
    github: 'gradual-systems/packwerk',
    branch: 'main', group: %w(development test)
    " >> Gemfile
    bundle install

    View Slide

  25. Stage 00 - part 2

    bundle binstubs bundler --force
    bundle binstub use_packs packwerk
    echo "pack_paths:
    - packs/*
    - .
    " > packs.yml
    echo "--require packs/rails/rspec" >> .rspec
    bin/packwerk init

    View Slide

  26. Packwerk might crash on your app
    In packwerk.yml
    exclude:
    - {bin,node_modules,script,tmp,vendor}/**/*
    -

    View Slide

  27. 01
    Extracting an admin pack

    View Slide

  28. Admin interfaces… everyone’s got one?

    View Slide

  29. How do you identify your admin files?
    #1
    Learn your domain, speak to the experts, have a
    meeting, do some analysis, make some proposals,
    draft a solution, make sure to get it right, …
    #2
    find . | grep admin

    View Slide

  30. Stage 01
    echo "Create admin pack"
    bin/packs create packs/admin
    bin/packs move packs/admin app/*/admin
    bin/packs move . packs/admin/app/javascript
    bin/packwerk update
    which dot && bin/packs visualize

    View Slide

  31. Why not move JS?
    Because there is usually other stuff going on
    that is hard to see with ruby tooling
    Because JS is often “it’s own application”

    View Slide

  32. Moving files typically just works…
    Except for when it doesn’t…
    #1 Explicit, manual requires
    #2 Autoloading failing to find stuff
    (metaprogramming)

    View Slide

  33. bin/packs visualize #(for step 01)

    View Slide

  34. 02
    Improve admin pack dependencies

    View Slide

  35. We can learn a lot from packwerk
    violations…
    Do we see if any other code should go in the
    admin pack?

    View Slide

  36. Stage 02
    echo "Fix some admin package dependencies"
    bin/packs move packs/admin app/controllers/api/*/admin
    bin/packwerk update
    which dot && bin/packs visualize

    View Slide

  37. bin/packs visualize #(for step 02)

    View Slide

  38. 03
    Create the “messy middle” pack

    View Slide

  39. What do we do with the fact that there are
    SO MANY violations from the admin pack
    and (relatively) few from the root pack???
    Moving the few violating files into their own
    pack would tell us more about their
    entanglement

    View Slide

  40. Stage 03
    echo "Create pack for 'messy middle'"
    bin/packs create packs/messy_middle
    bin/packs move packs/messy_middle \
    `../railsconf-workshop/03_find_files_to_root_violations.rb`
    bin/packwerk update
    which dot && bin/packs visualize

    View Slide

  41. bin/packs visualize #(for step 03)

    View Slide

  42. 04
    Analyze and empty out the “messy middle”

    View Slide

  43. Get yourself a clean
    dependency graph with
    this one weird trick…

    View Slide

  44. We can move the messy middle into the root
    package!

    View Slide

  45. Stage 04
    echo "Merge 'messy middle' with root pack"
    bin/packs move . packs/messy_middle
    rm -rf packs/messy_middle
    bin/packwerk update
    which dot && bin/packs visualize

    View Slide

  46. bin/packs visualize #(for step 04)

    View Slide

  47. 05
    Create user facing app pack

    View Slide

  48. Alex and I have a long standing feud with
    code in the root pack…

    View Slide

  49. If packages allow us to separate concepts
    within our domain, any domain code that is in
    the root package is entangled with the code
    that configures the app and makes it
    runnable…
    We like the root pack to only do that:
    application configuration and setup

    View Slide

  50. What would we name the pack we
    extract out of the root pack?

    What is currently left there?
    The app the user sees! Let’s make that
    explicit

    View Slide

  51. Stage 05
    echo "Create user facing app pack"
    bin/packs create packs/user_facing_app
    bin/packs move packs/user_facing_app app
    bin/packs move . packs/user_facing_app/app/javascript
    bin/packwerk update
    which dot && bin/packs visualize

    View Slide

  52. bin/packs visualize #(for step 05)

    View Slide

  53. 06
    Accept packwerk dependencies

    View Slide

  54. These dependencies
    look pretty good!
    Let’s accept them.

    View Slide

  55. Stage 06
    echo "Accept packwerk dependencies"
    bin/packs add_dependency . packs/user_facing_app
    bin/packs add_dependency packs/admin packs/user_facing_app
    bin/packwerk update
    which dot && bin/packs visualize

    View Slide

  56. bin/packs visualize #(for step 06)

    View Slide

  57. There is a step 07…

    View Slide

  58. Not yet… but we’re close
    3559 tests
    425 tests 158 tests
    Did we save mastodon some money on CI?

    View Slide

  59. What about…
    Routes?
    Initializers?
    Other metaprogramming?
    Does packwerk see all dependencies?
    😬
    ❌ Dynamic method parameters
    ❌ Metaprogramming
    ❌ Monkey-patching

    View Slide

  60. Did we save mastodon some money on CI?
    Not yet… but we’re close
    3559 tests
    425 tests 158 tests
    425 + 158 = 583 tests to
    validate changes
    158 tests to validate
    package
    3559 + 425 + 158 = 4142
    tests to validate changes
    At Gusto we
    save about
    20% on CI
    doing this

    View Slide

  61. 😬 There is no open-source library for this yet
    Care to contribute? → github.com/rubyatscale 😏
    For now, this gist will do the basics 😁
    https://tinyurl.com/conditionalpackbuilds
    How do you benefit from this?

    View Slide

  62. Moving files around (even loads of them) isn’t scary!
    Moving files around leads to interesting insights!
    Packages give helpful context to our code!
    Packages can save money on CI!
    Recap and Summary

    View Slide

  63. Packs Rails Package Folders
    Packwerk Dependency checker
    Rubocop Packs Packs/TypedPublicApis cop
    Packs/DocumentedPublicApis cop
    Packs/ClassMethodsAsPublicApis cop
    Packs/RootNamespaceIsPackName cop
    Packwerk extensions Privacy checker
    Visibility checker
    Architecture checker
    Code Ownership Package-oriented ownership
    Use Packs CLI tools to manage and move files to packs
    Pack Stats Statistics reporting for packs and packwerk/rubocop usage
    Gradual Modularization - Tools

    View Slide

  64. Packs Rails Package Folders
    Packwerk Dependency checker
    Rubocop Packs Packs/TypedPublicApis cop
    Packs/DocumentedPublicApis cop
    Packs/ClassMethodsAsPublicApis cop
    Packs/RootNamespaceIsPackName cop
    Packwerk extensions Privacy checker
    Visibility checker
    Architecture checker
    Code Ownership Package-oriented ownership
    Use Packs CLI tools to manage and move files to packs
    Pack Stats Statistics reporting for packs and packwerk/rubocop usage
    Gradual Modularization - Tools

    View Slide

  65. We hope this got you excited to start your
    modularization journey!

    View Slide

  66. Thank you!
    󰗞
    I am Stephan
    󰗞
    I am Alex

    View Slide

  67. Wait… we have more time?

    View Slide

  68. Your own app!
    Step 07

    View Slide