Gradually Modularizing your Monolith with Ruby Packs and Packwerk

Are you struggling with large amounts of complexity in your application? A big org with lots of cross-team dependencies? Too much coupling, causing bugs and incidents? Difficult time ramping up new engineers? Or do you just want to improve product optionality by having more reusable building blocks to create new features for your customers?

Join us to apply the packs ecosystem and packwerk to gradually modularize your application. We'll dive into decomposing your application, moving files into packs, and coming up with a technical and managerial plan to successfully roll this out to and get buy-in from the rest of your organization.

Transcript

1. Gradually Modularizing your
   Monolith with Ruby Packs and
   Packwerk
   Railsconf 2023 - Workshop
   Alex Evanczuk and Stephan Hagemann
   

   View Slide

2. 󰗞
   I am Stephan
   Head of
   Product Infrastructure at Gusto
   https://ruby.social/@shageman
   stephanhagemann.com
   gradualmodularization.com
   󰗞
   I am Alex
   Product Infrastructure,
   Modularity at Gusto
   alexevanczuk@gmail.com
   

   View Slide

3. cd YOUR_WORKSPACE #ideally, next to your app
   git clone --depth=1 https://github.com/gradual-systems/mastodon.git
   git clone https://github.com/gradual-systems/railsconf-workshop.git
   cd mastodon; ../railsconf-workshop/00.sh
   cd YOUR_WORKSPACE #ideally, next to your app
   git clone --depth=1 https://github.com/gradual-systems/mastodon.git
   git clone https://github.com/gradual-systems/railsconf-workshop.git
   cd mastodon; ../railsconf-workshop/00.sh
   

   View Slide

4. Hundreds of engineers
   Millions of lines of code
   Thousands of commits per month
   Packwerk in production dev since 2020
   Developing github.com/rubyatscale since 2022
   cd YOUR_WORKSPACE #ideally, next to your app
   git clone --depth=1 https://github.com/gradual-systems/mastodon.git
   git clone https://github.com/gradual-systems/railsconf-workshop.git
   cd mastodon; ../railsconf-workshop/00.sh
   Gusto Context
   

   View Slide

5. … like gems but with less overhead
   … and specifically built for working within large apps
   … and with an extensible ecosystem to gradually modularize over
   time
   cd YOUR_WORKSPACE #ideally, next to your app
   git clone --depth=1 https://github.com/gradual-systems/mastodon.git
   git clone https://github.com/gradual-systems/railsconf-workshop.git
   cd mastodon; ../railsconf-workshop/00.sh
   Packages / Packs !?1!!
   

   View Slide

6. ● Create a high-level structure within your application
   ● Understand and manage app-internal dependencies
   ● Reduce incidental complexity by removing dependency cycles
   ● Create better-defined, domain-specific internal APIs
   ● Reduce CI build time and cost
   ● Untangle the ball-of-mud
   cd YOUR_WORKSPACE #ideally, next to your app
   git clone --depth=1 https://github.com/gradual-systems/mastodon.git
   git clone https://github.com/gradual-systems/railsconf-workshop.git
   cd mastodon; ../railsconf-workshop/00.sh
   Why packages?
   Why gradual modularization?
   

   View Slide

7. leanpub.com/package-based-rails-applications/c/railsconf2023
   cd YOUR_WORKSPACE #ideally, next to your app
   git clone --depth=1 https://github.com/gradual-systems/mastodon.git
   git clone https://github.com/gradual-systems/railsconf-workshop.git
   cd mastodon; ../railsconf-workshop/00.sh
   Ruby/Rails Modularity Slack
   Server:
   tinyurl.com/rubyslack
   

   View Slide

8. Gradual Modularization:
   The Tools
   cd YOUR_WORKSPACE #ideally, next to your app
   git clone --depth=1 https://github.com/gradual-systems/mastodon.git
   git clone https://github.com/gradual-systems/railsconf-workshop.git
   cd mastodon; ../railsconf-workshop/00.sh
   

   View Slide

9. Standing on the shoulders of giants!
   cd YOUR_WORKSPACE #ideally, next to your app
   git clone --depth=1 https://github.com/gradual-systems/mastodon.git
   git clone https://github.com/gradual-systems/railsconf-workshop.git
   cd mastodon; ../railsconf-workshop/00.sh
   Packwerk Rubocop Sorbet
   

   View Slide

10. https://github.com/rubyatscale
    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh
    

    View Slide

11. Packs Rails Package Folders
    Packwerk Dependency checker
    Rubocop Packs Packs/TypedPublicApis cop
    Packs/DocumentedPublicApis cop
    Packs/ClassMethodsAsPublicApis cop
    Packs/RootNamespaceIsPackName cop
    Packwerk extensions Privacy checker
    Visibility checker
    Architecture checker
    Code Ownership Package-oriented ownership
    Use Packs CLI tools to manage and move files to packs
    Pack Stats Statistics reporting for packs and packwerk/rubocop usage
    Gradual Modularization - Tools
    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh
    

    View Slide

12. Where should you start?
    Gradual Modularization is a bit like
    choose your own adventure
    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh
    

    View Slide

13. Packs Rails Package Folders
    Packwerk Dependency checker
    Rubocop Packs Packs/TypedPublicApis cop
    Packs/DocumentedPublicApis cop
    Packs/ClassMethodsAsPublicApis cop
    Packs/RootNamespaceIsPackName cop
    Packwerk extensions Privacy checker
    Visibility checker
    Architecture checker
    Code Ownership Package-oriented ownership
    Use Packs CLI tools to manage and move files to packs
    Pack Stats Statistics reporting for packs and packwerk/rubocop usage
    Gradual Modularization - Tools
    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh
    

    View Slide

14. https://tinyurl.com/mod-tools-analysis
    ✅
    ✅
    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh
    

    View Slide

15. Packs Rails Package Folders
    Packwerk Dependency checker
    Rubocop Packs Packs/TypedPublicApis cop
    Packs/DocumentedPublicApis cop
    Packs/ClassMethodsAsPublicApis cop
    Packs/RootNamespaceIsPackName cop
    Packwerk extensions Privacy checker
    Visibility checker
    Architecture checker
    Code Ownership Package-oriented ownership
    Use Packs CLI tools to manage and move files to packs
    Pack Stats Statistics reporting for packs and packwerk/rubocop usage
    Gradual Modularization - Tools
    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh
    

    View Slide

16. Packs Rails Package Folders
    Packwerk Dependency checker
    Rubocop Packs Packs/TypedPublicApis cop
    Packs/DocumentedPublicApis cop
    Packs/ClassMethodsAsPublicApis cop
    Packs/RootNamespaceIsPackName cop
    Packwerk extensions Privacy checker
    Visibility checker
    Architecture checker
    Code Ownership Package-oriented ownership
    Use Packs CLI tools to manage and move files to packs
    Pack Stats Statistics reporting for packs and packwerk/rubocop usage
    Gradual Modularization - Tools
    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh
    

    View Slide

17. Let’s get practical!
    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh
    

    View Slide

18. Say hi to our helpers!
    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh
    

    View Slide

19. If you get stuck, send a message to our
    slack channel:
    #ws-gradually-modularizing-your-monolith-with-ruby-packs-and-packwerk
    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh
    

    View Slide

20. cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh
    Intro
    Install workshop
    Next Stage!
    (00 - 06)
    Understand the
    steps of the
    stage
    Execute all steps
    against
    mastodon
    Execute steps
    against your app
    Profit
    Good stage cadence >
    stage completion
    ● Best effort
    ● Didn’t finish? Continue
    during the next stage
    ● We’ll be around after, if
    you want to discuss more
    

    View Slide

21. Step through script
    cd mastodon
    code ../railsconf-workshop/XX.sh
    # => run individual transforms manually
    Checkout the code for the stage
    cd mastodon
    git clean -fd && git checkout . && git checkout XX
    Run & read script
    cd mastodon
    code ../railsconf-workshop/XX.sh
    ../railsconf-workshop/XX.sh
    Options for getting through the workshop
    

    View Slide

22. Goal of this workshop:
    Let’s save mastodon some money on CI!
    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh
    

    View Slide

23. 00
    Getting set up
    cd YOUR_WORKSPACE #ideally, next to your app
    git clone --depth=1 https://github.com/gradual-systems/mastodon.git
    git clone https://github.com/gradual-systems/railsconf-workshop.git
    cd mastodon; ../railsconf-workshop/00.sh
    

    View Slide

24. Stage 00 - part 1
    
    echo "
    gem 'packs-rails'
    gem 'use_packs', group: %w(development test)
    gem 'visualize_packwerk', group: %w(development test)
    gem 'packwerk',
    github: 'gradual-systems/packwerk',
    branch: 'main', group: %w(development test)
    " >> Gemfile
    bundle install
    

    View Slide

25. Stage 00 - part 2
    
    bundle binstubs bundler --force
    bundle binstub use_packs packwerk
    echo "pack_paths:
    - packs/*
    - .
    " > packs.yml
    echo "--require packs/rails/rspec" >> .rspec
    bin/packwerk init
    

    View Slide

26. Packwerk might crash on your app
    In packwerk.yml
    exclude:
    - {bin,node_modules,script,tmp,vendor}/**/*
    -
    ⚠
    

    View Slide

27. 01
    Extracting an admin pack
    

    View Slide

28. Admin interfaces… everyone’s got one?
    
    

    View Slide

29. How do you identify your admin files?
    #1
    Learn your domain, speak to the experts, have a
    meeting, do some analysis, make some proposals,
    draft a solution, make sure to get it right, …
    #2
    find . | grep admin
    
    

    View Slide

30. Stage 01
    echo "Create admin pack"
    bin/packs create packs/admin
    bin/packs move packs/admin app/*/admin
    bin/packs move . packs/admin/app/javascript
    bin/packwerk update
    which dot && bin/packs visualize
    
    

    View Slide

31. Why not move JS?
    Because there is usually other stuff going on
    that is hard to see with ruby tooling
    Because JS is often “it’s own application”
    ⚠
    

    View Slide

32. Moving files typically just works…
    Except for when it doesn’t…
    #1 Explicit, manual requires
    #2 Autoloading failing to find stuff
    (metaprogramming)
    ⚠
    

    View Slide

33. bin/packs visualize #(for step 01)
    
    

    View Slide

34. 02
    Improve admin pack dependencies
    

    View Slide

35. We can learn a lot from packwerk
    violations…
    Do we see if any other code should go in the
    admin pack?
    
    

    View Slide

36. Stage 02
    echo "Fix some admin package dependencies"
    bin/packs move packs/admin app/controllers/api/*/admin
    bin/packwerk update
    which dot && bin/packs visualize
    
    

    View Slide

37. bin/packs visualize #(for step 02)
    
    

    View Slide

38. 03
    Create the “messy middle” pack
    

    View Slide

39. What do we do with the fact that there are
    SO MANY violations from the admin pack
    and (relatively) few from the root pack???
    Moving the few violating files into their own
    pack would tell us more about their
    entanglement
    
    

    View Slide

40. Stage 03
    echo "Create pack for 'messy middle'"
    bin/packs create packs/messy_middle
    bin/packs move packs/messy_middle \
    `../railsconf-workshop/03_find_files_to_root_violations.rb`
    bin/packwerk update
    which dot && bin/packs visualize
    
    

    View Slide

41. bin/packs visualize #(for step 03)
    
    

    View Slide

42. 04
    Analyze and empty out the “messy middle”
    

    View Slide

43. Get yourself a clean
    dependency graph with
    this one weird trick…
    
    

    View Slide

44. We can move the messy middle into the root
    package!
    
    

    View Slide

45. Stage 04
    echo "Merge 'messy middle' with root pack"
    bin/packs move . packs/messy_middle
    rm -rf packs/messy_middle
    bin/packwerk update
    which dot && bin/packs visualize
    
    

    View Slide

46. bin/packs visualize #(for step 04)
    
    

    View Slide

47. 05
    Create user facing app pack
    

    View Slide

48. Alex and I have a long standing feud with
    code in the root pack…
    
    

    View Slide

49. If packages allow us to separate concepts
    within our domain, any domain code that is in
    the root package is entangled with the code
    that configures the app and makes it
    runnable…
    We like the root pack to only do that:
    application configuration and setup
    
    

    View Slide

50. What would we name the pack we
    extract out of the root pack?
    
    What is currently left there?
    The app the user sees! Let’s make that
    explicit
    

    View Slide

51. Stage 05
    echo "Create user facing app pack"
    bin/packs create packs/user_facing_app
    bin/packs move packs/user_facing_app app
    bin/packs move . packs/user_facing_app/app/javascript
    bin/packwerk update
    which dot && bin/packs visualize
    
    

    View Slide

52. bin/packs visualize #(for step 05)
    
    

    View Slide

53. 06
    Accept packwerk dependencies
    

    View Slide

54. These dependencies
    look pretty good!
    Let’s accept them.
    
    

    View Slide

55. Stage 06
    echo "Accept packwerk dependencies"
    bin/packs add_dependency . packs/user_facing_app
    bin/packs add_dependency packs/admin packs/user_facing_app
    bin/packwerk update
    which dot && bin/packs visualize
    
    

    View Slide

56. bin/packs visualize #(for step 06)
    
    

    View Slide

57. There is a step 07…
    

    View Slide

58. Not yet… but we’re close
    3559 tests
    425 tests 158 tests
    Did we save mastodon some money on CI?
    

    View Slide

59. What about…
    Routes?
    Initializers?
    Other metaprogramming?
    Does packwerk see all dependencies?
    😬
    ❌ Dynamic method parameters
    ❌ Metaprogramming
    ❌ Monkey-patching
    ⚠
    

    View Slide

60. Did we save mastodon some money on CI?
    Not yet… but we’re close
    3559 tests
    425 tests 158 tests
    425 + 158 = 583 tests to
    validate changes
    158 tests to validate
    package
    3559 + 425 + 158 = 4142
    tests to validate changes
    At Gusto we
    save about
    20% on CI
    doing this
    

    View Slide

61. 😬 There is no open-source library for this yet
    Care to contribute? → github.com/rubyatscale 😏
    For now, this gist will do the basics 😁
    https://tinyurl.com/conditionalpackbuilds
    How do you benefit from this?
    

    View Slide

62. Moving files around (even loads of them) isn’t scary!
    Moving files around leads to interesting insights!
    Packages give helpful context to our code!
    Packages can save money on CI!
    Recap and Summary
    

    View Slide

63. Packs Rails Package Folders
    Packwerk Dependency checker
    Rubocop Packs Packs/TypedPublicApis cop
    Packs/DocumentedPublicApis cop
    Packs/ClassMethodsAsPublicApis cop
    Packs/RootNamespaceIsPackName cop
    Packwerk extensions Privacy checker
    Visibility checker
    Architecture checker
    Code Ownership Package-oriented ownership
    Use Packs CLI tools to manage and move files to packs
    Pack Stats Statistics reporting for packs and packwerk/rubocop usage
    Gradual Modularization - Tools
    

    View Slide

64. Packs Rails Package Folders
    Packwerk Dependency checker
    Rubocop Packs Packs/TypedPublicApis cop
    Packs/DocumentedPublicApis cop
    Packs/ClassMethodsAsPublicApis cop
    Packs/RootNamespaceIsPackName cop
    Packwerk extensions Privacy checker
    Visibility checker
    Architecture checker
    Code Ownership Package-oriented ownership
    Use Packs CLI tools to manage and move files to packs
    Pack Stats Statistics reporting for packs and packwerk/rubocop usage
    Gradual Modularization - Tools
    

    View Slide

65. We hope this got you excited to start your
    modularization journey!
    

    View Slide

66. Thank you!
    󰗞
    I am Stephan
    󰗞
    I am Alex
    

    View Slide

67. Wait… we have more time?
    

    View Slide

68. Your own app!
    Step 07
    

    View Slide