Modularizing with Packwerk / Modularizando con Packwerk

The talk was given in Spanish but the slides are in English. Video here: https://www.youtube.com/watch?v=TAMMnmuXhE8.

In this talk I expose a bit about our experience modularizing a monolith application leveraging Packwerk from a pragmatic and realistic POV. It includes some useful patterns to decouple God objects and breaking circular dependencies, as well as an example of how to measure progress over time.

Transcript

1. Modularizing with Packwerk Diego Algorta oboxodo Ruby Montevideo Meetup 2024-04-24
2. Who am I?
3. I read these books in 2003 and fell in ♥ with Ruby
4. First Ruby Meetup in UY? 󰑕 - UYLUG 2007-07-28
5. Oh, the youth…
6. Oh, the youth… so full of black hair.
7. so full of black hair. so full of hair. Oh, the youth…
8. Who am I? In relation to Ruby… • Learnt Ruby in 2003 but it didn’t pay the bills • Working remotely with Rails since 2007 for various clients • Cubox 2009-2014 • RubyConf Uruguay 2010-2014 • First tech hire at Quimbee.com 2014-...
9. RubyConf Uruguay 2010, 2011, 2012, 2013, 2014
10. Let’s get serious
11. Example app
12. Example app - just models for brevity Classic patterns from a Rails monolith that has grown over time: • Started as a simple learning system with users taking courses and keeping track of their progress.
13. Example app - just models for brevity Classic patterns from a Rails monolith that has grown over time: • Started as a simple learning system with users taking courses and keeping track of their progress. • After some time a blogging system is added but commenting is implemented as a generic feature for any model (polymorphic). • User ends up being a “God object”. • Namespacing as unenforced modularization. • Original models are not namespaced.
14. What is Packwerk?
15. What is Packwerk? - From Shopify - Straight from their readme file Ruby gem used to enforce boundaries and modularize Rails applications. Packwerk can be used to: • Combine groups of files into packages • Define package-level constant visibility (i.e. have publicly accessible constants) • Help existing codebases to become more modular without obstructing development Source: https://github.com/Shopify/packwerk
16. Combine groups of files into packages (aka packs)
17. Combine groups of files into packages (aka packs) …progressively ♥
18. Ruby at Scale Set of gems built around packwerk • packs • packs-rails • packs-specification • code_ownership • visualize_packs • …and a bunch more… https://github.com/rubyatscale Tools from gusto.com
19. Stuff you can define per pack Built-in: • Dependency Extensions: • Privacy • Layer (formerly “Architecture”) • Ownership • …and more. # packs/my_pack/package.yml enforce_dependencies : true enforce_privacy : false enforce_layers : true owner: Team A layer: Feature dependencies: - packs/foo - packs/bar - packs/baz
20. Example from visualize_packs gem
21. How does a modularized monolith look like? . ├── app │ ├── controllers │ ├── models │ ├── views │ └── ... ├── lib └── spec
22. How does a modularized monolith look like? . ├── app │ ├── controllers │ ├── models │ ├── views │ └── ... ├── lib └── spec . ├── package.yml ├── app │ └── ...as empty as possible... ├── lib ├── packs │ ├── foo │ │ ├── package.yml │ │ ├── package_todo.yml │ │ ├── app │ │ │ ├── controllers │ │ │ ├── models │ │ │ └── views │ │ ├── config │ │ ├── lib │ │ └── spec │ └── bar │ ├── package.yml │ └── ... └── spec
23. How does a modularized monolith look like? . ├── app │ ├── controllers │ ├── models │ ├── views │ └── ... ├── lib └── spec . ├── package.yml ├── app │ └── ...as empty as possible... ├── lib ├── packs │ ├── foo │ │ ├── package.yml │ │ ├── package_todo.yml │ │ ├── app │ │ │ ├── controllers │ │ │ ├── models │ │ │ └── views │ │ ├── config │ │ ├── lib │ │ └── spec │ └── bar │ ├── package.yml │ └── ... └── spec
24. Most important commands # Validates configuration correctness. Run it on the CI. bin/packs validate # Checks code against package_todo.yml files. Run it on the CI! bin/packs check # Updates package_todo.yml files from current code. Be careful! bin/packs update # Help... duh. bin/packs help
25. Splitting strategies You can mix these as needed Vertical • One pack per functionality with all its files (controllers, models, views, etc) Horizontal • Separate packs for Models, Views, Controllers, etc.
26. Original
27. Original Adding Packs and Layers
28. Just a few patterns to fix violations/to-dos
29. ⚠ None of these aim for complete decoupling (like microservices)
30. Aim for “I could eventually extract this as a gem or engine” if I really wanted
31. Break belongs_to & has_one/many circular dependency # Given an inter-packs `belongs_to` definition # packs/learning/app/models/enrollment.rb class Enrollment < ApplicationRecord # Adds dependency from Enrollment to User. # Necessary. belongs_to :user end # Bad ❌ # packs/users/app/models/user.rb class User < ApplicationRecord # Adds dependency from User to Enrollment # Necessary? (most of the time, not really). has_many :enrollments, dependent: :destroy end
32. Break belongs_to & has_one/many circular dependency # Given an inter-packs `belongs_to` definition # packs/learning/app/models/enrollment.rb class Enrollment < ApplicationRecord # Adds dependency from Enrollment to User. # Necessary. belongs_to :user end # Bad ❌ # packs/users/app/models/user.rb class User < ApplicationRecord # Adds dependency from User to Enrollment # Necessary? (most of the time, not really). has_many :enrollments, dependent: :destroy end # Good ✅ # Replace `user.enrollments` with: Enrollment.where(user: user) # packs/learning/config/initializers/learning.rb Rails.application.configure do config.to_prepare do # Subscribe to destroy notifications User.before_destroy do |o| Enrollment.where(user: o).destroy_all end end end
33. belongs_to Avoid referring to a model in a higher-layer pack… unless it’s polymorphic which inverts the dependency. <<concern>> Commenting::Commentable Commenting::Comment Post Lesson
34. belongs_to Avoid referring to a model in a higher-layer pack… unless it’s polymorphic which inverts the dependency. <<concern>> Commenting::Commentable Commenting::Comment Post Lesson
35. belongs_to Avoid referring to a model in a higher-layer pack… unless it’s polymorphic which inverts the dependency. <<concern>> Commenting::Commentable Commenting::Comment Post Lesson
36. Polymorphic belongs_to inverts dependency # packs/blogging/app/models/post.rb class Post < ApplicationRecord # Adds dependency from blogging to commenting # pack, which is in lower layer ✅. include Commenting::Commentable end
37. Polymorphic belongs_to inverts dependency # packs/blogging/app/models/post.rb class Post < ApplicationRecord # Adds dependency from blogging to commenting # pack, which is in lower layer ✅. include Commenting::Commentable end # packs/commenting/app/models/concerns/commenting/commentable.rb class Commenting::Commentable < ApplicationRecord extend ActiveSupport::Concern included do has_many :comments, as: :commentable, dependent: :destroy end end # packs/commenting/app/models/commenting/comment.rb class Comment < ApplicationRecord # DOESN’T add dependency on Post which is in # a higher layer ✅. belongs_to :commentable, polymorphic: true end
38. Remove knowledge from God objects # Bad ❌ # app/models/user.rb class User < ApplicationRecord # Adds unnecessary dependency from User to # Post. It’s only important when the user # actually uses the blogging feature. has_many :posts validates :name, presence: true validates :blogger_alias, presence: true end
39. Remove knowledge from God objects # Bad ❌ # app/models/user.rb class User < ApplicationRecord # Adds unnecessary dependency from User to # Post. It’s only important when the user # actually uses the blogging feature. has_many :posts validates :name, presence: true validates :blogger_alias, presence: true end # Good ✅ # packs/users/app/models/user.rb class User < ApplicationRecord # This model being in a pack on the lowest layer # can be referred to from anywhere but must # be very light on dependencies of its own. validates :name, presence: true end # packs/blogging/app/models/blogging/profile.rb class Blogging::Profile < ApplicationRecord # Acts as the central point of contact from the # blogging pack to the User. belongs_to :user delegate :name, to: :user has_many :posts validates :blogger_alias, presence: true end
40. Avoid has_many :through through models in lower layers # Bad ❌ class User has_one :profile, class_name: "Blogging::Profile" has_many :enrollments end class Blogging::Profile belongs_to :user # This forces User to know about Enrollment has_many :enrollments, through: :user end class Enrollment belongs_to :user end
41. Avoid has_many :through through models in lower layers # Bad ❌ class User has_one :profile, class_name: "Blogging::Profile" has_many :enrollments end class Blogging::Profile belongs_to :user # This forces User to know about Enrollment has_many :enrollments, through: :user end class Enrollment belongs_to :user end
42. Avoid has_many :through through models in lower layers # Bad ❌ class User has_one :profile, class_name: "Blogging::Profile" has_many :enrollments end class Blogging::Profile belongs_to :user # This forces User to know about Enrollment has_many :enrollments, through: :user end class Enrollment belongs_to :user end
43. Avoid has_many :through through models in lower layers # Bad ❌ class User has_one :profile, class_name: "Blogging::Profile" has_many :enrollments end class Blogging::Profile belongs_to :user # This forces User to know about Enrollment has_many :enrollments, through: :user end class Enrollment belongs_to :user end
44. Avoid has_many :through through models in lower layers # Bad ❌ class User has_one :profile, class_name: "Blogging::Profile" has_many :enrollments end class Blogging::Profile belongs_to :user # This forces User to know about Enrollment has_many :enrollments, through: :user end class Enrollment belongs_to :user end # Good ✅ class User; end class Blogging::Profile belongs_to :user # Join directly using data in common (user_id), # but avoiding the need for User model to know # about the Enrollment model. has_many :enrollments, foreign_key: :user_id, primary_key: :user_id end class Enrollment belongs_to :user end
45. Use configuration when needed - contrived example, I know 󰤇 # Bad ❌ <!-- Inside some view in packs/commenting --> <p> Commenter: <%= user_display_name(@comment.user) %> </p> # packs/commenting/app/helpers/commenting/comments_helper.rb module Commenting::CommentsHelper def user_display_name(user) "#{user.name} (#{user.posts.count})" end end
46. Use configuration when needed - contrived example, I know 󰤇 # Bad ❌ <!-- Inside some view in packs/commenting --> <p> Commenter: <%= user_display_name(@comment.user) %> </p> # packs/commenting/app/helpers/commenting/comments_helper.rb module Commenting::CommentsHelper def user_display_name(user) "#{user.name} (#{user.posts.count})" end end # Good ✅ <!-- Inside some view in packs/commenting --> <p> Commenter: <%= Commenting.user_display_name[@comment.user] %> </p> # packs/commenting/app/models/commenting.rb module Commenting mattr_accessor :user_display_name, default: ->(user) { user.name } end # config/initializers/commenting.rb Rails.application.configure do config.to_prepare do Commenting.user_display_name = ->(user) { profile = Profile.find_by(user: user) "#{user.name} (#{profile.posts.count})" } end end
47. How much should we modularize?
48. Let’s define a “Maturity Framework” Idea from: https://rubyrailsmodularity.com/t/seeking-advice-on-implementing-a-package-level-guide-to-categorize-and-track-progress/799
49. Milestone / Level 0 1 2 3 4 Packs exist ✅ ✅ ✅ ✅ ✅ All ruby files exist within a pack ✅ ✅ ✅ ✅ Checks are integrated into the build pipeline ✅ ✅ ✅ ✅ There are no architectural violations ✅ ✅ ✅ Packs have code owner and basic documentation ✅ ✅ ✅ Packs have an initial public API ✅ ✅ Packs are well-documented ✅ ✅ There are no dependency violations ✅ ✅ Packs interfaces don’t use ActiveRecord objects ✅ There are no privacy violations ✅ Maturity Framework
50. One size doesn’t fit all
51. Pareto principle (aka 80/20 rule)
52. Maturity Framework Milestone / Level 0 1 ⭐ 3 4 Packs exist ✅ ✅ ✅ ✅ ✅ All ruby files exist within a pack ✅ ✅ ✅ ✅ Checks are integrated into the build pipeline ✅ ✅ ✅ ✅ There are no architectural violations ✅ ✅ ✅ Packs have code owner and basic documentation ✅ ✅ ✅ Packs have an initial public API ✅ ✅ Packs are well-documented ✅ ✅ There are no dependency violations ✅ ✅ Packs interfaces don’t use ActiveRecord objects ✅ There are no privacy violations ✅
53. We found great value on Layers Focus on layers dependencies first • Higher layer ➡ lower layer ✅ • Lower layer ➡ higher layer ❌ Then: • A pack on Application layer can depend on packs on ANY other layer. • A pack on Infrastructure layer can ONLY depend on packs on the same layer. # example layers in packwerk.yml layers: - Application # Rails app itself - Products # Cohesive “sub-apps” - Features # Shareable or small - Infrastructure # Globals
54. Measuring progress over time
55. Measuring progress over time $ bin/packs get_info --format csv --types layer --include-date Date,Pack name,Owned by,Size,Public API,Inbound layer violations,Outbound layer violations 2025-04-20,.,No one,95,app/public,0,0 2025-04-20,packs/blogging,No one,13,packs/blogging/app/public,1,0 2025-04-20,packs/learning,No one,12,packs/learning/app/public,4,0 2025-04-20,packs/content,No one,12,packs/content/app/public,0,3 2025-04-20,packs/users,No one,12,packs/users/app/public,0,3 2025-04-20,packs/commenting,No one,8,packs/commenting/app/public,1,0 2025-04-20,packs/rails_shims,No one,5,packs/rails_shims/app/public,0,0
56. Date Pack name Owned by Size Public API Inbound layer violations Outbound layer violations 2025-04-20 . No one 95 app/public 0 0 2025-04-20 packs/blogging No one 13 packs/blogging/app/public 1 0 2025-04-20 packs/learning No one 12 packs/learning/app/public 4 0 2025-04-20 packs/content No one 12 packs/content/app/public 0 3 2025-04-20 packs/users No one 12 packs/users/app/public 0 3 2025-04-20 packs/commenting No one 8 packs/commenting/app/public 1 0 2025-04-20 packs/rails_shims No one 5 packs/rails_shims/app/public 0 0 Measuring progress over time
57. Real Quimbee modularization progress pivot table sample
58. Real Quimbee modularization progress pivot table sample Decided to stop bleeding!
59. Real Quimbee modularization progress pivot table sample Real push during Jan-Apr 2024
60. Real Quimbee modularization progress pivot table sample Reduced our layer violations by almost 60%!
61. Real Quimbee modularization progress pivot table sample Damn it! We lost momentum and started bleeding a bit again…
62. Real Quimbee modularization progress pivot table sample But we’ve done a decent job at keeping the absolute number of problem packs reduced
63. The End No time for more… 🫤
64. Questions? Diego Algorta @oboxodo https://ob.oxo.do diego@oboxodo.com Some resources: https://leanpub.com/package-based-rails-applications Stephan Hagemann https://rubyrailsmodularity.com https://github.com/rubyatscale https://railsatscale.com