Gradually Modularizing your Monolith with Ruby Packs and Packwerk

Slide 1

Slide 1 text

Gradually Modularizing your Monolith with Ruby Packs and Packwerk Railsconf 2023 - Workshop Alex Evanczuk and Stephan Hagemann

Slide 2

Slide 2 text

󰗞 I am Stephan Head of Product Infrastructure at Gusto https://ruby.social/@shageman stephanhagemann.com gradualmodularization.com 󰗞 I am Alex Product Infrastructure, Modularity at Gusto alexevanczuk@gmail.com

Slide 3

Slide 3 text

cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh

Slide 4

Slide 4 text

Hundreds of engineers Millions of lines of code Thousands of commits per month Packwerk in production dev since 2020 Developing github.com/rubyatscale since 2022 cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh Gusto Context

Slide 5

Slide 5 text

… like gems but with less overhead … and specifically built for working within large apps … and with an extensible ecosystem to gradually modularize over time cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh Packages / Packs !?1!!

Slide 6

Slide 6 text

● Create a high-level structure within your application ● Understand and manage app-internal dependencies ● Reduce incidental complexity by removing dependency cycles ● Create better-defined, domain-specific internal APIs ● Reduce CI build time and cost ● Untangle the ball-of-mud cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh Why packages? Why gradual modularization?

Slide 7

Slide 7 text

leanpub.com/package-based-rails-applications/c/railsconf2023 cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh Ruby/Rails Modularity Slack Server: tinyurl.com/rubyslack

Slide 8

Slide 8 text

Gradual Modularization: The Tools cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh

Slide 9

Slide 9 text

Standing on the shoulders of giants! cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh Packwerk Rubocop Sorbet

Slide 10

Slide 10 text

https://github.com/rubyatscale cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh

Slide 11

Slide 11 text

Packs Rails Package Folders Packwerk Dependency checker Rubocop Packs Packs/TypedPublicApis cop Packs/DocumentedPublicApis cop Packs/ClassMethodsAsPublicApis cop Packs/RootNamespaceIsPackName cop Packwerk extensions Privacy checker Visibility checker Architecture checker Code Ownership Package-oriented ownership Use Packs CLI tools to manage and move files to packs Pack Stats Statistics reporting for packs and packwerk/rubocop usage Gradual Modularization - Tools cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh

Slide 12

Slide 12 text

Where should you start? Gradual Modularization is a bit like choose your own adventure cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh

Slide 13

Slide 13 text

Slide 14

Slide 14 text

https://tinyurl.com/mod-tools-analysis ✅ ✅ cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Let’s get practical! cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh

Slide 18

Slide 18 text

Say hi to our helpers! cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh

Slide 19

Slide 19 text

If you get stuck, send a message to our slack channel: #ws-gradually-modularizing-your-monolith-with-ruby-packs-and-packwerk cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh

Slide 20

Slide 20 text

cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh Intro Install workshop Next Stage! (00 - 06) Understand the steps of the stage Execute all steps against mastodon Execute steps against your app Profit Good stage cadence > stage completion ● Best effort ● Didn’t finish? Continue during the next stage ● We’ll be around after, if you want to discuss more

Slide 21

Slide 21 text

Step through script cd mastodon code ../railsconf-workshop/XX.sh # => run individual transforms manually Checkout the code for the stage cd mastodon git clean -fd && git checkout . && git checkout XX Run & read script cd mastodon code ../railsconf-workshop/XX.sh ../railsconf-workshop/XX.sh Options for getting through the workshop

Slide 22

Slide 22 text

Goal of this workshop: Let’s save mastodon some money on CI! cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh

Slide 23

Slide 23 text

00 Getting set up cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh

Slide 24

Slide 24 text

Stage 00 - part 1 echo " gem 'packs-rails' gem 'use_packs', group: %w(development test) gem 'visualize_packwerk', group: %w(development test) gem 'packwerk', github: 'gradual-systems/packwerk', branch: 'main', group: %w(development test) " >> Gemfile bundle install

Slide 25

Slide 25 text

Stage 00 - part 2 bundle binstubs bundler --force bundle binstub use_packs packwerk echo "pack_paths: - packs/* - . " > packs.yml echo "--require packs/rails/rspec" >> .rspec bin/packwerk init

Slide 26

Slide 26 text

Packwerk might crash on your app In packwerk.yml exclude: - {bin,node_modules,script,tmp,vendor}/**/* - ⚠

Slide 27

Slide 27 text

01 Extracting an admin pack

Slide 28

Slide 28 text

Admin interfaces… everyone’s got one?

Slide 29

Slide 29 text

How do you identify your admin files? #1 Learn your domain, speak to the experts, have a meeting, do some analysis, make some proposals, draft a solution, make sure to get it right, … #2 find . | grep admin

Slide 30

Slide 30 text

Stage 01 echo "Create admin pack" bin/packs create packs/admin bin/packs move packs/admin app/*/admin bin/packs move . packs/admin/app/javascript bin/packwerk update which dot && bin/packs visualize

Slide 31

Slide 31 text

Why not move JS? Because there is usually other stuff going on that is hard to see with ruby tooling Because JS is often “it’s own application” ⚠

Slide 32

Slide 32 text

Moving files typically just works… Except for when it doesn’t… #1 Explicit, manual requires #2 Autoloading failing to find stuff (metaprogramming) ⚠

Slide 33

Slide 33 text

bin/packs visualize #(for step 01)

Slide 34

Slide 34 text

02 Improve admin pack dependencies

Slide 35

Slide 35 text

We can learn a lot from packwerk violations… Do we see if any other code should go in the admin pack?

Slide 36

Slide 36 text

Stage 02 echo "Fix some admin package dependencies" bin/packs move packs/admin app/controllers/api/*/admin bin/packwerk update which dot && bin/packs visualize

Slide 37

Slide 37 text

bin/packs visualize #(for step 02)

Slide 38

Slide 38 text

03 Create the “messy middle” pack

Slide 39

Slide 39 text

What do we do with the fact that there are SO MANY violations from the admin pack and (relatively) few from the root pack??? Moving the few violating files into their own pack would tell us more about their entanglement

Slide 40

Slide 40 text

Stage 03 echo "Create pack for 'messy middle'" bin/packs create packs/messy_middle bin/packs move packs/messy_middle \ `../railsconf-workshop/03_find_files_to_root_violations.rb` bin/packwerk update which dot && bin/packs visualize

Slide 41

Slide 41 text

bin/packs visualize #(for step 03)

Slide 42

Slide 42 text

04 Analyze and empty out the “messy middle”

Slide 43

Slide 43 text

Get yourself a clean dependency graph with this one weird trick…

Slide 44

Slide 44 text

We can move the messy middle into the root package!

Slide 45

Slide 45 text

Stage 04 echo "Merge 'messy middle' with root pack" bin/packs move . packs/messy_middle rm -rf packs/messy_middle bin/packwerk update which dot && bin/packs visualize

Slide 46

Slide 46 text

bin/packs visualize #(for step 04)

Slide 47

Slide 47 text

05 Create user facing app pack

Slide 48

Slide 48 text

Alex and I have a long standing feud with code in the root pack…

Slide 49

Slide 49 text

If packages allow us to separate concepts within our domain, any domain code that is in the root package is entangled with the code that configures the app and makes it runnable… We like the root pack to only do that: application configuration and setup

Slide 50

Slide 50 text

What would we name the pack we extract out of the root pack? What is currently left there? The app the user sees! Let’s make that explicit

Slide 51

Slide 51 text

Stage 05 echo "Create user facing app pack" bin/packs create packs/user_facing_app bin/packs move packs/user_facing_app app bin/packs move . packs/user_facing_app/app/javascript bin/packwerk update which dot && bin/packs visualize

Slide 52

Slide 52 text

bin/packs visualize #(for step 05)

Slide 53

Slide 53 text

06 Accept packwerk dependencies

Slide 54

Slide 54 text

These dependencies look pretty good! Let’s accept them.

Slide 55

Slide 55 text

Stage 06 echo "Accept packwerk dependencies" bin/packs add_dependency . packs/user_facing_app bin/packs add_dependency packs/admin packs/user_facing_app bin/packwerk update which dot && bin/packs visualize

Slide 56

Slide 56 text

bin/packs visualize #(for step 06)

Slide 57

Slide 57 text

There is a step 07…

Slide 58

Slide 58 text

Not yet… but we’re close 3559 tests 425 tests 158 tests Did we save mastodon some money on CI?

Slide 59

Slide 59 text

What about… Routes? Initializers? Other metaprogramming? Does packwerk see all dependencies? 😬 ❌ Dynamic method parameters ❌ Metaprogramming ❌ Monkey-patching ⚠

Slide 60

Slide 60 text

Did we save mastodon some money on CI? Not yet… but we’re close 3559 tests 425 tests 158 tests 425 + 158 = 583 tests to validate changes 158 tests to validate package 3559 + 425 + 158 = 4142 tests to validate changes At Gusto we save about 20% on CI doing this

Slide 61

Slide 61 text

😬 There is no open-source library for this yet Care to contribute? → github.com/rubyatscale 😏 For now, this gist will do the basics 😁 https://tinyurl.com/conditionalpackbuilds How do you benefit from this?

Slide 62

Slide 62 text

Moving files around (even loads of them) isn’t scary! Moving files around leads to interesting insights! Packages give helpful context to our code! Packages can save money on CI! Recap and Summary