The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024

As Rails applications grow over time and turn into a so-called “ball of mud”, organizations ask themselves what’s next? Should we stay the course with a monolith or migrate to microservices? At Shopify we went down the path of modularizing our monolith and since then GitHub, Gusto, and others have followed our lead. But after 6 years it’s time to ask ourselves: “Did we fix what we set out to fix? Is this better than before?”

Transcript

1. The Myth of the Modular Monolith Eileen M. Uchitelle Rails World 2024
2. Eileen M. Uchitelle @eileencodes eileencodes.com
3. Ruby & Rails Infra at Shopify
4. Rails Core Team since 2017
5. 2010 🧍 My 14 years of Rails Small & large companies
6. 2010 2024 My 14 years of Rails Small & large companies 🌎 🧍
7. 2010 My 14 years of Rails New & legacy applications 👧
8. 2010 2024 👧 My 14 years of Rails New & legacy applications 👵
9. 2010 👩💻 My 14 years of Rails New developers & DHH
10. 2010 2024 My 14 years of Rails New developers & DHH 🏎 👩💻
11. 2014 My 14 years of Rails Applications I've worked on
12. 2014 2022 My 14 years of Rails Applications I've worked on
14. 2014 2024 My 14 years of Rails Applications I've worked on 2017
15. 💩💻
21. The Myth of the Modular Monolith
22. Common problems in large monoliths
23. Common problems: Architectural
24. Architectural ‣ Lack of organization and structure
25. Architectural ‣ Lack of organization and structure ‣ Tight coupling & no boundaries
26. Common problems: Operational
27. Operational ‣ Flaky tests and slow CI Architectural ‣ Lack of organization and structure ‣ Tight coupling & no boundaries
28. Operational ‣ Flaky tests and slow CI ‣ Scalability of deployments Architectural ‣ Lack of organization and structure ‣ Tight coupling & no boundaries
29. Common problems: Organizational
30. Operational ‣ Flaky tests and slow CI ‣ Scalability of deployments Organizational ‣ Dif fi cult to assign and fi nd owners Architectural ‣ Lack of organization and structure ‣ Tight coupling & no boundaries
31. Operational ‣ Flaky tests and slow CI ‣ Scalability of deployments Organizational ‣ Dif fi cult to assign and fi nd owners ‣ Onboarding new hires takes too long Architectural ‣ Lack of organization and structure ‣ Tight coupling & no boundaries
32. "I can't hold the whole app in my head!"
33. *slaps roof of head*
34. *slaps roof of head* This bad boy can fi t so many models.
35. "I can't hold the whole app in my head!"
36. Operational ‣ Flaky tests and slow CI ‣ Scalability of deployments Organizational ‣ Dif fi cult to assign and fi nd owners ‣ Onboarding new hires is dif fi cult Architectural ‣ Lack of organization and structure ‣ Tight coupling & no boundaries
37. What is a modular monolith?
38. Modular monolith: Use the same language.
39. Modular monolith: Use the same infrastructure.
40. Modular monolith: Path to package isolation.
41. How to implement a modular monolith
42. 📁 app 📁 models 📄 dog.rb 📄 cat.rb 📄 tree.rb 📄 flower.rb 📁 controllers 📄 dogs_controller.rb 📄 cats_controller.rb Fur & Foliage Application
43. 📁 packages 📁 pets 📄 dog.rb 📄 cat.rb 📁 app 📁 models 📁 plants 📄 tree.rb 📄 flower.rb 📁 app 📁 models
44. 📁 packages 📁 pets 📄 dog.rb 📄 cat.rb 📁 app 📁 models 📁 plants 📄 tree.rb 📄 flower.rb 📁 app 📁 models
45. 📁 packages 📁 pets 📄 dog.rb 📄 cat.rb 📁 app 📁 models 📁 plants 📄 tree.rb 📄 flower.rb 📁 app 📁 models
46. 📁 packages 📁 pets 📄 dog.rb 📄 cat.rb 📁 app 📁 models 📁 plants 📄 tree.rb 📄 flower.rb 📁 app 📁 models
47. How to enforce monolith boundaries
48. Packwerk: github.com/Shopify/ packwerk
49. # packages/pets/package.yml metadata: stewards: - "@FurAndFoliage/pets" slack_channels: - "#help-pets" - "#team-pets" enforce_dependencies: strict dependencies: - packages/plants
50. # packages/pets/package_todo.yml packages/billing: "::Billing::Subscriptions::TestHelper": violations: - dependency files: - packages/pets/test/helpers/dog_test_helper.rb packages/stores: "::Store::TimelineEvent": violations: - dependency files: - packages/pets/app/models/dog.rb - packages/pets/app/controllers/dogs_controller.rb
51. The problem with full isolation
52. The problem with full isolation
53. New Challenges: Primitive obsession.
54. # passing primitives instead of the cat AR object class Flower < ApplicationRecord def cat_can_eat?(cat_id) cat = Pets::CatGetter.find_cat(cat_id) cat && self.safe_for_cat?(cat) end end
55. New Challenges: Primitive obsession.
56. New Challenges: Ownership obsession.
57. Pets Plants X
58. New Challenges: Package obsession.
59. Canine Feline Daisy Birch Maple Aster
60. New Challenges: Code duplication.
61. New Challenges: Circular dependencies.
62. Pets Plants
63. How modularization falls short
64. Operational ‣ Flaky tests and slow CI ‣ Scalability of deployments Organizational ‣ Dif fi cult to assign and fi nd owners ‣ Onboarding new hires is dif fi cult Architectural ‣ Lack of organization and structure ‣ Tight coupling & no boundaries
65. Architectural Cultural ‣ Lack of organization and structure ‣ Tight coupling & no boundaries Operational Cultural ‣ Flaky tests and slow CI ‣ Scalability of deployments Organizational Cultural ‣ Dif fi cult to assign and fi nd owners ‣ Onboarding new hires is dif fi cult
66. Improve structure & organization?
67. Improve structure & organization? No.
68. Code is less tightly coupled?
69. Code is less tightly coupled? No.
70. Architectural Cultural ‣ Lack of organization and structure ‣ Tight coupling & no boundaries Operational Cultural ‣ Flaky tests and slow CI ‣ Scalability of deployments Organizational Cultural ‣ Dif fi cult to assign and fi nd owners ‣ Onboarding new hires is dif fi cult
71. Architectural Cultural ‣ Lack of organization and structure ‣ Tight coupling & no boundaries Operational Cultural ‣ Flaky tests and slow CI ‣ Scalability of deployments Organizational Cultural ‣ Dif fi cult to assign and fi nd owners ‣ Onboarding new hires is dif fi cult
72. CI is faster with less fl aky tests?
73. CI is faster with less fl aky tests? No.
74. Deploys are more scalable?
75. Deploys are more scalable? Not yet.
76. Architectural Cultural ‣ Lack of organization and structure ‣ Tight coupling & no boundaries Operational Cultural ‣ Flaky tests and slow CI ‣ Scalability of deployments Organizational Cultural ‣ Dif fi cult to assign and fi nd owners ‣ Onboarding new hires is dif fi cult
77. Architectural Cultural ‣ Lack of organization and structure ‣ Tight coupling & no boundaries Operational Cultural ‣ Flaky tests and slow CI ‣ Scalability of deployments Organizational Cultural ‣ Dif fi cult to assign and fi nd owners ‣ Onboarding new hires is dif fi cult
78. Find & assigning owners is easier?
79. Find & assigning owners is easier? Somewhat.
80. Onboarding new hires is easier?
81. Onboarding new hires is easier? No.
82. Why do we have these problems?
83. You cannot solve human problems with modularity.
84. What causes these problems?
85. Causes: Pressure to ship.
86. Causes: Allowing tech debt to grow.
87. Causes: Hiring faster than you can onboard.
88. Causes: Misaligned incentives.
89. Causes: Blame-based development.
90. How do we fi x these problems?
91. Team culture: Improve developer education.
92. Team culture: Indoctrinate non-Rubyists.
93. Team culture: Reprioritize quality.
94. Team culture: Realign reward incentives.
95. When to implement modularization
96. Modularization: Start with just a few packages.
97. Modularization: Functional isolation.
98. Modularization: Don't do it prematurely.
99. Don't fall for the sunk cost fallacy.
100. Scaling large applications is hard
101. We act like the ball of mud Is inevitable.
102. These problems aren't caused by Ruby on Rails.
103. You can't engineer yourself out of a large org.
104. We want someone or something to blame.
105. Engineering culture comes from the top.
106. You cannot solve human problems with modularity.
107. Let's invest in fi xing our human problems.
108. Thank You ❤