[RailsConf 2020] Between monoliths and microservices

Video: https://railsconf.com/2020/video/vladimir-dementyev-between-monoliths-and-microservices

Code: https://github.com/palkan/engems

Rails applications tend to turn into giant monoliths. Keeping the codebase maintainable usually requires architectural changes. Microservices? A distributed monolith is still a monolith. What about breaking the code into pieces and rebuild a monolithic puzzle out of them?

In Rails, we have the right tool for the job: engines. With engines, you can break your application into components—the same way as Rails combines all its parts, which are engines, too.

This path is full of snares and pitfalls. Let me be your guide in this journey and share the story of a monolith engine-ification.

Transcript

1. BETWEEN Vladimir Dementyev Evil Martians Microservices Monoliths
2. 2 rails new monolith
4. The Big Four vs Others 4
5. We are doomed 5
6. Can you escape your destiny? 6 V.Yurlov, for “The war of the worlds” by H.G.Wells
8. 8 A software system is called “monolithic” if it has a monolithic architecture, in which functionally distinguishable aspects are all interwoven, rather than containing architecturally separate components.—Wiki
9. Components 9
10. Monolith 10
11. 11 Monoliths Microservices Modular monoliths
12. Shopify 12 engineering.shopify.com/blogs/engineering/deconstructing-monolith-designing-software-maximizes-developer-productivity
13. Shopify 13 Source: speakerdeck.com/ssnickolay/evolution-of-rails-application-architecture-14-years-in-production
14. Hanami 14
15. Elixir 15
16. Rails 16
17. About me Or how I ended up here 17 github.com/palkan
18. evilmartians.com 18
19. evilmartians.com 19 Ruby Next Transpiler for Ruby
20. evl.ms/blog 20
21. evilmartians.com 21
22. What we’ve done Or how we decided to go with engines 22
23. Co-living rental service 23
24. Our mission • Given a property and lease management application • Build a community portal for users within the same Rails app 24
25. Community • Events • Perks • Chat • Billing 25
26. Phase #1: Namespaces 26 app/ controllers/ chat/… models/ chat/… jobs/ chat/…
27. Namespaces • Quick start • Fake isolation 27
28. • Prepare the code to re-use in the future spinoff projects • Given a property and lease management application • Build a community portal for users within the same Rails app Our mission evolves 28
29. Phase #2: Engines & Gems 29 app/… engines/ chat/ app/ controllers/… … lib/… gems/…
30. The Modular Monolith: Rails Architecture 30 medium.com/@dan_manges/the-modular-monolith-rails-architecture-fb1023826fc4
31. Engems Building Rails apps from engines and gems 31 rails plugin new \ my_engine --mountable
32. Crash Course in Engines $ rails plugin new my_engine \ --mountable # or --full create create README.md create Rakefile create Gemfile ... create my_engine.gemspec 32 Engine is a gem
33. Crash Course in Engines 33 my_engine/ app/ controllers/ ... ... config/routes.rb lib/ my_engine/engine.rb my_engine.rb test/ ... Added to autoload paths Isolated tests
34. Crash Course in Engines 34 # my_engine/lib/my_engine/engine.rb module MyEngine class Engine < ::Rails ::Engine # only in mountable engines isolate_namespace MyEngine end end
35. Crash Course in Engines 35 # my_engine/config/routes.rb MyEngine ::Engine.routes.draw do get "/railsconf", to: redirect("/stay_home") end
36. Crash Course in Engines 36 # <root>/Gemfile gem "my_engine", path: "my_engine" # <root>/config/routes.rb Rails.application.routes.draw do mount MyEngine ::Engine => "/my_engine" end
37. Popular Engines 37 actioncable/ actionmailbox/ actiontext/ activestorage/ railtie/
38. 38 gem "devise"
39. The end 39
42. Common Engines 42 connect_by meet_by chat_by manage_by perks_by main app active-storage-proxy common-events
44. Dependencies • How to use non-Rubygems deps in gems? • How to share common deps? • How to sync versions? 44
45. Path/Git problem 45 # engines/my_engine/Gemfile gem "local-lib", path: " ../local-lib" gem "git-lib", github: "palkan/git-lib" # <root>/Gemfile gem "my_engine", path: "engines/my_engine" gem "local-lib", path: " ../local-lib" gem "git-lib", github: "palkan/git-lib"
46. eval_gemfile 46 # engines/my_engine/Gemfile eval_gemfile "./Gemfile.runtime" # engines/my_engine/Gemfile.runtime gem "local-lib", path: " ../local-lib" gem "git-lib", github: "palkan/git-lib" # <root>/Gemfile gem "my_engine", path: "engines/my_engine" eval_gemfile "engines/my_engine/Gemfile.runtime"
47. eval_gemfile 47
48. Shared Gemfiles 48 gemfiles/ profilers.gemfile rails.gemfile gem "stackprof", "0.2.12" gem "ruby-prof", "0.17.0" gem "memory_profiler", "0.9.12" gem "rails", "6.0.0.rc1"
49. Shared Gemfiles 49 # engines/my_engine/Gemfile eval_gemfile " ../gemfiles/rails.runtime" eval_gemfile " ../gemfiles/profilers.runtime" # <root>/Gemfile eval_gemfile "gemfiles/rails.runtime" eval_gemfile "gemfiles/profilers.runtime"
50. Keep Versions N’Sync 50 gem "transdeps" A gem to find inconsistent dependency versions in component-based Ruby apps.
51. transdeps 51 action_policy-graphql (0.3.2) != action_policy-graphql (0.4.0) (chat_by ) faraday (0.17.3) != faraday (1.0.1) (chat_by ) graphql (1.10.4) != graphql (1.10.5) (chat_by ) parser (2.7.0.3) != parser (2.7.0.4) (chat_by ) pg (1.2.2) != pg (1.2.3) (chat_by ) rspec-expectations (3.9.0) != rspec-expectations (3.9.1) (chat_by ) rspec-rails (4.0.0.beta4) != rspec-rails (4.0.0.rc1) (chat_by ) thor (0.20.3) != thor (1.0.1) (chat_by ) twilio-ruby (5.31.1) != twilio-ruby (5.32.0) (chat_by ) faraday (0.17.3) != faraday (1.0.0) (manage_by ) parser (2.7.0.3) != parser (2.7.0.4) (manage_by ) pg (1.2.2) != pg (1.2.3) (manage_by ) rspec-expectations (3.9.0) != rspec-expectations (3.9.1) (manage_by ) rspec-rails (4.0.0.beta4) != rspec-rails (4.0.0.rc1) (manage_by ) thor (0.20.3) != thor (1.0.1) (manage_by ) twilio-ruby (5.31.1) != twilio-ruby (5.32.0) (manage_by ) action_policy-graphql (0.3.2) != action_policy-graphql (0.4.0) (core_by ) faraday (0.17.3) != faraday (1.0.1) (core_by ) graphql (1.10.4) != graphql (1.10.5) (core_by ) parser (2.7.0.3) != parser (2.7.0.4) (core_by ) pg (1.2.2) != pg (1.2.3) (core_by ) rspec-expectations (3.9.0) != rspec-expectations (3.9.1) (core_by ) rspec-rails (4.0.0.beta4) != rspec-rails (4.0.0.rc1) (core_by ) thor (0.20.3) != thor (1.0.1) (core_by ) twilio-ruby (5.31.1) != twilio-ruby (5.32.0) (core_by ) action_policy-graphql (0.3.2) != action_policy-graphql (0.4.0) (meet_by ) faraday (0.17.3) != faraday (1.0.1) (meet_by ) graphql (1.10.4) != graphql (1.10.5) (meet_by ) parser (2.7.0.3) != parser (2.7.0.4) (meet_by ) pg (1.2.2) != pg (1.2.3) (meet_by ) rspec-expectations (3.9.0) != rspec-expectations (3.9.1) (meet_by ) rspec-rails (4.0.0.beta4) != rspec-rails (4.0.0.rc1) (meet_by ) thor (0.20.3) != thor (1.0.1) (meet_by ) twilio-ruby (5.31.1) != twilio-ruby (5.32.0) (meet_by )
52. transdeps: cons • Does not take into account env: it is impossible to ignore development, test deps • Non-configurable: it is impossible to ignore patch level differences 52
53. Syncing Versions 2.0 53 BUNDLE_GEMFILE=" ../ ../Gemfile" bundle install BUNDLE_GEMFILE=" ../ ../Gemfile" bundle exec rspec Use the main app's Gemfile in engines!
54. BUNDLE_GEMFILE: Caveats • How to define engine's dev and test deps? • How to avoid loading everything in dummy apps (in engine tests)? 54
55. component 55 # <root>/Gemfile class Bundler ::Dsl def component(name) group :default, name.to_sym do gem name, path: "engines/ #{name}" eval_gemfile "engines/ #{name}/Gemfile.runtime" end group name.to_sym do dev_deps_from_gemspec("engines/ #{name}.gemspec").each do gem _1.name, _1.requirement end end end end component "connect_by"
56. Bundler.require 56 # <engine>/test/dummy/config/application.rb # ... # Require the gems listed in Gemfile, including any gems # you've limited to :test, :development, or :production. - Bundler.require(*Rails.groups) + Bundler.require(:connect_by)
57. BUNDLE_GEMFILE: Tips 57 bundle config --local gemfile=" ../ ../Gemfile"
59. DB vs. Engines • How to manage migrations? • How to load seeds? 59
60. Migrations. Option #1 Install migrations: rails my_engine:install:migrations 60
61. Migrations. Option #2 “Mount“ migrations: 61 # engines/my_engine/lib/my_engine/engine.rb initializer "my_engine.migrations" do |app| app.config.paths["db/migrate"].concat( config.paths["db/migrate"].expanded ) # For checking pending migrations ActiveRecord ::Migrator.migrations_paths += config.paths["db/migrate"].expanded.flatten end
62. Seeds 62 # <root>/db/seed.rb ActiveRecord ::Base.transaction do ConnectBy ::Engine.load_seed PerksBy ::Engine.load_seed ChatBy ::Engine.load_seed MeetBy ::Engine.load_seed end
63. How to test engines? 63
64. Testing. Option #1 Using a full-featured Dummy app: 64 spec/ dummy/ app/ controllers/ … config/ db/ test/ …
65. Testing. Option #2 65 gem "combustion" A library to help you test your Rails Engines in a simple and effective manner, instead of creating a full Rails application in your spec or test folder.
66. Combustion 66 Combustion.initialize! :active_record, :active_job do config.logger = Logger.new(nil) config.log_level = :fatal config.autoloader = :zeitwerk config.active_storage.service = :test config.active_job.queue_adapter = :test end
67. Combustion 67 spec/ internal/ config/ storage.yml db/ schema.rb
68. Combustion • Load only required Rails frameworks • No boilerplate, only the files you need • Automatically re-runs migrations for every test run 68
69. Engines on CI • Run tests separately for each engine • Run tests only for dirty engines 㱺 decease build times! 69
70. pattern = File.join( __dir __, " ../{engines,gems}/*/*.gemspec") gemspecs = Dir.glob(pattern).map(Gem ::Specification.:load) # Parses git diff $(git merge-base origin/master HEAD) --name-only dirty = fetch_dirty_libraries index = build_inverted_index(gemspecs) lib_name = ARGV[0] if (index[lib_name] & dirty).empty? $stdout.puts "[Dirty Check] No changes for #{lib_name}. Skip" exit(0) else exit(1 end .ci/is-dirty 70
71. CI 71 .ci/is-dirty connect_by || \ (cd engines/connect_by && bundle exec rspec)
73. Base & Behaviour • Base Rails classes within an engine MUST be configurable • They also MAY require to have a certain “interface” 73
74. Base & Behaviour 74 module ConnectBy class ApplicationController < Engine.config.application_controller.constantize raise "Must include ConnectBy ::ControllerBehaviour" unless self < ConnectBy ::ControllerBehaviour raise "Must implement #current_user method" unless instance_methods.include?(:current_user) end end
75. Base & Behaviour 75 module ConnectBy class ApplicationController < Engine.config.application_controller.constantize raise "Must include ConnectBy ::ControllerBehaviour" unless self < ConnectBy ::ControllerBehaviour raise "Must implement #current_user method" unless instance_methods.include?(:current_user) end end Configurable
76. Base & Behaviour 76 # config/application.rb config.connect_by.application_controller = "MyApplicationController"
77. Base & Behaviour 77 module ConnectBy class ApplicationController < Engine.config.application_controller.constantize raise "Must include ConnectBy ::ControllerBehaviour" unless self < ConnectBy ::ControllerBehaviour raise "Must implement #current_user method" unless instance_methods.include?(:current_user) end end “Interface”
78. class ApplicationController < ActionController ::API include ConnectBy ::ControllerBehaviour include JWTAuth include RavenContext if defined?(Raven) end Base & Behaviour 78
79. Modify Models • Prefer extensions over patches • Use load hooks (to support autoload/ reload) 79
80. Extension 80 # engines/meet_by/app/models/ext/connect_by/city.rb module MeetBy module Ext module ConnectBy module City extend ActiveSupport ::Concern included do has_many :events, class_name: "MeetBy ::Events ::Member", inverse_of: :user, foreign_key: :user_id, dependent: :destroy end end end end end
81. Load Hooks 81 # engines/connect_by/app/models/connect_by/city.rb module ConnectBy class City < ActiveRecord ::Base # ... ActiveSupport.run_load_hooks( "connect_by/city", self ) end end
82. Load Hooks 82 # engines/meet_by/lib/meet_by/engine.rb initializer "meet_by.extensions" do ActiveSupport.on_load("connect_by/city") do include ::MeetBy ::Ext ::ConnectBy ::City end end
83. How to communicate between engines? 83
84. 84 Every time new users register in the app we want them to automatically join the default city chat Example
85. Problem 85 connect_by chat_by depends on User Registration ???
86. Solution • Events • Events • Events 86
87. Tools • Hanami Events • Rails Events Store • Wisper (?) • dry-events (?) 87
88. 88 railseventstore.org
89. 89 gem "active_event_store" Wrapper over Rails Event Store with conventions and transparent Rails integration
90. AES vs. RES • Class-independent event types • Uses RES as interchangeable adapter • Better testing tools • Less verbose API and convention over configuration 90
91. Active Event Store 91 class ProfileCompleted < ActiveEventStore ::Event # (optional) event identifier is used for transmitting events # to subscribers. # # By default, identifier is equal to `name.underscore.gsub('/', '.')`. self.identifier = "profile_completed" # Add attributes accessors attributes :user_id # Sync attributes only available for sync subscribers sync_attributes :user end
92. 92 event = ProfileCompleted.new(user_id: user.id) # or with metadata event = ProfileCompleted.new( user_id: user.id, metadata: { ip: request.remote_ip } ) # then publish the event ActiveEventStore.publish(event) Active Event Store
93. 93 initializer "my_engine.subscribe_to_events" do ActiveSupport.on_load :active_event_store do |store| # async subscriber is invoked from background job, # enqueued after the current transaction commits store.subscribe MyEventHandler, to: ProfileCreated # anonymous handler (could only be synchronous) store.subscribe(to: ProfileCreated, sync: true) do |event| # do something end # subscribes to ProfileCreated automatically store.subscribe OnProfileCreated ::DoThat end end Convention over configuration Active Event Store
94. engines/ connect_by/ app/ events/ connect_by/ users/ registered.rb chat_by/ app/ subscribers/ connect_by/ users/ on_registered/ create_chat_account.rb 94 Active Event Store
95. What should we leave in the main app? 95
96. Extreme Case 96 app/ bin/ config/ db/ engines/ gems/ lib/ spec/
97. Common Case 97 app/ controllers/ application_controller.rb sessions_controller.rb bin/ config/ db/ engines/ gems/ lib/ spec/
98. Main App • Feature/system tests • Locales and templates overrides • Instrumentation and exception handling • Configuration 98
99. Gems Or stop putting everything into lib/ext folder 99
100. Gems • Organized lib/ext • Shared non-application specific code • Isolated tests • Ability to share between applications in the future (e.g., GitHub Package Registry) • Ability to open-source in the future 100
101. Gems 101 gems/ common-rubocop/ common-testing/ common-graphql/ …
102. Why engines? And why not 102
103. Why engines? • Modular monolith instead of monolith monsters or distributed monoliths • Code (and tests) isolation • Loose coupling (no more spaghetti logic) 103
104. Why engines? • Easy to upgrade frameworks (e.g., Rails) component by component • Easy to re-use logic in another app • …and we can migrate to microservices in the future (if we’re brave enough) 104
105. Why not engines? • Not so good third-party gems support • Not so good support within Rails itself 105
106. github.com/palkan/engems 106 Rails component-based architecture on top of engines and gems showroom
107. Thank you! Vladimir Dementyev Evil Martians @palkan @palkan_tula evilmartians.com @evilmartians