[RailsConf 2020] Between monoliths and microservices

[RailsConf 2020] Between monoliths and microservices

Video: https://railsconf.com/2020/video/vladimir-dementyev-between-monoliths-and-microservices

Code: https://github.com/palkan/engems

Rails applications tend to turn into giant monoliths. Keeping the codebase maintainable usually requires architectural changes. Microservices? A distributed monolith is still a monolith. What about breaking the code into pieces and rebuild a monolithic puzzle out of them?

In Rails, we have the right tool for the job: engines. With engines, you can break your application into components—the same way as Rails combines all its parts, which are engines, too.

This path is full of snares and pitfalls. Let me be your guide in this journey and share the story of a monolith engine-ification.

52cc8a838bf44a589d2572833b2dd1b9?s=128

Vlad Dem

May 04, 2020
Tweet

Transcript

  1. BETWEEN Vladimir Dementyev Evil Martians Microservices Monoliths

  2. 2 rails new monolith

  3. 3

  4. The Big Four vs Others 4

  5. We are doomed 5

  6. Can you escape your destiny? 6 V.Yurlov, for “The war

    of the worlds” by H.G.Wells
  7. 7

  8. 8 A software system is called “monolithic” if it has

    a monolithic architecture, in which functionally distinguishable aspects are all interwoven, rather than containing architecturally separate components.—Wiki
  9. Components 9

  10. Monolith 10

  11. 11 Monoliths Microservices Modular monoliths

  12. Shopify 12 engineering.shopify.com/blogs/engineering/deconstructing-monolith-designing-software-maximizes-developer-productivity

  13. Shopify 13 Source: speakerdeck.com/ssnickolay/evolution-of-rails-application-architecture-14-years-in-production

  14. Hanami 14

  15. Elixir 15

  16. Rails 16

  17. About me Or how I ended up here 17 github.com/palkan

  18. evilmartians.com 18

  19. evilmartians.com 19 Ruby Next Transpiler for Ruby

  20. evl.ms/blog 20

  21. evilmartians.com 21

  22. What we’ve done Or how we decided to go with

    engines 22
  23. Co-living rental service 23

  24. Our mission • Given a property and lease management application

    • Build a community portal for users within the same Rails app 24
  25. Community • Events • Perks • Chat • Billing 25

  26. Phase #1: Namespaces 26 app/ controllers/ chat/… models/ chat/… jobs/

    chat/…
  27. Namespaces • Quick start • Fake isolation 27

  28. • Prepare the code to re-use in the future spinoff

    projects • Given a property and lease management application • Build a community portal for users within the same Rails app Our mission evolves 28
  29. Phase #2: Engines & Gems 29 app/… engines/ chat/ app/

    controllers/… … lib/… gems/…
  30. The Modular Monolith: Rails Architecture 30 medium.com/@dan_manges/the-modular-monolith-rails-architecture-fb1023826fc4

  31. Engems Building Rails apps from engines and gems 31 rails

    plugin new \ my_engine --mountable
  32. Crash Course in Engines $ rails plugin new my_engine \

    --mountable # or --full create create README.md create Rakefile create Gemfile ... create my_engine.gemspec 32 Engine is a gem
  33. Crash Course in Engines 33 my_engine/ app/ controllers/ ... ...

    config/routes.rb lib/ my_engine/engine.rb my_engine.rb test/ ... Added to autoload paths Isolated tests
  34. Crash Course in Engines 34 # my_engine/lib/my_engine/engine.rb module MyEngine class

    Engine < ::Rails ::Engine # only in mountable engines isolate_namespace MyEngine end end
  35. Crash Course in Engines 35 # my_engine/config/routes.rb MyEngine ::Engine.routes.draw do

    get "/railsconf", to: redirect("/stay_home") end
  36. Crash Course in Engines 36 # <root>/Gemfile gem "my_engine", path:

    "my_engine" # <root>/config/routes.rb Rails.application.routes.draw do mount MyEngine ::Engine => "/my_engine" end
  37. Popular Engines 37 actioncable/ actionmailbox/ actiontext/ activestorage/ railtie/

  38. 38 gem "devise"

  39. The end 39

  40. 40

  41. 41

  42. Common Engines 42 connect_by meet_by chat_by manage_by perks_by main app

    active-storage-proxy common-events
  43. 43

  44. Dependencies • How to use non-Rubygems deps in gems? •

    How to share common deps? • How to sync versions? 44
  45. Path/Git problem 45 # engines/my_engine/Gemfile gem "local-lib", path: " ../local-lib"

    gem "git-lib", github: "palkan/git-lib" # <root>/Gemfile gem "my_engine", path: "engines/my_engine" gem "local-lib", path: " ../local-lib" gem "git-lib", github: "palkan/git-lib"
  46. eval_gemfile 46 # engines/my_engine/Gemfile eval_gemfile "./Gemfile.runtime" # engines/my_engine/Gemfile.runtime gem "local-lib",

    path: " ../local-lib" gem "git-lib", github: "palkan/git-lib" # <root>/Gemfile gem "my_engine", path: "engines/my_engine" eval_gemfile "engines/my_engine/Gemfile.runtime"
  47. eval_gemfile 47

  48. Shared Gemfiles 48 gemfiles/ profilers.gemfile rails.gemfile gem "stackprof", "0.2.12" gem

    "ruby-prof", "0.17.0" gem "memory_profiler", "0.9.12" gem "rails", "6.0.0.rc1"
  49. Shared Gemfiles 49 # engines/my_engine/Gemfile eval_gemfile " ../gemfiles/rails.runtime" eval_gemfile "

    ../gemfiles/profilers.runtime" # <root>/Gemfile eval_gemfile "gemfiles/rails.runtime" eval_gemfile "gemfiles/profilers.runtime"
  50. Keep Versions N’Sync 50 gem "transdeps" A gem to find

    inconsistent dependency versions in component-based Ruby apps.
  51. transdeps 51 action_policy-graphql (0.3.2) != action_policy-graphql (0.4.0) (chat_by ) faraday

    (0.17.3) != faraday (1.0.1) (chat_by ) graphql (1.10.4) != graphql (1.10.5) (chat_by ) parser (2.7.0.3) != parser (2.7.0.4) (chat_by ) pg (1.2.2) != pg (1.2.3) (chat_by ) rspec-expectations (3.9.0) != rspec-expectations (3.9.1) (chat_by ) rspec-rails (4.0.0.beta4) != rspec-rails (4.0.0.rc1) (chat_by ) thor (0.20.3) != thor (1.0.1) (chat_by ) twilio-ruby (5.31.1) != twilio-ruby (5.32.0) (chat_by ) faraday (0.17.3) != faraday (1.0.0) (manage_by ) parser (2.7.0.3) != parser (2.7.0.4) (manage_by ) pg (1.2.2) != pg (1.2.3) (manage_by ) rspec-expectations (3.9.0) != rspec-expectations (3.9.1) (manage_by ) rspec-rails (4.0.0.beta4) != rspec-rails (4.0.0.rc1) (manage_by ) thor (0.20.3) != thor (1.0.1) (manage_by ) twilio-ruby (5.31.1) != twilio-ruby (5.32.0) (manage_by ) action_policy-graphql (0.3.2) != action_policy-graphql (0.4.0) (core_by ) faraday (0.17.3) != faraday (1.0.1) (core_by ) graphql (1.10.4) != graphql (1.10.5) (core_by ) parser (2.7.0.3) != parser (2.7.0.4) (core_by ) pg (1.2.2) != pg (1.2.3) (core_by ) rspec-expectations (3.9.0) != rspec-expectations (3.9.1) (core_by ) rspec-rails (4.0.0.beta4) != rspec-rails (4.0.0.rc1) (core_by ) thor (0.20.3) != thor (1.0.1) (core_by ) twilio-ruby (5.31.1) != twilio-ruby (5.32.0) (core_by ) action_policy-graphql (0.3.2) != action_policy-graphql (0.4.0) (meet_by ) faraday (0.17.3) != faraday (1.0.1) (meet_by ) graphql (1.10.4) != graphql (1.10.5) (meet_by ) parser (2.7.0.3) != parser (2.7.0.4) (meet_by ) pg (1.2.2) != pg (1.2.3) (meet_by ) rspec-expectations (3.9.0) != rspec-expectations (3.9.1) (meet_by ) rspec-rails (4.0.0.beta4) != rspec-rails (4.0.0.rc1) (meet_by ) thor (0.20.3) != thor (1.0.1) (meet_by ) twilio-ruby (5.31.1) != twilio-ruby (5.32.0) (meet_by )
  52. transdeps: cons • Does not take into account env: it

    is impossible to ignore development, test deps • Non-configurable: it is impossible to ignore patch level differences 52
  53. Syncing Versions 2.0 53 BUNDLE_GEMFILE=" ../ ../Gemfile" bundle install BUNDLE_GEMFILE="

    ../ ../Gemfile" bundle exec rspec Use the main app's Gemfile in engines!
  54. BUNDLE_GEMFILE: Caveats • How to define engine's dev and test

    deps? • How to avoid loading everything in dummy apps (in engine tests)? 54
  55. component 55 # <root>/Gemfile class Bundler ::Dsl def component(name) group

    :default, name.to_sym do gem name, path: "engines/ #{name}" eval_gemfile "engines/ #{name}/Gemfile.runtime" end group name.to_sym do dev_deps_from_gemspec("engines/ #{name}.gemspec").each do gem _1.name, _1.requirement end end end end component "connect_by"
  56. Bundler.require 56 # <engine>/test/dummy/config/application.rb # ... # Require the gems

    listed in Gemfile, including any gems # you've limited to :test, :development, or :production. - Bundler.require(*Rails.groups) + Bundler.require(:connect_by)
  57. BUNDLE_GEMFILE: Tips 57 bundle config --local gemfile=" ../ ../Gemfile"

  58. 58

  59. DB vs. Engines • How to manage migrations? • How

    to load seeds? 59
  60. Migrations. Option #1 Install migrations: rails my_engine:install:migrations 60

  61. Migrations. Option #2 “Mount“ migrations: 61 # engines/my_engine/lib/my_engine/engine.rb initializer "my_engine.migrations"

    do |app| app.config.paths["db/migrate"].concat( config.paths["db/migrate"].expanded ) # For checking pending migrations ActiveRecord ::Migrator.migrations_paths += config.paths["db/migrate"].expanded.flatten end
  62. Seeds 62 # <root>/db/seed.rb ActiveRecord ::Base.transaction do ConnectBy ::Engine.load_seed PerksBy

    ::Engine.load_seed ChatBy ::Engine.load_seed MeetBy ::Engine.load_seed end
  63. How to test engines? 63

  64. Testing. Option #1 Using a full-featured Dummy app: 64 spec/

    dummy/ app/ controllers/ … config/ db/ test/ …
  65. Testing. Option #2 65 gem "combustion" A library to help

    you test your Rails Engines in a simple and effective manner, instead of creating a full Rails application in your spec or test folder.
  66. Combustion 66 Combustion.initialize! :active_record, :active_job do config.logger = Logger.new(nil) config.log_level

    = :fatal config.autoloader = :zeitwerk config.active_storage.service = :test config.active_job.queue_adapter = :test end
  67. Combustion 67 spec/ internal/ config/ storage.yml db/ schema.rb

  68. Combustion • Load only required Rails frameworks • No boilerplate,

    only the files you need • Automatically re-runs migrations for every test run 68
  69. Engines on CI • Run tests separately for each engine

    • Run tests only for dirty engines 㱺 decease build times! 69
  70. pattern = File.join( __dir __, " ../{engines,gems}/*/*.gemspec") gemspecs = Dir.glob(pattern).map(Gem

    ::Specification.:load) # Parses git diff $(git merge-base origin/master HEAD) --name-only dirty = fetch_dirty_libraries index = build_inverted_index(gemspecs) lib_name = ARGV[0] if (index[lib_name] & dirty).empty? $stdout.puts "[Dirty Check] No changes for #{lib_name}. Skip" exit(0) else exit(1 end .ci/is-dirty 70
  71. CI 71 .ci/is-dirty connect_by || \ (cd engines/connect_by && bundle

    exec rspec)
  72. 72

  73. Base & Behaviour • Base Rails classes within an engine

    MUST be configurable • They also MAY require to have a certain “interface” 73
  74. Base & Behaviour 74 module ConnectBy class ApplicationController < Engine.config.application_controller.constantize

    raise "Must include ConnectBy ::ControllerBehaviour" unless self < ConnectBy ::ControllerBehaviour raise "Must implement #current_user method" unless instance_methods.include?(:current_user) end end
  75. Base & Behaviour 75 module ConnectBy class ApplicationController < Engine.config.application_controller.constantize

    raise "Must include ConnectBy ::ControllerBehaviour" unless self < ConnectBy ::ControllerBehaviour raise "Must implement #current_user method" unless instance_methods.include?(:current_user) end end Configurable
  76. Base & Behaviour 76 # config/application.rb config.connect_by.application_controller = "MyApplicationController"

  77. Base & Behaviour 77 module ConnectBy class ApplicationController < Engine.config.application_controller.constantize

    raise "Must include ConnectBy ::ControllerBehaviour" unless self < ConnectBy ::ControllerBehaviour raise "Must implement #current_user method" unless instance_methods.include?(:current_user) end end “Interface”
  78. class ApplicationController < ActionController ::API include ConnectBy ::ControllerBehaviour include JWTAuth

    include RavenContext if defined?(Raven) end Base & Behaviour 78
  79. Modify Models • Prefer extensions over patches • Use load

    hooks (to support autoload/ reload) 79
  80. Extension 80 # engines/meet_by/app/models/ext/connect_by/city.rb module MeetBy module Ext module ConnectBy

    module City extend ActiveSupport ::Concern included do has_many :events, class_name: "MeetBy ::Events ::Member", inverse_of: :user, foreign_key: :user_id, dependent: :destroy end end end end end
  81. Load Hooks 81 # engines/connect_by/app/models/connect_by/city.rb module ConnectBy class City <

    ActiveRecord ::Base # ... ActiveSupport.run_load_hooks( "connect_by/city", self ) end end
  82. Load Hooks 82 # engines/meet_by/lib/meet_by/engine.rb initializer "meet_by.extensions" do ActiveSupport.on_load("connect_by/city") do

    include ::MeetBy ::Ext ::ConnectBy ::City end end
  83. How to communicate between engines? 83

  84. 84 Every time new users register in the app we

    want them to automatically join the default city chat Example
  85. Problem 85 connect_by chat_by depends on User Registration ???

  86. Solution • Events • Events • Events 86

  87. Tools • Hanami Events • Rails Events Store • Wisper

    (?) • dry-events (?) 87
  88. 88 railseventstore.org

  89. 89 gem "active_event_store" Wrapper over Rails Event Store with conventions

    and transparent Rails integration
  90. AES vs. RES • Class-independent event types • Uses RES

    as interchangeable adapter • Better testing tools • Less verbose API and convention over configuration 90
  91. Active Event Store 91 class ProfileCompleted < ActiveEventStore ::Event #

    (optional) event identifier is used for transmitting events # to subscribers. # # By default, identifier is equal to `name.underscore.gsub('/', '.')`. self.identifier = "profile_completed" # Add attributes accessors attributes :user_id # Sync attributes only available for sync subscribers sync_attributes :user end
  92. 92 event = ProfileCompleted.new(user_id: user.id) # or with metadata event

    = ProfileCompleted.new( user_id: user.id, metadata: { ip: request.remote_ip } ) # then publish the event ActiveEventStore.publish(event) Active Event Store
  93. 93 initializer "my_engine.subscribe_to_events" do ActiveSupport.on_load :active_event_store do |store| # async

    subscriber is invoked from background job, # enqueued after the current transaction commits store.subscribe MyEventHandler, to: ProfileCreated # anonymous handler (could only be synchronous) store.subscribe(to: ProfileCreated, sync: true) do |event| # do something end # subscribes to ProfileCreated automatically store.subscribe OnProfileCreated ::DoThat end end Convention over configuration Active Event Store
  94. engines/ connect_by/ app/ events/ connect_by/ users/ registered.rb chat_by/ app/ subscribers/

    connect_by/ users/ on_registered/ create_chat_account.rb 94 Active Event Store
  95. What should we leave in the main app? 95

  96. Extreme Case 96 app/ bin/ config/ db/ engines/ gems/ lib/

    spec/
  97. Common Case 97 app/ controllers/ application_controller.rb sessions_controller.rb bin/ config/ db/

    engines/ gems/ lib/ spec/
  98. Main App • Feature/system tests • Locales and templates overrides

    • Instrumentation and exception handling • Configuration 98
  99. Gems Or stop putting everything into lib/ext folder 99

  100. Gems • Organized lib/ext • Shared non-application specific code •

    Isolated tests • Ability to share between applications in the future (e.g., GitHub Package Registry) • Ability to open-source in the future 100
  101. Gems 101 gems/ common-rubocop/ common-testing/ common-graphql/ …

  102. Why engines? And why not 102

  103. Why engines? • Modular monolith instead of monolith monsters or

    distributed monoliths • Code (and tests) isolation • Loose coupling (no more spaghetti logic) 103
  104. Why engines? • Easy to upgrade frameworks (e.g., Rails) component

    by component • Easy to re-use logic in another app • …and we can migrate to microservices in the future (if we’re brave enough) 104
  105. Why not engines? • Not so good third-party gems support

    • Not so good support within Rails itself 105
  106. github.com/palkan/engems 106 Rails component-based architecture on top of engines and

    gems showroom
  107. Thank you! Vladimir Dementyev Evil Martians @palkan @palkan_tula evilmartians.com @evilmartians