[RailsConf 2020] Between monoliths and microservices

Video: https://railsconf.com/2020/video/vladimir-dementyev-between-monoliths-and-microservices

Code: https://github.com/palkan/engems

Rails applications tend to turn into giant monoliths. Keeping the codebase maintainable usually requires architectural changes. Microservices? A distributed monolith is still a monolith. What about breaking the code into pieces and rebuild a monolithic puzzle out of them?

In Rails, we have the right tool for the job: engines. With engines, you can break your application into components—the same way as Rails combines all its parts, which are engines, too.

This path is full of snares and pitfalls. Let me be your guide in this journey and share the story of a monolith engine-ification.

Transcript

1. BETWEEN
   Vladimir Dementyev
   Evil Martians
   Microservices
   Monoliths
   

   View Slide

2. 2
   rails new monolith
   

   View Slide

3. View Slide

4. The Big Four
   vs
   Others
   4
   

   View Slide

5. We are
   doomed
   5
   

   View Slide

6. Can you escape
   your destiny?
   6
   V.Yurlov, for “The war of the worlds” by H.G.Wells
   

   View Slide

7. View Slide

8. 8
   A software system is called
   “monolithic” if it has a monolithic
   architecture, in which functionally
   distinguishable aspects are all
   interwoven, rather than containing
   architecturally separate
   components.—Wiki
   

   View Slide

9. Components
   9
   
   

   View Slide

10. Monolith
    10
    
    

    View Slide

11. 11
    Monoliths Microservices
    Modular monoliths
    
    

    View Slide

12. Shopify
    12
    engineering.shopify.com/blogs/engineering/deconstructing-monolith-designing-software-maximizes-developer-productivity
    

    View Slide

13. Shopify
    13
    Source: speakerdeck.com/ssnickolay/evolution-of-rails-application-architecture-14-years-in-production
    

    View Slide

14. Hanami
    14
    

    View Slide

15. Elixir
    15
    

    View Slide

16. Rails
    16
    

    View Slide

17. About me
    Or how I ended up here
    17
    github.com/palkan
    

    View Slide

18. evilmartians.com
    18
    

    View Slide

19. evilmartians.com
    19
    Ruby Next
    Transpiler for
    Ruby
    

    View Slide

20. evl.ms/blog
    20
    

    View Slide

21. evilmartians.com
    21
    

    View Slide

22. What we’ve
    done
    Or how we decided to
    go with engines
    22
    

    View Slide

23. Co-living rental service
    23
    

    View Slide

24. Our mission
    • Given a property and lease
    management application
    • Build a community portal for users
    within the same Rails app
    24
    

    View Slide

25. Community
    • Events
    • Perks
    • Chat
    • Billing
    25
    

    View Slide

26. Phase #1: Namespaces
    26
    app/
    controllers/
    chat/…
    models/
    chat/…
    jobs/
    chat/…
    

    View Slide

27. Namespaces
    • Quick start
    • Fake isolation
    27
    

    View Slide

28. • Prepare the code to re-use in the
    future spinoff projects
    • Given a property and lease
    management application
    • Build a community portal for users
    within the same Rails app
    Our mission evolves
    28
    

    View Slide

29. Phase #2: Engines & Gems
    29
    app/…
    engines/
    chat/
    app/
    controllers/…
    …
    lib/…
    gems/…
    

    View Slide

30. The Modular Monolith:
    Rails Architecture
    30
    medium.com/@dan_manges/the-modular-monolith-rails-architecture-fb1023826fc4
    

    View Slide

31. Engems
    Building Rails apps from engines
    and gems
    31
    rails plugin new \
    my_engine --mountable
    

    View Slide

32. Crash Course in Engines
    $ rails plugin new my_engine \
    --mountable # or --full
    create
    create README.md
    create Rakefile
    create Gemfile
    ...
    create my_engine.gemspec
    32
    Engine is a gem
    

    View Slide

33. Crash Course in Engines
    33
    my_engine/
    app/
    controllers/ ...
    ...
    config/routes.rb
    lib/
    my_engine/engine.rb
    my_engine.rb
    test/ ...
    Added to autoload paths
    Isolated tests
    

    View Slide

34. Crash Course in Engines
    34
    # my_engine/lib/my_engine/engine.rb
    module MyEngine
    class Engine < ::Rails ::Engine
    # only in mountable engines
    isolate_namespace MyEngine
    end
    end
    

    View Slide

35. Crash Course in Engines
    35
    # my_engine/config/routes.rb
    MyEngine ::Engine.routes.draw do
    get "/railsconf",
    to: redirect("/stay_home")
    end
    

    View Slide

36. Crash Course in Engines
    36
    # /Gemfile
    gem "my_engine", path: "my_engine"
    # /config/routes.rb
    Rails.application.routes.draw do
    mount MyEngine ::Engine => "/my_engine"
    end
    

    View Slide

37. Popular Engines
    37
    actioncable/
    actionmailbox/
    actiontext/
    activestorage/
    railtie/
    

    View Slide

38. 38
    gem "devise"
    

    View Slide

39. The end
    39
    

    View Slide

40. View Slide

41. View Slide

42. Common Engines
    42
    connect_by
    meet_by
    chat_by
    manage_by
    perks_by
    main app
    active-storage-proxy
    common-events
    

    View Slide

43. View Slide

44. Dependencies
    • How to use non-Rubygems deps in
    gems?
    • How to share common deps?
    • How to sync versions?
    44
    

    View Slide

45. Path/Git problem
    45
    # engines/my_engine/Gemfile
    gem "local-lib", path: " ../local-lib"
    gem "git-lib", github: "palkan/git-lib"
    # /Gemfile
    gem "my_engine", path: "engines/my_engine"
    gem "local-lib", path: " ../local-lib"
    gem "git-lib", github: "palkan/git-lib"
    

    View Slide

46. eval_gemfile
    46
    # engines/my_engine/Gemfile
    eval_gemfile "./Gemfile.runtime"
    # engines/my_engine/Gemfile.runtime
    gem "local-lib", path: " ../local-lib"
    gem "git-lib", github: "palkan/git-lib"
    # /Gemfile
    gem "my_engine", path: "engines/my_engine"
    eval_gemfile "engines/my_engine/Gemfile.runtime"
    

    View Slide

47. eval_gemfile
    47
    

    View Slide

48. Shared Gemfiles
    48
    gemfiles/
    profilers.gemfile
    rails.gemfile
    gem "stackprof", "0.2.12"
    gem "ruby-prof", "0.17.0"
    gem "memory_profiler", "0.9.12"
    gem "rails", "6.0.0.rc1"
    

    View Slide

49. Shared Gemfiles
    49
    # engines/my_engine/Gemfile
    eval_gemfile " ../gemfiles/rails.runtime"
    eval_gemfile " ../gemfiles/profilers.runtime"
    # /Gemfile
    eval_gemfile "gemfiles/rails.runtime"
    eval_gemfile "gemfiles/profilers.runtime"
    

    View Slide

50. Keep Versions N’Sync
    50
    gem "transdeps"
    A gem to find inconsistent dependency
    versions in component-based Ruby apps.
    

    View Slide

51. transdeps
    51
    action_policy-graphql (0.3.2) != action_policy-graphql (0.4.0) (chat_by )
    faraday (0.17.3) != faraday (1.0.1) (chat_by )
    graphql (1.10.4) != graphql (1.10.5) (chat_by )
    parser (2.7.0.3) != parser (2.7.0.4) (chat_by )
    pg (1.2.2) != pg (1.2.3) (chat_by )
    rspec-expectations (3.9.0) != rspec-expectations (3.9.1) (chat_by )
    rspec-rails (4.0.0.beta4) != rspec-rails (4.0.0.rc1) (chat_by )
    thor (0.20.3) != thor (1.0.1) (chat_by )
    twilio-ruby (5.31.1) != twilio-ruby (5.32.0) (chat_by )
    faraday (0.17.3) != faraday (1.0.0) (manage_by )
    parser (2.7.0.3) != parser (2.7.0.4) (manage_by )
    pg (1.2.2) != pg (1.2.3) (manage_by )
    rspec-expectations (3.9.0) != rspec-expectations (3.9.1) (manage_by )
    rspec-rails (4.0.0.beta4) != rspec-rails (4.0.0.rc1) (manage_by )
    thor (0.20.3) != thor (1.0.1) (manage_by )
    twilio-ruby (5.31.1) != twilio-ruby (5.32.0) (manage_by )
    action_policy-graphql (0.3.2) != action_policy-graphql (0.4.0) (core_by )
    faraday (0.17.3) != faraday (1.0.1) (core_by )
    graphql (1.10.4) != graphql (1.10.5) (core_by )
    parser (2.7.0.3) != parser (2.7.0.4) (core_by )
    pg (1.2.2) != pg (1.2.3) (core_by )
    rspec-expectations (3.9.0) != rspec-expectations (3.9.1) (core_by )
    rspec-rails (4.0.0.beta4) != rspec-rails (4.0.0.rc1) (core_by )
    thor (0.20.3) != thor (1.0.1) (core_by )
    twilio-ruby (5.31.1) != twilio-ruby (5.32.0) (core_by )
    action_policy-graphql (0.3.2) != action_policy-graphql (0.4.0) (meet_by )
    faraday (0.17.3) != faraday (1.0.1) (meet_by )
    graphql (1.10.4) != graphql (1.10.5) (meet_by )
    parser (2.7.0.3) != parser (2.7.0.4) (meet_by )
    pg (1.2.2) != pg (1.2.3) (meet_by )
    rspec-expectations (3.9.0) != rspec-expectations (3.9.1) (meet_by )
    rspec-rails (4.0.0.beta4) != rspec-rails (4.0.0.rc1) (meet_by )
    thor (0.20.3) != thor (1.0.1) (meet_by )
    twilio-ruby (5.31.1) != twilio-ruby (5.32.0) (meet_by )
    
    

    View Slide

52. transdeps: cons
    • Does not take into account env: it is
    impossible to ignore development, test
    deps
    • Non-configurable: it is impossible to
    ignore patch level differences
    52
    

    View Slide

53. Syncing Versions 2.0
    53
    BUNDLE_GEMFILE=" ../ ../Gemfile" bundle install
    BUNDLE_GEMFILE=" ../ ../Gemfile" bundle exec rspec
    Use the main app's Gemfile in engines!
    

    View Slide

54. BUNDLE_GEMFILE: Caveats
    • How to define engine's dev and test
    deps?
    • How to avoid loading everything in
    dummy apps (in engine tests)?
    54
    

    View Slide

55. component
    55
    # /Gemfile
    class Bundler ::Dsl
    def component(name)
    group :default, name.to_sym do
    gem name, path: "engines/ #{name}"
    eval_gemfile "engines/ #{name}/Gemfile.runtime"
    end
    group name.to_sym do
    dev_deps_from_gemspec("engines/ #{name}.gemspec").each do
    gem _1.name, _1.requirement
    end
    end
    end
    end
    component "connect_by"
    

    View Slide

56. Bundler.require
    56
    # /test/dummy/config/application.rb
    # ...
    # Require the gems listed in Gemfile, including any gems
    # you've limited to :test, :development, or :production.
    - Bundler.require(*Rails.groups)
    + Bundler.require(:connect_by)
    

    View Slide

57. BUNDLE_GEMFILE: Tips
    57
    bundle config --local gemfile=" ../ ../Gemfile"
    

    View Slide

58. View Slide

59. DB vs. Engines
    • How to manage migrations?
    • How to load seeds?
    59
    

    View Slide

60. Migrations. Option #1
    Install migrations:
    rails my_engine:install:migrations
    60
    
    

    View Slide

61. Migrations. Option #2
    “Mount“ migrations:
    61
    # engines/my_engine/lib/my_engine/engine.rb
    initializer "my_engine.migrations" do |app|
    app.config.paths["db/migrate"].concat(
    config.paths["db/migrate"].expanded
    )
    # For checking pending migrations
    ActiveRecord ::Migrator.migrations_paths +=
    config.paths["db/migrate"].expanded.flatten
    end
    
    

    View Slide

62. Seeds
    62
    # /db/seed.rb
    ActiveRecord ::Base.transaction do
    ConnectBy ::Engine.load_seed
    PerksBy ::Engine.load_seed
    ChatBy ::Engine.load_seed
    MeetBy ::Engine.load_seed
    end
    

    View Slide

63. How to test
    engines?
    63
    

    View Slide

64. Testing. Option #1
    Using a full-featured Dummy app:
    64
    
    spec/
    dummy/
    app/
    controllers/
    …
    config/
    db/
    test/
    …
    

    View Slide

65. Testing. Option #2
    65
    
    gem "combustion"
    A library to help you test your Rails Engines in
    a simple and effective manner, instead of
    creating a full Rails application in your spec or
    test folder.
    

    View Slide

66. Combustion
    66
    Combustion.initialize! :active_record, :active_job do
    config.logger = Logger.new(nil)
    config.log_level = :fatal
    config.autoloader = :zeitwerk
    config.active_storage.service = :test
    config.active_job.queue_adapter = :test
    end
    

    View Slide

67. Combustion
    67
    spec/
    internal/
    config/
    storage.yml
    db/
    schema.rb
    

    View Slide

68. Combustion
    • Load only required Rails frameworks
    • No boilerplate, only the files you need
    • Automatically re-runs migrations for
    every test run
    68
    

    View Slide

69. Engines on CI
    • Run tests separately for each engine
    • Run tests only for dirty engines 㱺
    decease build times!
    69
    

    View Slide

70. pattern = File.join( __dir __, " ../{engines,gems}/*/*.gemspec")
    gemspecs = Dir.glob(pattern).map(Gem ::Specification.:load)
    # Parses git diff $(git merge-base origin/master HEAD) --name-only
    dirty = fetch_dirty_libraries
    index = build_inverted_index(gemspecs)
    lib_name = ARGV[0]
    if (index[lib_name] & dirty).empty?
    $stdout.puts "[Dirty Check] No changes for #{lib_name}. Skip"
    exit(0)
    else
    exit(1
    end
    .ci/is-dirty
    70
    

    View Slide

71. CI
    71
    .ci/is-dirty connect_by || \
    (cd engines/connect_by && bundle exec rspec)
    

    View Slide

72. View Slide

73. Base & Behaviour
    • Base Rails classes within an engine
    MUST be configurable
    • They also MAY require to have a certain
    “interface”
    73
    

    View Slide

74. Base & Behaviour
    74
    module ConnectBy
    class ApplicationController < Engine.config.application_controller.constantize
    raise "Must include ConnectBy ::ControllerBehaviour" unless
    self < ConnectBy ::ControllerBehaviour
    raise "Must implement #current_user method" unless
    instance_methods.include?(:current_user)
    end
    end
    

    View Slide

75. Base & Behaviour
    75
    module ConnectBy
    class ApplicationController < Engine.config.application_controller.constantize
    raise "Must include ConnectBy ::ControllerBehaviour" unless
    self < ConnectBy ::ControllerBehaviour
    raise "Must implement #current_user method" unless
    instance_methods.include?(:current_user)
    end
    end
    Configurable
    

    View Slide

76. Base & Behaviour
    76
    # config/application.rb
    config.connect_by.application_controller =
    "MyApplicationController"
    

    View Slide

77. Base & Behaviour
    77
    module ConnectBy
    class ApplicationController < Engine.config.application_controller.constantize
    raise "Must include ConnectBy ::ControllerBehaviour" unless
    self < ConnectBy ::ControllerBehaviour
    raise "Must implement #current_user method" unless
    instance_methods.include?(:current_user)
    end
    end
    “Interface”
    

    View Slide

78. class ApplicationController < ActionController ::API
    include ConnectBy ::ControllerBehaviour
    include JWTAuth
    include RavenContext if defined?(Raven)
    end
    Base & Behaviour
    78
    

    View Slide

79. Modify Models
    • Prefer extensions over patches
    • Use load hooks (to support autoload/
    reload)
    79
    

    View Slide

80. Extension
    80
    # engines/meet_by/app/models/ext/connect_by/city.rb
    module MeetBy
    module Ext
    module ConnectBy
    module City
    extend ActiveSupport ::Concern
    included do
    has_many :events, class_name: "MeetBy ::Events ::Member",
    inverse_of: :user,
    foreign_key: :user_id,
    dependent: :destroy
    end
    end
    end
    end
    end
    

    View Slide

81. Load Hooks
    81
    # engines/connect_by/app/models/connect_by/city.rb
    module ConnectBy
    class City < ActiveRecord ::Base
    # ...
    ActiveSupport.run_load_hooks(
    "connect_by/city",
    self
    )
    end
    end
    

    View Slide

82. Load Hooks
    82
    # engines/meet_by/lib/meet_by/engine.rb
    initializer "meet_by.extensions" do
    ActiveSupport.on_load("connect_by/city") do
    include ::MeetBy ::Ext ::ConnectBy ::City
    end
    end
    

    View Slide

83. How to communicate
    between engines?
    83
    

    View Slide

84. 84
    Every time new users register in the app
    we want them to automatically join the
    default city chat
    Example
    

    View Slide

85. Problem
    85
    connect_by
    chat_by
    depends on
    User Registration
    ???
    

    View Slide

86. Solution
    • Events
    • Events
    • Events
    86
    

    View Slide

87. Tools
    • Hanami Events
    • Rails Events Store
    • Wisper (?)
    • dry-events (?)
    87
    

    View Slide

88. 88
    railseventstore.org
    

    View Slide

89. 89
    gem "active_event_store"
    Wrapper over Rails Event Store with
    conventions and transparent Rails integration
    

    View Slide

90. AES vs. RES
    • Class-independent event types
    • Uses RES as interchangeable adapter
    • Better testing tools
    • Less verbose API and convention over
    configuration
    90
    

    View Slide

91. Active Event Store
    91
    class ProfileCompleted < ActiveEventStore ::Event
    # (optional) event identifier is used for transmitting events
    # to subscribers.
    #
    # By default, identifier is equal to `name.underscore.gsub('/', '.')`.
    self.identifier = "profile_completed"
    # Add attributes accessors
    attributes :user_id
    # Sync attributes only available for sync subscribers
    sync_attributes :user
    end
    

    View Slide

92. 92
    event = ProfileCompleted.new(user_id: user.id)
    # or with metadata
    event = ProfileCompleted.new(
    user_id: user.id,
    metadata: { ip: request.remote_ip }
    )
    # then publish the event
    ActiveEventStore.publish(event)
    Active Event Store
    

    View Slide

93. 93
    initializer "my_engine.subscribe_to_events" do
    ActiveSupport.on_load :active_event_store do |store|
    # async subscriber is invoked from background job,
    # enqueued after the current transaction commits
    store.subscribe MyEventHandler, to: ProfileCreated
    # anonymous handler (could only be synchronous)
    store.subscribe(to: ProfileCreated, sync: true) do |event|
    # do something
    end
    # subscribes to ProfileCreated automatically
    store.subscribe OnProfileCreated ::DoThat
    end
    end Convention over
    configuration
    Active Event Store
    

    View Slide

94. engines/
    connect_by/
    app/
    events/
    connect_by/
    users/
    registered.rb
    chat_by/
    app/
    subscribers/
    connect_by/
    users/
    on_registered/
    create_chat_account.rb
    94
    Active Event Store
    

    View Slide

95. What should we leave
    in the main app?
    95
    

    View Slide

96. Extreme Case
    96
    app/
    bin/
    config/
    db/
    engines/
    gems/
    lib/
    spec/
    
    

    View Slide

97. Common Case
    97
    app/
    controllers/
    application_controller.rb
    sessions_controller.rb
    bin/
    config/
    db/
    engines/
    gems/
    lib/
    spec/
    

    View Slide

98. Main App
    • Feature/system tests
    • Locales and templates overrides
    • Instrumentation and exception
    handling
    • Configuration
    98
    

    View Slide

99. Gems
    Or stop putting everything into
    lib/ext folder
    99
    

    View Slide

100. Gems
     • Organized lib/ext
     • Shared non-application specific code
     • Isolated tests
     • Ability to share between applications in the future
     (e.g., GitHub Package Registry)
     • Ability to open-source in the future
     100
     

     View Slide

101. Gems
     101
     gems/
     common-rubocop/
     common-testing/
     common-graphql/
     …
     

     View Slide

102. Why
     engines?
     And why not
     102
     

     View Slide

103. Why engines?
     • Modular monolith instead of monolith
     monsters or distributed monoliths
     • Code (and tests) isolation
     • Loose coupling (no more spaghetti
     logic)
     103
     

     View Slide

104. Why engines?
     • Easy to upgrade frameworks (e.g.,
     Rails) component by component
     • Easy to re-use logic in another app
     • …and we can migrate to microservices
     in the future (if we’re brave enough)
     104
     

     View Slide

105. Why not engines?
     • Not so good third-party gems support
     • Not so good support within Rails itself
     105
     

     View Slide

106. github.com/palkan/engems
     106
     Rails component-based architecture on top of
     engines and gems showroom
     

     View Slide

107. Thank you!
     Vladimir Dementyev
     Evil Martians
     @palkan
     @palkan_tula
     evilmartians.com
     @evilmartians
     

     View Slide