RailsConf 2018 | The Future of Rails 6: Scalable by Default

RailsConf 2018 | The Future of Rails 6: Scalable by Default

We've all heard the phrase "Rails doesn't scale". Long running test suites and no standard for implementing multiple databases makes it hard scale monolithic Rails applications. Rails 6 will start making Rails scalable by default with parallel testing and improved support for using multiple databases. You'll no longer be forced to reinvent the wheel and create your own solution to these problems. In this talk we'll take a look why these improvements are important, how they work, and ways in which small ideas can quickly snowball into major changes. This is just the beginning of Rails 6.

C44e1f7e22c3f23cff7bc130871047ef?s=128

Eileen M. Uchitelle

April 18, 2018
Tweet

Transcript

  1. The Future of Rails 6 Scalable by Default

  2. Eileen M. Uchitelle eileencodes.com @eileencodes

  3. a

  4. Core Team

  5. “Rails doesn’t scale!”

  6. None
  7. Rails doesn’t scale

  8. Rails doesn’t scale easily

  9. Rails doesn’t scale easily (yet)

  10. What does “scalable” mean?

  11. Your test suite shouldn’t block development

  12. Your application can handle data & traffic

  13. The Future of Rails 6 Scalable by Default

  14. Parallel Testing Speeding up your test suite

  15. a

  16. a

  17. Forking - vs - Threads

  18. Forking Processes with dRB

  19. # test/test_helper.rb module ActiveSupport::TestCase parallelize(workers: 2) end

  20. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]

    return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
  21. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]

    return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
  22. # activesupport/lib/active_support/testing/parallelization module ActiveSupport::Testing::Parallelization […] def initialize(queue_size) @queue_size = queue_size

    @queue = Server.new @pool = [] @url = DRb.start_service("drbunix:", @queue).uri end […] end
  23. # activesupport/lib/active_support/testing/parallelization module ActiveSupport::Testing::Parallelization […] def initialize(queue_size) @queue_size = queue_size

    @queue = Server.new @pool = [] @url = DRb.start_service("drbunix:", @queue).uri end […] end
  24. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]

    return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
  25. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]
  26. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]
  27. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]
  28. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]
  29. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]
  30. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]
  31. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]
  32. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]
  33. module ActiveSupport::Testing::Parallelization […] def start […] end def shutdown @queue_size.times

    { @queue << nil } @pool.each { |pid| Process.waitpid(pid) } end […] end
  34. # test/test_helper.rb module ActiveSupport::TestCase parallelize(workers: 2) end

  35. test-database-0 test-database-1

  36. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]
  37. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]
  38. # test/test_helper.rb module ActiveSupport::TestCase parallelize(workers: 2) parallelize_setup do |worker| ActiveRecord::Tasks::DatabaseTasks.create(:animals)

    ActiveRecord::Base.establish_connection(:animals) ActiveRecord::Tasks::DatabaseTasks.load_schema end parallelize_teardown do |worker| ActiveRecord::Tasks::DatabaseTasks.drop(:animals) end end
  39. Threaded testing with Minitest

  40. # test/test_helper.rb module ActiveSupport::TestCase parallelize(workers: 2, with: :threads) end

  41. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]

    return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
  42. None
  43. Connection ID: #03938752 Connection ID: #83321340

  44. Connection ID: #83321340 Connection ID: #83321340

  45. This is so weird. We seem to have an isolation

    problem…
  46. None
  47. Rails opens a connection

  48. Puma opens a second connection Rails opens a connection

  49. Puma opens a second connection Rails opens a connection

  50. Test thread connection number 1 Test thread connection number 2

  51. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]

    return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
  52. $ rails test

  53. $ PARALLEL_WORKERS=14 rails test

  54. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]

    return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
  55. Parallel Testing PR github.com/rails/rails/pull/31900

  56. Your test suite shouldn’t block development

  57. Your application can handle data & traffic

  58. None
  59. None
  60. DB 1 DB 2 DB 3

  61. Multiple Databases Making multi-db behavior easy

  62. Primary

  63. DB 1 DB 2 DB 3

  64. Table id: integer name: string Table id: integer name: string

    Table id: integer name: string Table id: integer name: string Primary Animals
  65. X Best practices are undocumented The State of Multi-db in

    Rails 5
  66. # database.yml production: <<: *default database: db_production production_animals: <<: *default

    database: db_animals_production
  67. # database.yml production: <<: *default database: db_production production_animals: <<: *default

    database: db_animals_production
  68. # 3-tier database.yml production: primary: <<: *default database: db_production animals:

    <<: *default database: db_animals_production
  69. db/animals_migrate ↳ 20180206210429_create_dogs.rb ↳ 20180206210437_create_cats.rb

  70. ActiveRecord::Migrator.migrations_paths = "db/animals_migrate" ActiveRecord::Base.migrate

  71. X Best practices are undocumented X Migrations don’t work The

    State of Multi-db in Rails 5
  72. # app/models/animals_base.rb class AnimalsBase < ActiveRecord::Base self.abstract_class = true establish_connection

    :animals end
  73. # app/models/dog.rb class Dog < AnimalsBase […] end # app/models/cat.rb

    class Cat < AnimalsBase […] end
  74. X Best practices are undocumented X Migrations don’t work X

    Database tasks don’t exist The State of Multi-db in Rails 5
  75. ActiveRecord::AdapterNotSpecified: 'production' database is not configured.

  76. # activerecord/lib/active_record/railtie.rb initializer "active_record.initialize_database" do ActiveSupport.on_load(:active_record) do self.configurations = Rails.application.config.database_configuration

    begin establish_connection rescue ActiveRecord::NoDatabaseError warn <<-end_warning
  77. # activerecord/lib/active_record/railtie.rb initializer "active_record.initialize_database" do ActiveSupport.on_load(:active_record) do self.configurations = Rails.application.config.database_configuration

    begin establish_connection rescue ActiveRecord::NoDatabaseError warn <<-end_warning
  78. X Best practices are undocumented X Migrations don’t work X

    Database tasks don’t exist X Default connection was broken The State of Multi-db in Rails 5
  79. Migrations Connections don’t work Rake tasks

  80. None
  81. Fixing migrations for multiple databases

  82. path = Rails.root.to_s + "/db/animals_migrate" desc "Migrate the cluster db"

    task migrate: :environment do ActiveRecord::Migrator.migrations_paths = path ActiveRecord::Base.establish_connection(:animals) ActiveRecord::Tasks::DatabaseTasks.migrate db_namespace["db:schema:dump"].invoke end
  83. # 3-tier database.yml production: primary: <<: *default database: db_production animals:

    <<: *default database: db_animals_production migrations_paths: db/animals_migrate
  84. >> Dog.connection.migrations_paths => "db/animals_migrate"

  85. Migrations Paths Refactoring PR github.com/rails/rails/pull/31727

  86. X Best practices are undocumented ✔ Migrations don’t work X

    Database tasks don’t exist X Default connection was broken The State of Multi-db in Rails 5
  87. Migrations Connections don’t work Rake tasks

  88. Improving the database tasks for multi-dbs

  89. rails db:drop rails db:create rails db:migrate

  90. >> ActiveRecord::Base.configurations["production"] >> { "adapter"=>"mysql2", "database"=>"single_database_production" }

  91. >> ActiveRecord::Base.configurations["production"] >> { "primary"=>{"adapter"=>"mysql2", "database"=>"multiple_databases_production"}, "animals"=>{"adapter"=>"mysql2", “database"=>"multiple_databases_production"} }

  92. module ActiveRecord module DatabaseConfigurations class DatabaseConfig attr_reader :env_name, :spec_name, :config

    def initialize(env_name, spec_name, config) @env_name = env_name @spec_name = spec_name @config = config end end end end
  93. module ActiveRecord module DatabaseConfigurations class DatabaseConfig attr_reader :env_name, :spec_name, :config

    def initialize(env_name, spec_name, config) @env_name = env_name @spec_name = spec_name @config = config end end end end
  94. module ActiveRecord module DatabaseConfigurations class DatabaseConfig attr_reader :env_name, :spec_name, :config

    def initialize(env_name, spec_name, config) @env_name = env_name @spec_name = spec_name @config = config end end end end
  95. module ActiveRecord module DatabaseConfigurations class DatabaseConfig attr_reader :env_name, :spec_name, :config

    def initialize(env_name, spec_name, config) @env_name = env_name @spec_name = spec_name @config = config end end end end
  96. >> db_config = DatabaseConfig.new("production", "animals", {"adapter"=>"mysql2", "database"=>"multiple_databases_production_animals" }) >> #<ActiveRecord::DatabaseConfigurations::DatabaseCon

    fig:0x00007f994571edb0 @env_name="production", @spec_name="animals", @config={"adapter"=>"mysql2", "database"=>"multiple_databases_production_animals" }>
  97. >> db_config.env_name "production"

  98. >> db_config.env_name "production" >> db_config.spec_name "animals"

  99. >> db_config.env_name "production" >> db_config.spec_name "animals" >> db_config.config {…"database"=>"multiple_databases_producti on_animals"}

  100. def self.db_configs(configs = configurations) configs.each_pair.flat_map do |env_name, config| walk_configs(env_name, "primary",

    config) end end
  101. def self.configs_for(env, configs, &blk) env_with_configs = db_configs(configs).select do |db_config| db_config.env_name

    == env end if block_given? env_with_configs.each do |ewc| yield ewc.spec_name, ewc.config end else env_with_configs end end
  102. def self.configs_for(env, configs, &blk) env_with_configs = db_configs(configs).select do |db_config| db_config.env_name

    == env end if block_given? env_with_configs.each do |ewc| yield ewc.spec_name, ewc.config end else env_with_configs end end
  103. def self.configs_for(env, configs, &blk) env_with_configs = db_configs(configs).select do |db_config| db_config.env_name

    == env end if block_given? env_with_configs.each do |ewc| yield ewc.spec_name, ewc.config end else env_with_configs end end
  104. >> DatabaseConfigurations.configs_for("production") >>[#<ActiveRecord::DatabaseConfigurations::Database Config:0x00007f994571edb0 @env_name="production", @spec_name="primary", @config={"adapter"=>"mysql2", "database"=>"multiple_databases_production"}>, #<ActiveRecord::DatabaseConfigurations::DatabaseCon fig:0x00007f994571edb0

    @env_name="production", @spec_name="animals", @config={"adapter"=>"mysql2", "database"=>"multiple_databases_production_animals" }>]
  105. task migrate: :load_config do ActiveRecord::Tasks::DatabaseTasks.migrate db_namespace["_dump"].invoke end

  106. task migrate: :load_config do ActiveRecord::DatabaseConfigurations.configs_for( Rails.env ) do |_, config|

    ActiveRecord::Base.establish_connection(config) ActiveRecord::Tasks::DatabaseTasks.migrate end db_namespace["_dump"].invoke end
  107. task migrate: :load_config do ActiveRecord::DatabaseConfigurations.configs_for( Rails.env ) do |_, config|

    ActiveRecord::Base.establish_connection(config) ActiveRecord::Tasks::DatabaseTasks.migrate end db_namespace["_dump"].invoke end
  108. task migrate: :load_config do ActiveRecord::DatabaseConfigurations.configs_for( Rails.env ) do |_, config|

    ActiveRecord::Base.establish_connection(config) ActiveRecord::Tasks::DatabaseTasks.migrate end db_namespace["_dump"].invoke end
  109. task migrate: :load_config do ActiveRecord::DatabaseConfigurations.configs_for( Rails.env ) do |_, config|

    ActiveRecord::Base.establish_connection(config) ActiveRecord::Tasks::DatabaseTasks.migrate end db_namespace["_dump"].invoke end
  110. rails db:drop rails db:create rails db:migrate

  111. rails db:drop:primary rails db:drop:animals rails db:create:primary rails db:create:animals rails db:migrate:primary

    rails db:migrate:animals
  112. namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task

    spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end
  113. namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task

    spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end
  114. namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task

    spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end
  115. namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task

    spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end
  116. namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task

    spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end
  117. rails db:drop:primary rails db:drop:animals rails db:create:primary rails db:create:animals rails db:migrate:primary

    rails db:migrate:animals
  118. Database task improvements github.com/rails/rails/pull/32274 github.com/rails/rails/compare/ ab43b5405ea2...5ddcda6d5f56

  119. X Best practices are undocumented ✔ Migrations don’t work ✔

    Database tasks don’t exist X Default connection was broken The State of Multi-db in Rails 5
  120. Connections don’t work Migrations Rake tasks

  121. None
  122. Thanks @tenderlove!

  123. “Are you concerned that Rails will become bloated?”

  124. What would Rails look like if we’d upstreamed these features

    5 years ago?
  125. a +

  126. ' ( ) * + ,

  127. None
  128. Maybe Rails doesn’t scale…?

  129. Your application isn’t special.

  130. Forces you to write generic code

  131. Train your future workforce

  132. Give back to the community

  133. ' ( ) * + ,

  134. a♥

  135. Scaling should make us happy

  136. Rails isn’t dying, it’s maturing

  137. None
  138. None
  139. None
  140. “Rails does scale!”

  141. The Future of Rails 6 Scalable by Default

  142. Eileen M. Uchitelle eileencodes.com @eileencodes