Paris Ruby Conference 2018: The Future of Rails 6

Paris Ruby Conference 2018: The Future of Rails 6

We've all heard the phrase "Rails doesn't scale". Long running test suites and no standard for implementing multiple databases makes it hard scale monolithic Rails applications. Rails 6 will start making Rails scalable by default with parallel testing and improved support for using multiple databases. You'll no longer be forced to reinvent the wheel and create your own solution to these problems. In this talk we'll take a look why these improvements are important, how they work, and ways in which small ideas can quickly snowball into major changes. This is just the beginning of Rails 6.

C44e1f7e22c3f23cff7bc130871047ef?s=128

Eileen M. Uchitelle

June 28, 2018
Tweet

Transcript

  1. The Future of Rails 6 Scalable by Default

  2. Eileen M. Uchitelle eileencodes.com @eileencodes

  3. a

  4. Core Team

  5. “Rails doesn’t scale!”

  6. None
  7. Rails doesn’t scale

  8. Rails doesn’t scale easily

  9. Rails doesn’t scale easily (yet)

  10. What does “scalable” mean?

  11. Your test suite shouldn’t block development

  12. Your application can handle data & traffic

  13. The Future of Rails 6 Scalable by Default

  14. Parallel Testing Speeding up your test suite

  15. a

  16. a

  17. Forking - vs - Threads

  18. Forking Processes with dRB

  19. # test/test_helper.rb module ActiveSupport::TestCase parallelize(workers: 2) end

  20. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]

    return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
  21. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]

    return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
  22. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]

    return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
  23. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]
  24. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]
  25. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]
  26. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]
  27. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]
  28. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]
  29. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]
  30. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]
  31. module ActiveSupport::Testing::Parallelization […] def start […] end def shutdown @queue_size.times

    { @queue << nil } @pool.each { |pid| Process.waitpid(pid) } end […] end
  32. # test/test_helper.rb module ActiveSupport::TestCase parallelize(workers: 2) end

  33. test-database-0 test-database-1

  34. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]
  35. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]
  36. Threaded testing with Minitest

  37. # test/test_helper.rb module ActiveSupport::TestCase parallelize(workers: 2, with: :threads) end

  38. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]

    return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
  39. None
  40. Connection ID: #03938752 Connection ID: #83321340

  41. Connection ID: #83321340 Connection ID: #83321340

  42. This is so weird. We seem to have an isolation

    problem…
  43. None
  44. Rails opens a connection

  45. Puma opens a second connection Rails opens a connection

  46. Puma opens a second connection Rails opens a connection

  47. Test thread connection number 1 Test thread connection number 2

  48. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]

    return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
  49. $ rails test

  50. Parallel Testing PR github.com/rails/rails/pull/31900

  51. Your test suite shouldn’t block development

  52. Your application can handle data & traffic

  53. None
  54. None
  55. DB 1 DB 2 DB 3

  56. Multiple Databases Making multi-db behavior easy

  57. Table id: integer name: string Table id: integer name: string

    Table id: integer name: string Table id: integer name: string Primary Animals
  58. X Best practices are undocumented The State of Multi-db in

    Rails 5
  59. # 3-tier database.yml production: primary: <<: *default database: db_production animals:

    <<: *default database: db_animals_production
  60. db/animals_migrate ↳ 20180206210429_create_dogs.rb ↳ 20180206210437_create_cats.rb

  61. ActiveRecord::Migrator.migrations_paths = "db/animals_migrate" ActiveRecord::Base.migrate

  62. X Best practices are undocumented X Migrations don’t work The

    State of Multi-db in Rails 5
  63. # app/models/animals_base.rb class AnimalsBase < ActiveRecord::Base self.abstract_class = true establish_connection

    :animals end
  64. # app/models/dog.rb class Dog < AnimalsBase […] end # app/models/cat.rb

    class Cat < AnimalsBase […] end
  65. X Best practices are undocumented X Migrations don’t work X

    Database tasks don’t exist The State of Multi-db in Rails 5
  66. ActiveRecord::AdapterNotSpecified: 'production' database is not configured.

  67. # activerecord/lib/active_record/railtie.rb initializer "active_record.initialize_database" do ActiveSupport.on_load(:active_record) do self.configurations = Rails.application.config.database_configuration

    begin establish_connection rescue ActiveRecord::NoDatabaseError warn <<-end_warning
  68. # activerecord/lib/active_record/railtie.rb initializer "active_record.initialize_database" do ActiveSupport.on_load(:active_record) do self.configurations = Rails.application.config.database_configuration

    begin establish_connection rescue ActiveRecord::NoDatabaseError warn <<-end_warning
  69. X Best practices are undocumented X Migrations don’t work X

    Database tasks don’t exist X Default connection was broken The State of Multi-db in Rails 5
  70. Migrations Connections don’t work Rake tasks

  71. None
  72. Fixing migrations for multiple databases

  73. path = Rails.root.to_s + "/db/animals_migrate" desc "Migrate the cluster db"

    task migrate: :environment do ActiveRecord::Migrator.migrations_paths = path ActiveRecord::Base.establish_connection(:animals) ActiveRecord::Tasks::DatabaseTasks.migrate db_namespace["db:schema:dump"].invoke end
  74. # 3-tier database.yml production: primary: <<: *default database: db_production animals:

    <<: *default database: db_animals_production migrations_paths: db/animals_migrate
  75. >> Dog.connection.migrations_paths => "db/animals_migrate"

  76. Migrations Paths Refactoring PR github.com/rails/rails/pull/31727

  77. X Best practices are undocumented ✔ Migrations don’t work X

    Database tasks don’t exist X Default connection was broken The State of Multi-db in Rails 5
  78. Migrations Connections don’t work Rake tasks

  79. Improving the database tasks for multi-dbs

  80. rails db:drop rails db:create rails db:migrate

  81. >> ActiveRecord::Base.configurations["production"] >> { "adapter"=>"mysql2", "database"=>"single_database_production" }

  82. >> ActiveRecord::Base.configurations["production"] >> { "primary"=>{"adapter"=>"mysql2", "database"=>"multiple_databases_production"}, "animals"=>{"adapter"=>"mysql2", “database"=>"multiple_databases_production"} }

  83. module ActiveRecord module DatabaseConfigurations class DatabaseConfig attr_reader :env_name, :spec_name, :config

    def initialize(env_name, spec_name, config) @env_name = env_name @spec_name = spec_name @config = config end end end end
  84. >> db_config = DatabaseConfig.new("production", "animals", {"adapter"=>"mysql2", "database"=>"multiple_databases_production_animals" }) >> #<ActiveRecord::DatabaseConfigurations::DatabaseCon

    fig:0x00007f994571edb0 @env_name="production", @spec_name="animals", @config={"adapter"=>"mysql2", "database"=>"multiple_databases_production_animals" }>
  85. >> db_config.env_name "production"

  86. >> db_config.env_name "production" >> db_config.spec_name "animals"

  87. >> db_config.env_name "production" >> db_config.spec_name "animals" >> db_config.config {…"database"=>"multiple_databases_producti on_animals"}

  88. def self.db_configs(configs = configurations) configs.each_pair.flat_map do |env_name, config| walk_configs(env_name, "primary",

    config) end end
  89. def self.configs_for(env, configs, &blk) env_with_configs = db_configs(configs).select do |db_config| db_config.env_name

    == env end if block_given? env_with_configs.each do |ewc| yield ewc.spec_name, ewc.config end else env_with_configs end end
  90. >> DatabaseConfigurations.configs_for("production") >>[#<ActiveRecord::DatabaseConfigurations::Database Config:0x00007f994571edb0 @env_name="production", @spec_name="primary", @config={"adapter"=>"mysql2", "database"=>"multiple_databases_production"}>, #<ActiveRecord::DatabaseConfigurations::DatabaseCon fig:0x00007f994571edb0

    @env_name="production", @spec_name="animals", @config={"adapter"=>"mysql2", "database"=>"multiple_databases_production_animals" }>]
  91. task migrate: :load_config do ActiveRecord::Tasks::DatabaseTasks.migrate db_namespace["_dump"].invoke end

  92. task migrate: :load_config do ActiveRecord::DatabaseConfigurations.configs_for( Rails.env ) do |_, config|

    ActiveRecord::Base.establish_connection(config) ActiveRecord::Tasks::DatabaseTasks.migrate end db_namespace["_dump"].invoke end
  93. task migrate: :load_config do ActiveRecord::DatabaseConfigurations.configs_for( Rails.env ) do |_, config|

    ActiveRecord::Base.establish_connection(config) ActiveRecord::Tasks::DatabaseTasks.migrate end db_namespace["_dump"].invoke end
  94. task migrate: :load_config do ActiveRecord::DatabaseConfigurations.configs_for( Rails.env ) do |_, config|

    ActiveRecord::Base.establish_connection(config) ActiveRecord::Tasks::DatabaseTasks.migrate end db_namespace["_dump"].invoke end
  95. task migrate: :load_config do ActiveRecord::DatabaseConfigurations.configs_for( Rails.env ) do |_, config|

    ActiveRecord::Base.establish_connection(config) ActiveRecord::Tasks::DatabaseTasks.migrate end db_namespace["_dump"].invoke end
  96. rails db:drop rails db:create rails db:migrate

  97. rails db:drop:primary rails db:drop:animals rails db:create:primary rails db:create:animals rails db:migrate:primary

    rails db:migrate:animals
  98. namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task

    spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end
  99. namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task

    spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end
  100. namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task

    spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end
  101. namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task

    spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end
  102. namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task

    spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end
  103. rails db:drop:primary rails db:drop:animals rails db:create:primary rails db:create:animals rails db:migrate:primary

    rails db:migrate:animals
  104. Database task improvements github.com/rails/rails/pull/32274 github.com/rails/rails/compare/ ab43b5405ea2...5ddcda6d5f56

  105. Connections don’t work Migrations Rake tasks

  106. None
  107. “Are you concerned that Rails will become bloated?”

  108. What would Rails look like if we’d upstreamed these features

    5 years ago?
  109. a +

  110. ' ( ) * + ,

  111. None
  112. Maybe Rails doesn’t scale…?

  113. Your application isn’t special.

  114. Forces you to write generic code

  115. Train your future workforce

  116. Give back to the community

  117. ' ( ) * + ,

  118. a♥

  119. Scaling should make us happy

  120. Rails isn’t dying, it’s maturing

  121. None
  122. None
  123. None
  124. “Rails does scale!”

  125. The Future of Rails 6 Scalable by Default

  126. Eileen M. Uchitelle eileencodes.com @eileencodes