Paris Ruby Conference 2018: The Future of Rails 6

We've all heard the phrase "Rails doesn't scale". Long running test suites and no standard for implementing multiple databases makes it hard scale monolithic Rails applications. Rails 6 will start making Rails scalable by default with parallel testing and improved support for using multiple databases. You'll no longer be forced to reinvent the wheel and create your own solution to these problems. In this talk we'll take a look why these improvements are important, how they work, and ways in which small ideas can quickly snowball into major changes. This is just the beginning of Rails 6.

Transcript

1. The Future of Rails 6 Scalable by Default
2. Eileen M. Uchitelle eileencodes.com @eileencodes
3. a
4. Core Team
5. “Rails doesn’t scale!”
7. Rails doesn’t scale
8. Rails doesn’t scale easily
9. Rails doesn’t scale easily (yet)
10. What does “scalable” mean?
11. Your test suite shouldn’t block development
12. Your application can handle data & traffic
13. The Future of Rails 6 Scalable by Default
14. Parallel Testing Speeding up your test suite
15. a
16. a
17. Forking - vs - Threads
18. Forking Processes with dRB
19. # test/test_helper.rb module ActiveSupport::TestCase parallelize(workers: 2) end
20. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"] return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
21. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"] return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
22. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"] return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
23. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker| fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]
24. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker| fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]
25. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker| fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]
26. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker| fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]
27. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker| fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]
28. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker| fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]
29. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker| fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]
30. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker| fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]
31. module ActiveSupport::Testing::Parallelization […] def start […] end def shutdown @queue_size.times { @queue << nil } @pool.each { |pid| Process.waitpid(pid) } end […] end
32. # test/test_helper.rb module ActiveSupport::TestCase parallelize(workers: 2) end
33. test-database-0 test-database-1
34. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker| fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]
35. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker| fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]
36. Threaded testing with Minitest
37. # test/test_helper.rb module ActiveSupport::TestCase parallelize(workers: 2, with: :threads) end
38. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"] return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
40. Connection ID: #03938752 Connection ID: #83321340
41. Connection ID: #83321340 Connection ID: #83321340
42. This is so weird. We seem to have an isolation problem…
44. Rails opens a connection
45. Puma opens a second connection Rails opens a connection
46. Puma opens a second connection Rails opens a connection
47. Test thread connection number 1 Test thread connection number 2
48. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"] return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
49. $ rails test
50. Parallel Testing PR github.com/rails/rails/pull/31900
51. Your test suite shouldn’t block development
52. Your application can handle data & traffic
55. DB 1 DB 2 DB 3
56. Multiple Databases Making multi-db behavior easy
57. Table id: integer name: string Table id: integer name: string Table id: integer name: string Table id: integer name: string Primary Animals
58. X Best practices are undocumented The State of Multi-db in Rails 5
59. # 3-tier database.yml production: primary: <<: *default database: db_production animals: <<: *default database: db_animals_production
60. db/animals_migrate ↳ 20180206210429_create_dogs.rb ↳ 20180206210437_create_cats.rb
61. ActiveRecord::Migrator.migrations_paths = "db/animals_migrate" ActiveRecord::Base.migrate
62. X Best practices are undocumented X Migrations don’t work The State of Multi-db in Rails 5
63. # app/models/animals_base.rb class AnimalsBase < ActiveRecord::Base self.abstract_class = true establish_connection :animals end
64. # app/models/dog.rb class Dog < AnimalsBase […] end # app/models/cat.rb class Cat < AnimalsBase […] end
65. X Best practices are undocumented X Migrations don’t work X Database tasks don’t exist The State of Multi-db in Rails 5
66. ActiveRecord::AdapterNotSpecified: 'production' database is not configured.
67. # activerecord/lib/active_record/railtie.rb initializer "active_record.initialize_database" do ActiveSupport.on_load(:active_record) do self.configurations = Rails.application.config.database_configuration begin establish_connection rescue ActiveRecord::NoDatabaseError warn <<-end_warning
68. # activerecord/lib/active_record/railtie.rb initializer "active_record.initialize_database" do ActiveSupport.on_load(:active_record) do self.configurations = Rails.application.config.database_configuration begin establish_connection rescue ActiveRecord::NoDatabaseError warn <<-end_warning
69. X Best practices are undocumented X Migrations don’t work X Database tasks don’t exist X Default connection was broken The State of Multi-db in Rails 5
70. Migrations Connections don’t work Rake tasks
72. Fixing migrations for multiple databases
73. path = Rails.root.to_s + "/db/animals_migrate" desc "Migrate the cluster db" task migrate: :environment do ActiveRecord::Migrator.migrations_paths = path ActiveRecord::Base.establish_connection(:animals) ActiveRecord::Tasks::DatabaseTasks.migrate db_namespace["db:schema:dump"].invoke end
74. # 3-tier database.yml production: primary: <<: *default database: db_production animals: <<: *default database: db_animals_production migrations_paths: db/animals_migrate
75. >> Dog.connection.migrations_paths => "db/animals_migrate"
76. Migrations Paths Refactoring PR github.com/rails/rails/pull/31727
77. X Best practices are undocumented ✔ Migrations don’t work X Database tasks don’t exist X Default connection was broken The State of Multi-db in Rails 5
78. Migrations Connections don’t work Rake tasks
79. Improving the database tasks for multi-dbs
80. rails db:drop rails db:create rails db:migrate
81. >> ActiveRecord::Base.configurations["production"] >> { "adapter"=>"mysql2", "database"=>"single_database_production" }
82. >> ActiveRecord::Base.configurations["production"] >> { "primary"=>{"adapter"=>"mysql2", "database"=>"multiple_databases_production"}, "animals"=>{"adapter"=>"mysql2", “database"=>"multiple_databases_production"} }
83. module ActiveRecord module DatabaseConfigurations class DatabaseConfig attr_reader :env_name, :spec_name, :config def initialize(env_name, spec_name, config) @env_name = env_name @spec_name = spec_name @config = config end end end end
84. >> db_config = DatabaseConfig.new("production", "animals", {"adapter"=>"mysql2", "database"=>"multiple_databases_production_animals" }) >> #<ActiveRecord::DatabaseConfigurations::DatabaseCon fig:0x00007f994571edb0 @env_name="production", @spec_name="animals", @config={"adapter"=>"mysql2", "database"=>"multiple_databases_production_animals" }>
85. >> db_config.env_name "production"
86. >> db_config.env_name "production" >> db_config.spec_name "animals"
87. >> db_config.env_name "production" >> db_config.spec_name "animals" >> db_config.config {…"database"=>"multiple_databases_producti on_animals"}
88. def self.db_configs(configs = configurations) configs.each_pair.flat_map do |env_name, config| walk_configs(env_name, "primary", config) end end
89. def self.configs_for(env, configs, &blk) env_with_configs = db_configs(configs).select do |db_config| db_config.env_name == env end if block_given? env_with_configs.each do |ewc| yield ewc.spec_name, ewc.config end else env_with_configs end end
90. >> DatabaseConfigurations.configs_for("production") >>[#<ActiveRecord::DatabaseConfigurations::Database Config:0x00007f994571edb0 @env_name="production", @spec_name="primary", @config={"adapter"=>"mysql2", "database"=>"multiple_databases_production"}>, #<ActiveRecord::DatabaseConfigurations::DatabaseCon fig:0x00007f994571edb0 @env_name="production", @spec_name="animals", @config={"adapter"=>"mysql2", "database"=>"multiple_databases_production_animals" }>]
91. task migrate: :load_config do ActiveRecord::Tasks::DatabaseTasks.migrate db_namespace["_dump"].invoke end
92. task migrate: :load_config do ActiveRecord::DatabaseConfigurations.configs_for( Rails.env ) do |_, config| ActiveRecord::Base.establish_connection(config) ActiveRecord::Tasks::DatabaseTasks.migrate end db_namespace["_dump"].invoke end
93. task migrate: :load_config do ActiveRecord::DatabaseConfigurations.configs_for( Rails.env ) do |_, config| ActiveRecord::Base.establish_connection(config) ActiveRecord::Tasks::DatabaseTasks.migrate end db_namespace["_dump"].invoke end
94. task migrate: :load_config do ActiveRecord::DatabaseConfigurations.configs_for( Rails.env ) do |_, config| ActiveRecord::Base.establish_connection(config) ActiveRecord::Tasks::DatabaseTasks.migrate end db_namespace["_dump"].invoke end
95. task migrate: :load_config do ActiveRecord::DatabaseConfigurations.configs_for( Rails.env ) do |_, config| ActiveRecord::Base.establish_connection(config) ActiveRecord::Tasks::DatabaseTasks.migrate end db_namespace["_dump"].invoke end
96. rails db:drop rails db:create rails db:migrate
97. rails db:drop:primary rails db:drop:animals rails db:create:primary rails db:create:animals rails db:migrate:primary rails db:migrate:animals
98. namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end
99. namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end
100. namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end
101. namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end
102. namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end
103. rails db:drop:primary rails db:drop:animals rails db:create:primary rails db:create:animals rails db:migrate:primary rails db:migrate:animals
104. Database task improvements github.com/rails/rails/pull/32274 github.com/rails/rails/compare/ ab43b5405ea2...5ddcda6d5f56
105. Connections don’t work Migrations Rake tasks
107. “Are you concerned that Rails will become bloated?”
108. What would Rails look like if we’d upstreamed these features 5 years ago?
109. a +
110. ' ( ) * + ,
112. Maybe Rails doesn’t scale…?
113. Your application isn’t special.
114. Forces you to write generic code
115. Train your future workforce
116. Give back to the community
117. ' ( ) * + ,
118. a♥
119. Scaling should make us happy
120. Rails isn’t dying, it’s maturing
124. “Rails does scale!”
125. The Future of Rails 6 Scalable by Default
126. Eileen M. Uchitelle eileencodes.com @eileencodes