RailsConf 2018 | The Future of Rails 6: Scalable by Default

RailsConf 2018 | The Future of Rails 6: Scalable by Default

We've all heard the phrase "Rails doesn't scale". Long running test suites and no standard for implementing multiple databases makes it hard scale monolithic Rails applications. Rails 6 will start making Rails scalable by default with parallel testing and improved support for using multiple databases. You'll no longer be forced to reinvent the wheel and create your own solution to these problems. In this talk we'll take a look why these improvements are important, how they work, and ways in which small ideas can quickly snowball into major changes. This is just the beginning of Rails 6.

C44e1f7e22c3f23cff7bc130871047ef?s=128

Eileen M. Uchitelle

April 18, 2018
Tweet

Transcript

  1. 3.

    a

  2. 6.
  3. 15.

    a

  4. 16.

    a

  5. 20.

    def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]

    return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
  6. 21.

    def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]

    return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
  7. 24.

    def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]

    return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
  8. 25.

    module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]
  9. 26.

    module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]
  10. 27.

    module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]
  11. 28.

    module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]
  12. 29.

    module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]
  13. 30.

    module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]
  14. 31.

    module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]
  15. 32.

    module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]
  16. 33.

    module ActiveSupport::Testing::Parallelization […] def start […] end def shutdown @queue_size.times

    { @queue << nil } @pool.each { |pid| Process.waitpid(pid) } end […] end
  17. 36.

    module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]
  18. 37.

    module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]
  19. 38.

    # test/test_helper.rb module ActiveSupport::TestCase parallelize(workers: 2) parallelize_setup do |worker| ActiveRecord::Tasks::DatabaseTasks.create(:animals)

    ActiveRecord::Base.establish_connection(:animals) ActiveRecord::Tasks::DatabaseTasks.load_schema end parallelize_teardown do |worker| ActiveRecord::Tasks::DatabaseTasks.drop(:animals) end end
  20. 41.

    def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]

    return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
  21. 42.
  22. 46.
  23. 51.

    def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]

    return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
  24. 54.

    def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]

    return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
  25. 58.
  26. 59.
  27. 62.
  28. 64.

    Table id: integer name: string Table id: integer name: string

    Table id: integer name: string Table id: integer name: string Primary Animals
  29. 74.

    X Best practices are undocumented X Migrations don’t work X

    Database tasks don’t exist The State of Multi-db in Rails 5
  30. 78.

    X Best practices are undocumented X Migrations don’t work X

    Database tasks don’t exist X Default connection was broken The State of Multi-db in Rails 5
  31. 80.
  32. 82.

    path = Rails.root.to_s + "/db/animals_migrate" desc "Migrate the cluster db"

    task migrate: :environment do ActiveRecord::Migrator.migrations_paths = path ActiveRecord::Base.establish_connection(:animals) ActiveRecord::Tasks::DatabaseTasks.migrate db_namespace["db:schema:dump"].invoke end
  33. 83.

    # 3-tier database.yml production: primary: <<: *default database: db_production animals:

    <<: *default database: db_animals_production migrations_paths: db/animals_migrate
  34. 86.

    X Best practices are undocumented ✔ Migrations don’t work X

    Database tasks don’t exist X Default connection was broken The State of Multi-db in Rails 5
  35. 92.

    module ActiveRecord module DatabaseConfigurations class DatabaseConfig attr_reader :env_name, :spec_name, :config

    def initialize(env_name, spec_name, config) @env_name = env_name @spec_name = spec_name @config = config end end end end
  36. 93.

    module ActiveRecord module DatabaseConfigurations class DatabaseConfig attr_reader :env_name, :spec_name, :config

    def initialize(env_name, spec_name, config) @env_name = env_name @spec_name = spec_name @config = config end end end end
  37. 94.

    module ActiveRecord module DatabaseConfigurations class DatabaseConfig attr_reader :env_name, :spec_name, :config

    def initialize(env_name, spec_name, config) @env_name = env_name @spec_name = spec_name @config = config end end end end
  38. 95.

    module ActiveRecord module DatabaseConfigurations class DatabaseConfig attr_reader :env_name, :spec_name, :config

    def initialize(env_name, spec_name, config) @env_name = env_name @spec_name = spec_name @config = config end end end end
  39. 96.

    >> db_config = DatabaseConfig.new("production", "animals", {"adapter"=>"mysql2", "database"=>"multiple_databases_production_animals" }) >> #<ActiveRecord::DatabaseConfigurations::DatabaseCon

    fig:0x00007f994571edb0 @env_name="production", @spec_name="animals", @config={"adapter"=>"mysql2", "database"=>"multiple_databases_production_animals" }>
  40. 101.

    def self.configs_for(env, configs, &blk) env_with_configs = db_configs(configs).select do |db_config| db_config.env_name

    == env end if block_given? env_with_configs.each do |ewc| yield ewc.spec_name, ewc.config end else env_with_configs end end
  41. 102.

    def self.configs_for(env, configs, &blk) env_with_configs = db_configs(configs).select do |db_config| db_config.env_name

    == env end if block_given? env_with_configs.each do |ewc| yield ewc.spec_name, ewc.config end else env_with_configs end end
  42. 103.

    def self.configs_for(env, configs, &blk) env_with_configs = db_configs(configs).select do |db_config| db_config.env_name

    == env end if block_given? env_with_configs.each do |ewc| yield ewc.spec_name, ewc.config end else env_with_configs end end
  43. 106.

    task migrate: :load_config do ActiveRecord::DatabaseConfigurations.configs_for( Rails.env ) do |_, config|

    ActiveRecord::Base.establish_connection(config) ActiveRecord::Tasks::DatabaseTasks.migrate end db_namespace["_dump"].invoke end
  44. 107.

    task migrate: :load_config do ActiveRecord::DatabaseConfigurations.configs_for( Rails.env ) do |_, config|

    ActiveRecord::Base.establish_connection(config) ActiveRecord::Tasks::DatabaseTasks.migrate end db_namespace["_dump"].invoke end
  45. 108.

    task migrate: :load_config do ActiveRecord::DatabaseConfigurations.configs_for( Rails.env ) do |_, config|

    ActiveRecord::Base.establish_connection(config) ActiveRecord::Tasks::DatabaseTasks.migrate end db_namespace["_dump"].invoke end
  46. 109.

    task migrate: :load_config do ActiveRecord::DatabaseConfigurations.configs_for( Rails.env ) do |_, config|

    ActiveRecord::Base.establish_connection(config) ActiveRecord::Tasks::DatabaseTasks.migrate end db_namespace["_dump"].invoke end
  47. 112.

    namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task

    spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end
  48. 113.

    namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task

    spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end
  49. 114.

    namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task

    spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end
  50. 115.

    namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task

    spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end
  51. 116.

    namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task

    spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end
  52. 119.

    X Best practices are undocumented ✔ Migrations don’t work ✔

    Database tasks don’t exist X Default connection was broken The State of Multi-db in Rails 5
  53. 121.
  54. 125.

    a +

  55. 127.
  56. 134.
  57. 137.
  58. 138.
  59. 139.