Upgrade to Pro — share decks privately, control downloads, hide ads and more …

RailsConf 2018 | The Future of Rails 6: Scalabl...

RailsConf 2018 | The Future of Rails 6: Scalable by Default

We've all heard the phrase "Rails doesn't scale". Long running test suites and no standard for implementing multiple databases makes it hard scale monolithic Rails applications. Rails 6 will start making Rails scalable by default with parallel testing and improved support for using multiple databases. You'll no longer be forced to reinvent the wheel and create your own solution to these problems. In this talk we'll take a look why these improvements are important, how they work, and ways in which small ideas can quickly snowball into major changes. This is just the beginning of Rails 6.

Eileen M. Uchitelle

April 18, 2018
Tweet

More Decks by Eileen M. Uchitelle

Other Decks in Programming

Transcript

  1. a

  2. a

  3. a

  4. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]

    return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
  5. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]

    return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
  6. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]

    return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
  7. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]
  8. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]
  9. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]
  10. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]
  11. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]
  12. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]
  13. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]
  14. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]
  15. module ActiveSupport::Testing::Parallelization […] def start […] end def shutdown @queue_size.times

    { @queue << nil } @pool.each { |pid| Process.waitpid(pid) } end […] end
  16. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]
  17. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]
  18. # test/test_helper.rb module ActiveSupport::TestCase parallelize(workers: 2) parallelize_setup do |worker| ActiveRecord::Tasks::DatabaseTasks.create(:animals)

    ActiveRecord::Base.establish_connection(:animals) ActiveRecord::Tasks::DatabaseTasks.load_schema end parallelize_teardown do |worker| ActiveRecord::Tasks::DatabaseTasks.drop(:animals) end end
  19. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]

    return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
  20. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]

    return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
  21. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]

    return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
  22. Table id: integer name: string Table id: integer name: string

    Table id: integer name: string Table id: integer name: string Primary Animals
  23. X Best practices are undocumented X Migrations don’t work X

    Database tasks don’t exist The State of Multi-db in Rails 5
  24. X Best practices are undocumented X Migrations don’t work X

    Database tasks don’t exist X Default connection was broken The State of Multi-db in Rails 5
  25. path = Rails.root.to_s + "/db/animals_migrate" desc "Migrate the cluster db"

    task migrate: :environment do ActiveRecord::Migrator.migrations_paths = path ActiveRecord::Base.establish_connection(:animals) ActiveRecord::Tasks::DatabaseTasks.migrate db_namespace["db:schema:dump"].invoke end
  26. # 3-tier database.yml production: primary: <<: *default database: db_production animals:

    <<: *default database: db_animals_production migrations_paths: db/animals_migrate
  27. X Best practices are undocumented ✔ Migrations don’t work X

    Database tasks don’t exist X Default connection was broken The State of Multi-db in Rails 5
  28. module ActiveRecord module DatabaseConfigurations class DatabaseConfig attr_reader :env_name, :spec_name, :config

    def initialize(env_name, spec_name, config) @env_name = env_name @spec_name = spec_name @config = config end end end end
  29. module ActiveRecord module DatabaseConfigurations class DatabaseConfig attr_reader :env_name, :spec_name, :config

    def initialize(env_name, spec_name, config) @env_name = env_name @spec_name = spec_name @config = config end end end end
  30. module ActiveRecord module DatabaseConfigurations class DatabaseConfig attr_reader :env_name, :spec_name, :config

    def initialize(env_name, spec_name, config) @env_name = env_name @spec_name = spec_name @config = config end end end end
  31. module ActiveRecord module DatabaseConfigurations class DatabaseConfig attr_reader :env_name, :spec_name, :config

    def initialize(env_name, spec_name, config) @env_name = env_name @spec_name = spec_name @config = config end end end end
  32. >> db_config = DatabaseConfig.new("production", "animals", {"adapter"=>"mysql2", "database"=>"multiple_databases_production_animals" }) >> #<ActiveRecord::DatabaseConfigurations::DatabaseCon

    fig:0x00007f994571edb0 @env_name="production", @spec_name="animals", @config={"adapter"=>"mysql2", "database"=>"multiple_databases_production_animals" }>
  33. def self.configs_for(env, configs, &blk) env_with_configs = db_configs(configs).select do |db_config| db_config.env_name

    == env end if block_given? env_with_configs.each do |ewc| yield ewc.spec_name, ewc.config end else env_with_configs end end
  34. def self.configs_for(env, configs, &blk) env_with_configs = db_configs(configs).select do |db_config| db_config.env_name

    == env end if block_given? env_with_configs.each do |ewc| yield ewc.spec_name, ewc.config end else env_with_configs end end
  35. def self.configs_for(env, configs, &blk) env_with_configs = db_configs(configs).select do |db_config| db_config.env_name

    == env end if block_given? env_with_configs.each do |ewc| yield ewc.spec_name, ewc.config end else env_with_configs end end
  36. task migrate: :load_config do ActiveRecord::DatabaseConfigurations.configs_for( Rails.env ) do |_, config|

    ActiveRecord::Base.establish_connection(config) ActiveRecord::Tasks::DatabaseTasks.migrate end db_namespace["_dump"].invoke end
  37. task migrate: :load_config do ActiveRecord::DatabaseConfigurations.configs_for( Rails.env ) do |_, config|

    ActiveRecord::Base.establish_connection(config) ActiveRecord::Tasks::DatabaseTasks.migrate end db_namespace["_dump"].invoke end
  38. task migrate: :load_config do ActiveRecord::DatabaseConfigurations.configs_for( Rails.env ) do |_, config|

    ActiveRecord::Base.establish_connection(config) ActiveRecord::Tasks::DatabaseTasks.migrate end db_namespace["_dump"].invoke end
  39. task migrate: :load_config do ActiveRecord::DatabaseConfigurations.configs_for( Rails.env ) do |_, config|

    ActiveRecord::Base.establish_connection(config) ActiveRecord::Tasks::DatabaseTasks.migrate end db_namespace["_dump"].invoke end
  40. namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task

    spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end
  41. namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task

    spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end
  42. namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task

    spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end
  43. namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task

    spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end
  44. namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task

    spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end
  45. X Best practices are undocumented ✔ Migrations don’t work ✔

    Database tasks don’t exist X Default connection was broken The State of Multi-db in Rails 5
  46. a +