Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Paris Ruby Conference 2018: The Future of Rails 6

Paris Ruby Conference 2018: The Future of Rails 6

We've all heard the phrase "Rails doesn't scale". Long running test suites and no standard for implementing multiple databases makes it hard scale monolithic Rails applications. Rails 6 will start making Rails scalable by default with parallel testing and improved support for using multiple databases. You'll no longer be forced to reinvent the wheel and create your own solution to these problems. In this talk we'll take a look why these improvements are important, how they work, and ways in which small ideas can quickly snowball into major changes. This is just the beginning of Rails 6.

Eileen M. Uchitelle

June 28, 2018
Tweet

More Decks by Eileen M. Uchitelle

Other Decks in Programming

Transcript

  1. The Future of Rails 6
    Scalable by Default

    View full-size slide

  2. Eileen M. Uchitelle
    eileencodes.com
    @eileencodes

    View full-size slide

  3. “Rails doesn’t scale!”

    View full-size slide

  4. Rails doesn’t scale

    View full-size slide

  5. Rails doesn’t scale
    easily

    View full-size slide

  6. Rails doesn’t scale
    easily (yet)

    View full-size slide

  7. What does
    “scalable” mean?

    View full-size slide

  8. Your test suite shouldn’t
    block development

    View full-size slide

  9. Your application can
    handle data & traffic

    View full-size slide

  10. The Future of Rails 6
    Scalable by Default

    View full-size slide

  11. Parallel Testing
    Speeding up your test suite

    View full-size slide

  12. Forking
    - vs -
    Threads

    View full-size slide

  13. Forking Processes
    with dRB

    View full-size slide

  14. # test/test_helper.rb
    module ActiveSupport::TestCase
    parallelize(workers: 2)
    end

    View full-size slide

  15. def parallelize(workers: 2, with: :processes)
    workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]
    return if workers <= 1
    executor = case with
    when :processes
    Testing::Parallelization.new(workers)
    when :threads
    Minitest::Parallel::Executor.new(workers)
    else
    raise ArgumentError, "#{with} is 

    not a supported parallelization executor."
    end
    self.lock_threads = false if defined?(self.lock_threads) && 

    with == :threads
    Minitest.parallel_executor = executor
    parallelize_me!
    end

    View full-size slide

  16. def parallelize(workers: 2, with: :processes)
    workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]
    return if workers <= 1
    executor = case with
    when :processes
    Testing::Parallelization.new(workers)
    when :threads
    Minitest::Parallel::Executor.new(workers)
    else
    raise ArgumentError, "#{with} is 

    not a supported parallelization executor."
    end
    self.lock_threads = false if defined?(self.lock_threads) && 

    with == :threads
    Minitest.parallel_executor = executor
    parallelize_me!
    end

    View full-size slide

  17. def parallelize(workers: 2, with: :processes)
    workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]
    return if workers <= 1
    executor = case with
    when :processes
    Testing::Parallelization.new(workers)
    when :threads
    Minitest::Parallel::Executor.new(workers)
    else
    raise ArgumentError, "#{with} is 

    not a supported parallelization executor."
    end
    self.lock_threads = false if defined?(self.lock_threads) && 

    with == :threads
    Minitest.parallel_executor = executor
    parallelize_me!
    end

    View full-size slide

  18. module ActiveSupport::Testing::Parallelization
    […]
    def start
    @pool = @queue_size.times.map do |worker|
    fork do
    DRb.stop_service

    after_fork(worker)

    queue = DRbObject.new_with_uri(@url)

    while job = queue.pop
    klass = job[0]
    method = job[1]
    […]

    View full-size slide

  19. module ActiveSupport::Testing::Parallelization
    […]
    def start
    @pool = @queue_size.times.map do |worker|
    fork do
    DRb.stop_service

    after_fork(worker)

    queue = DRbObject.new_with_uri(@url)

    while job = queue.pop
    klass = job[0]
    method = job[1]
    […]

    View full-size slide

  20. module ActiveSupport::Testing::Parallelization
    […]
    def start
    @pool = @queue_size.times.map do |worker|
    fork do
    DRb.stop_service

    after_fork(worker)

    queue = DRbObject.new_with_uri(@url)

    while job = queue.pop
    klass = job[0]
    method = job[1]
    […]

    View full-size slide

  21. module ActiveSupport::Testing::Parallelization
    […]
    def start
    @pool = @queue_size.times.map do |worker|
    fork do
    DRb.stop_service

    after_fork(worker)

    queue = DRbObject.new_with_uri(@url)

    while job = queue.pop
    klass = job[0]
    method = job[1]
    […]

    View full-size slide

  22. module ActiveSupport::Testing::Parallelization
    […]
    def start
    @pool = @queue_size.times.map do |worker|
    fork do
    […]

    while job = queue.pop
    klass = job[0]
    method = job[1]
    reporter = job[2]
    result = Minitest.run_one_method(klass, method)
    queue.record(reporter, result)
    end
    run_cleanup(worker)
    […]

    View full-size slide

  23. module ActiveSupport::Testing::Parallelization
    […]
    def start
    @pool = @queue_size.times.map do |worker|
    fork do
    […]

    while job = queue.pop
    klass = job[0]
    method = job[1]
    reporter = job[2]
    result = Minitest.run_one_method(klass, method)
    queue.record(reporter, result)
    end
    run_cleanup(worker)
    […]

    View full-size slide

  24. module ActiveSupport::Testing::Parallelization
    […]
    def start
    @pool = @queue_size.times.map do |worker|
    fork do
    […]

    while job = queue.pop
    klass = job[0]
    method = job[1]
    reporter = job[2]
    result = Minitest.run_one_method(klass, method)
    queue.record(reporter, result)
    end
    run_cleanup(worker)
    […]

    View full-size slide

  25. module ActiveSupport::Testing::Parallelization
    […]
    def start
    @pool = @queue_size.times.map do |worker|
    fork do
    […]

    while job = queue.pop
    klass = job[0]
    method = job[1]
    reporter = job[2]
    result = Minitest.run_one_method(klass, method)
    queue.record(reporter, result)
    end
    run_cleanup(worker)
    […]

    View full-size slide

  26. module ActiveSupport::Testing::Parallelization
    […]
    def start
    […]
    end
    def shutdown
    @queue_size.times { @queue << nil }
    @pool.each { |pid| Process.waitpid(pid) }
    end
    […]
    end

    View full-size slide

  27. # test/test_helper.rb
    module ActiveSupport::TestCase
    parallelize(workers: 2)
    end

    View full-size slide


  28. test-database-0 test-database-1

    View full-size slide

  29. module ActiveSupport::Testing::Parallelization
    […]
    def start
    @pool = @queue_size.times.map do |worker|
    fork do
    DRb.stop_service

    after_fork(worker)

    queue = DRbObject.new_with_uri(@url)

    while job = queue.pop
    klass = job[0]
    method = job[1]
    […]

    View full-size slide

  30. module ActiveSupport::Testing::Parallelization
    […]
    def start
    @pool = @queue_size.times.map do |worker|
    fork do
    […]

    while job = queue.pop
    klass = job[0]
    method = job[1]
    reporter = job[2]
    result = Minitest.run_one_method(klass, method)
    queue.record(reporter, result)
    end
    run_cleanup(worker)
    […]

    View full-size slide

  31. Threaded testing
    with Minitest

    View full-size slide

  32. # test/test_helper.rb
    module ActiveSupport::TestCase
    parallelize(workers: 2, with: :threads)
    end

    View full-size slide

  33. def parallelize(workers: 2, with: :processes)
    workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]
    return if workers <= 1
    executor = case with
    when :processes
    Testing::Parallelization.new(workers)
    when :threads
    Minitest::Parallel::Executor.new(workers)
    else
    raise ArgumentError, "#{with} is 

    not a supported parallelization executor."
    end
    self.lock_threads = false if defined?(self.lock_threads) && 

    with == :threads
    Minitest.parallel_executor = executor
    parallelize_me!
    end

    View full-size slide


  34. Connection ID:
    #03938752
    Connection ID:
    #83321340

    View full-size slide


  35. Connection ID:
    #83321340
    Connection ID:
    #83321340

    View full-size slide

  36. This is so weird. We
    seem to have an
    isolation problem…

    View full-size slide


  37. Rails
    opens a
    connection

    View full-size slide


  38. Puma opens
    a second
    connection
    Rails
    opens a
    connection

    View full-size slide


  39. Puma opens
    a second
    connection
    Rails
    opens a
    connection

    View full-size slide

  40. Test thread
    connection
    number 1
    Test thread
    connection
    number 2

    View full-size slide

  41. def parallelize(workers: 2, with: :processes)
    workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]
    return if workers <= 1
    executor = case with
    when :processes
    Testing::Parallelization.new(workers)
    when :threads
    Minitest::Parallel::Executor.new(workers)
    else
    raise ArgumentError, "#{with} is 

    not a supported parallelization executor."
    end
    self.lock_threads = false if defined?(self.lock_threads) && 

    with == :threads
    Minitest.parallel_executor = executor
    parallelize_me!
    end

    View full-size slide

  42. $ rails test

    View full-size slide

  43. Parallel Testing PR
    github.com/rails/rails/pull/31900

    View full-size slide

  44. Your test suite shouldn’t
    block development

    View full-size slide

  45. Your application can
    handle data & traffic

    View full-size slide

  46. Multiple Databases
    Making multi-db behavior easy

    View full-size slide

  47. Table
    id: integer
    name: string
    Table
    id: integer
    name: string
    Table
    id: integer
    name: string
    Table
    id: integer
    name: string
    Primary Animals

    View full-size slide

  48. X Best practices are undocumented
    The State of Multi-db in Rails 5

    View full-size slide

  49. # 3-tier database.yml
    production:
    primary:
    <<: *default
    database: db_production
    animals:
    <<: *default
    database: db_animals_production

    View full-size slide

  50. db/animals_migrate
    ↳ 20180206210429_create_dogs.rb
    ↳ 20180206210437_create_cats.rb

    View full-size slide

  51. ActiveRecord::Migrator.migrations_paths =
    "db/animals_migrate"
    ActiveRecord::Base.migrate

    View full-size slide

  52. X Best practices are undocumented
    X Migrations don’t work
    The State of Multi-db in Rails 5

    View full-size slide

  53. # app/models/animals_base.rb
    class AnimalsBase < ActiveRecord::Base
    self.abstract_class = true
    establish_connection :animals
    end

    View full-size slide

  54. # app/models/dog.rb
    class Dog < AnimalsBase
    […]
    end
    # app/models/cat.rb
    class Cat < AnimalsBase
    […]
    end

    View full-size slide

  55. X Best practices are undocumented
    X Migrations don’t work
    X Database tasks don’t exist
    The State of Multi-db in Rails 5

    View full-size slide

  56. ActiveRecord::AdapterNotSpecified:
    'production' database is not configured.

    View full-size slide

  57. # activerecord/lib/active_record/railtie.rb
    initializer "active_record.initialize_database" do
    ActiveSupport.on_load(:active_record) do
    self.configurations =
    Rails.application.config.database_configuration
    begin
    establish_connection
    rescue ActiveRecord::NoDatabaseError
    warn <<-end_warning

    View full-size slide

  58. # activerecord/lib/active_record/railtie.rb
    initializer "active_record.initialize_database" do
    ActiveSupport.on_load(:active_record) do
    self.configurations =
    Rails.application.config.database_configuration
    begin
    establish_connection
    rescue ActiveRecord::NoDatabaseError
    warn <<-end_warning

    View full-size slide

  59. X Best practices are undocumented
    X Migrations don’t work
    X Database tasks don’t exist
    X Default connection was broken
    The State of Multi-db in Rails 5

    View full-size slide

  60. Migrations
    Connections
    don’t work
    Rake tasks

    View full-size slide

  61. Fixing migrations for
    multiple databases

    View full-size slide

  62. path = Rails.root.to_s + "/db/animals_migrate"
    desc "Migrate the cluster db"
    task migrate: :environment do
    ActiveRecord::Migrator.migrations_paths = path
    ActiveRecord::Base.establish_connection(:animals)
    ActiveRecord::Tasks::DatabaseTasks.migrate
    db_namespace["db:schema:dump"].invoke
    end

    View full-size slide

  63. # 3-tier database.yml
    production:
    primary:
    <<: *default
    database: db_production
    animals:
    <<: *default
    database: db_animals_production
    migrations_paths: db/animals_migrate

    View full-size slide

  64. >> Dog.connection.migrations_paths
    => "db/animals_migrate"

    View full-size slide

  65. Migrations Paths Refactoring PR
    github.com/rails/rails/pull/31727

    View full-size slide

  66. X Best practices are undocumented
    ✔ Migrations don’t work
    X Database tasks don’t exist
    X Default connection was broken
    The State of Multi-db in Rails 5

    View full-size slide

  67. Migrations
    Connections
    don’t work
    Rake tasks

    View full-size slide

  68. Improving the database
    tasks for multi-dbs

    View full-size slide

  69. rails db:drop
    rails db:create
    rails db:migrate

    View full-size slide

  70. >> ActiveRecord::Base.configurations["production"]
    >> {
    "adapter"=>"mysql2",
    "database"=>"single_database_production"
    }

    View full-size slide

  71. >> ActiveRecord::Base.configurations["production"]
    >> {
    "primary"=>{"adapter"=>"mysql2",
    "database"=>"multiple_databases_production"},
    "animals"=>{"adapter"=>"mysql2",
    “database"=>"multiple_databases_production"}
    }

    View full-size slide

  72. module ActiveRecord
    module DatabaseConfigurations
    class DatabaseConfig
    attr_reader :env_name, :spec_name, :config
    def initialize(env_name, spec_name, config)
    @env_name = env_name
    @spec_name = spec_name
    @config = config
    end
    end
    end
    end

    View full-size slide

  73. >> db_config = DatabaseConfig.new("production",
    "animals", {"adapter"=>"mysql2",
    "database"=>"multiple_databases_production_animals"
    })
    >>
    #fig:0x00007f994571edb0 @env_name="production",
    @spec_name="animals", @config={"adapter"=>"mysql2",
    "database"=>"multiple_databases_production_animals"
    }>

    View full-size slide

  74. >> db_config.env_name
    "production"

    View full-size slide

  75. >> db_config.env_name
    "production"
    >> db_config.spec_name
    "animals"

    View full-size slide

  76. >> db_config.env_name
    "production"
    >> db_config.spec_name
    "animals"
    >> db_config.config
    {…"database"=>"multiple_databases_producti
    on_animals"}

    View full-size slide

  77. def self.db_configs(configs = configurations)
    configs.each_pair.flat_map do |env_name, config|
    walk_configs(env_name, "primary", config)
    end
    end

    View full-size slide

  78. def self.configs_for(env, configs, &blk)
    env_with_configs =
    db_configs(configs).select do |db_config|
    db_config.env_name == env
    end
    if block_given?
    env_with_configs.each do |ewc|
    yield ewc.spec_name, ewc.config
    end
    else
    env_with_configs
    end
    end

    View full-size slide

  79. >> DatabaseConfigurations.configs_for("production")
    >>[#Config:0x00007f994571edb0 @env_name="production",
    @spec_name="primary", @config={"adapter"=>"mysql2",
    "database"=>"multiple_databases_production"}>,
    #fig:0x00007f994571edb0 @env_name="production",
    @spec_name="animals", @config={"adapter"=>"mysql2",
    "database"=>"multiple_databases_production_animals"
    }>]

    View full-size slide

  80. task migrate: :load_config do
    ActiveRecord::Tasks::DatabaseTasks.migrate
    db_namespace["_dump"].invoke
    end

    View full-size slide

  81. task migrate: :load_config do
    ActiveRecord::DatabaseConfigurations.configs_for(
    Rails.env
    ) do |_, config|
    ActiveRecord::Base.establish_connection(config)
    ActiveRecord::Tasks::DatabaseTasks.migrate
    end
    db_namespace["_dump"].invoke
    end

    View full-size slide

  82. task migrate: :load_config do
    ActiveRecord::DatabaseConfigurations.configs_for(
    Rails.env
    ) do |_, config|
    ActiveRecord::Base.establish_connection(config)
    ActiveRecord::Tasks::DatabaseTasks.migrate
    end
    db_namespace["_dump"].invoke
    end

    View full-size slide

  83. task migrate: :load_config do
    ActiveRecord::DatabaseConfigurations.configs_for(
    Rails.env
    ) do |_, config|
    ActiveRecord::Base.establish_connection(config)
    ActiveRecord::Tasks::DatabaseTasks.migrate
    end
    db_namespace["_dump"].invoke
    end

    View full-size slide

  84. task migrate: :load_config do
    ActiveRecord::DatabaseConfigurations.configs_for(
    Rails.env
    ) do |_, config|
    ActiveRecord::Base.establish_connection(config)
    ActiveRecord::Tasks::DatabaseTasks.migrate
    end
    db_namespace["_dump"].invoke
    end

    View full-size slide

  85. rails db:drop
    rails db:create
    rails db:migrate

    View full-size slide

  86. rails db:drop:primary
    rails db:drop:animals
    rails db:create:primary
    rails db:create:animals
    rails db:migrate:primary
    rails db:migrate:animals

    View full-size slide

  87. namespace :migrate
    ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name|
    desc "Migrate #{spec_name} database"
    task spec_name => :load_config do
    db_config = ActiveRecord::DatabaseConfigurations
    .config_for_env_and_spec(Rails.env, spec_name)
    ActiveRecord::Base.establish_connection(
    db_config.config
    )
    ActiveRecord::Tasks::DatabaseTasks.migrate
    end
    end
    end
    end

    View full-size slide

  88. namespace :migrate
    ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name|
    desc "Migrate #{spec_name} database"
    task spec_name => :load_config do
    db_config = ActiveRecord::DatabaseConfigurations
    .config_for_env_and_spec(Rails.env, spec_name)
    ActiveRecord::Base.establish_connection(
    db_config.config
    )
    ActiveRecord::Tasks::DatabaseTasks.migrate
    end
    end
    end
    end

    View full-size slide

  89. namespace :migrate
    ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name|
    desc "Migrate #{spec_name} database"
    task spec_name => :load_config do
    db_config = ActiveRecord::DatabaseConfigurations
    .config_for_env_and_spec(Rails.env, spec_name)
    ActiveRecord::Base.establish_connection(
    db_config.config
    )
    ActiveRecord::Tasks::DatabaseTasks.migrate
    end
    end
    end
    end

    View full-size slide

  90. namespace :migrate
    ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name|
    desc "Migrate #{spec_name} database"
    task spec_name => :load_config do
    db_config = ActiveRecord::DatabaseConfigurations
    .config_for_env_and_spec(Rails.env, spec_name)
    ActiveRecord::Base.establish_connection(
    db_config.config
    )
    ActiveRecord::Tasks::DatabaseTasks.migrate
    end
    end
    end
    end

    View full-size slide

  91. namespace :migrate
    ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name|
    desc "Migrate #{spec_name} database"
    task spec_name => :load_config do
    db_config = ActiveRecord::DatabaseConfigurations
    .config_for_env_and_spec(Rails.env, spec_name)
    ActiveRecord::Base.establish_connection(
    db_config.config
    )
    ActiveRecord::Tasks::DatabaseTasks.migrate
    end
    end
    end
    end

    View full-size slide

  92. rails db:drop:primary
    rails db:drop:animals
    rails db:create:primary
    rails db:create:animals
    rails db:migrate:primary
    rails db:migrate:animals

    View full-size slide

  93. Database task improvements
    github.com/rails/rails/pull/32274
    github.com/rails/rails/compare/
    ab43b5405ea2...5ddcda6d5f56

    View full-size slide

  94. Connections
    don’t work
    Migrations
    Rake tasks

    View full-size slide

  95. “Are you concerned
    that Rails will become
    bloated?”

    View full-size slide

  96. What would Rails look like
    if we’d upstreamed these
    features 5 years ago?

    View full-size slide

  97. Maybe Rails doesn’t
    scale…?

    View full-size slide

  98. Your application
    isn’t special.

    View full-size slide

  99. Forces you to write
    generic code

    View full-size slide

  100. Train your future
    workforce

    View full-size slide

  101. Give back to the
    community

    View full-size slide

  102. Scaling should
    make us happy

    View full-size slide

  103. Rails isn’t dying,
    it’s maturing

    View full-size slide

  104. “Rails does scale!”

    View full-size slide

  105. The Future of Rails 6
    Scalable by Default

    View full-size slide

  106. Eileen M. Uchitelle
    eileencodes.com
    @eileencodes

    View full-size slide