Upgrade to Pro — share decks privately, control downloads, hide ads and more …

やさしいActiveRecordのDB接続のしくみ

ANDPAD inc
October 27, 2023

 やさしいActiveRecordのDB接続のしくみ

久保 路 @amamanamam
2023 年 10 月 27 日
Kaigi on Rails 2023
私たちは database.yml に必要情報を記述し、適当なモデルの find / where メソッドを使うだけで、いつの間にか DB 接続の確立がなされて、ほしいデータがいつでも簡単に取得できるようになっています。

さて、 ActiveRecord 内部ではどのような処理を以て、この簡単さを実現しているのでしょうか?普段あまり着目しない部分ですので、ブラックボックス化してる方も少なくないと思います。 そこで、本セッションでは MySQL への接続確立するまでの ActiveRecord 内部の動きについて紹介します。 具体的には、接続に関する各クラスの役割( active_record/connection_handler 等)を適宜紹介しつつ、実際に find メソッドをデバッグ実行した際のスタックトレースを通して、接続確立までの ActiveRecord の内部処理の全容を持ち帰っていただきます。

ANDPAD inc

October 27, 2023
Tweet

More Decks by ANDPAD inc

Other Decks in Technology

Transcript

  1. Copyright © 2020 Present ANDPAD Inc. This information is confidential

    and was prepared by ANDPAD Inc. for the use of our client. It is not to be relied on by and 3rd party. Proprietary & Confidential 無断転載・無断複製の禁止 2023.10.27 @Kaigi on Rails くぼ(@amamanamam)/ 株式会社アンドパッド やさしいActiveRecordのDB接続のしくみ 

  2. 環境 • Ruby : 3.2.0 • Rails : 7.1.0.alpha •

    mysql2 : 0.5.5 • MySQL : 8.0.28-debug
  3. クライアント/サーバー間の通信 Copyright © 2020 Present ANDPAD Inc. This information is

    confidential and was prepared by ANDPAD Inc. for the use of our client. It is not to be relied on by and 3rd party. Proprietary & Confidential 無断転載・無断複製の禁止
  4. クライアント/サーバー側でやること Copyright © 2020 Present ANDPAD Inc. This information is

    confidential and was prepared by ANDPAD Inc. for the use of our client. It is not to be relied on by and 3rd party. Proprietary & Confidential 無断転載・無断複製の禁止
  5. 全体図 Active Record mysql2 Application DB C API (libmysqlclient) ORM

    adapter mysql_real_connect(..) • イニシャルハンド シェイクパケットの 送信 • 認証方式の決定 • 決定した方式で認証 • OKパケットかERRパ ケットの送信
  6. ActiveRecordの接続までの過程 (クラスとモジュールの説明) Copyright © 2020 Present ANDPAD Inc. This information

    is confidential and was prepared by ANDPAD Inc. for the use of our client. It is not to be relied on by and 3rd party. Proprietary & Confidential 無断転載・無断複製の禁止
  7. ActiveRecordの接続までの過程 (ソースの説明) Copyright © 2020 Present ANDPAD Inc. This information

    is confidential and was prepared by ANDPAD Inc. for the use of our client. It is not to be relied on by and 3rd party. Proprietary & Confidential 無断転載・無断複製の禁止
  8. 接続要求 module ConnectionAdapters # = Active Record MySQL2 Adapter class

    Mysql2Adapter < AbstractMysqlAdapter …… class << self def new_client(config) ::Mysql2::Client.new(config) rescue ::Mysql2::Error => error if error.error_number == ConnectionAdapters::Mysql2Adapter::ER_BAD_DB_ERROR raise ActiveRecord::NoDatabaseError.db_error(config[:database]) elsif error.error_number == ConnectionAdapters::Mysql2Adapter::ER_ACCESS_DENIED_ERROR raise ActiveRecord::DatabaseConnectionError.username_error(config[:username]) elsif [ConnectionAdapters::Mysql2Adapter::ER_CONN_HOST_ERROR, ConnectionAdapters::Mysql2Adapter::ER_UNKNOWN_HOST_ERROR].include?(error.error_number) raise ActiveRecord::DatabaseConnectionError.hostname_error(config[:host]) else raise ActiveRecord::ConnectionNotEstablished, error.message end end end
  9. 接続要求 module ConnectionAdapters # = Active Record MySQL2 Adapter class

    Mysql2Adapter < AbstractMysqlAdapter …… class << self def new_client(config) ::Mysql2::Client.new(config) rescue ::Mysql2::Error => error if error.error_number == ConnectionAdapters::Mysql2Adapter::ER_BAD_DB_ERROR raise ActiveRecord::NoDatabaseError.db_error(config[:database]) elsif error.error_number == ConnectionAdapters::Mysql2Adapter::ER_ACCESS_DENIED_ERROR raise ActiveRecord::DatabaseConnectionError.username_error(config[:username]) elsif [ConnectionAdapters::Mysql2Adapter::ER_CONN_HOST_ERROR, ConnectionAdapters::Mysql2Adapter::ER_UNKNOWN_HOST_ERROR].include?(error.error_number) raise ActiveRecord::DatabaseConnectionError.hostname_error(config[:host]) else raise ActiveRecord::ConnectionNotEstablished, error.message end end end mysql2のクライアントインスタンス作成 (config にはdatabase.ymlの情報)
  10. 接続要求 def initialize(opts = {}) raise Mysql2::Error, "Options parameter must

    be a Hash" unless opts.is_a? Hash opts = Mysql2::Util.key_hash_as_symbols(opts) ... user = opts[:username] || opts[:user] pass = opts[:password] || opts[:pass] host = opts[:host] || opts[:hostname] port = opts[:port] database = opts[:database] || opts[:dbname] || opts[:db] socket = opts[:socket] || opts[:sock] conn_attrs = parse_connect_attrs(opts[:connect_attrs]) ... connect user, pass, host, port, database, socket, flags, conn_attrs end mysq2/client.rb
  11. 接続要求 def initialize(opts = {}) raise Mysql2::Error, "Options parameter must

    be a Hash" unless opts.is_a? Hash opts = Mysql2::Util.key_hash_as_symbols(opts) ... user = opts[:username] || opts[:user] pass = opts[:password] || opts[:pass] host = opts[:host] || opts[:hostname] port = opts[:port] database = opts[:database] || opts[:dbname] || opts[:db] socket = opts[:socket] || opts[:sock] conn_attrs = parse_connect_attrs(opts[:connect_attrs]) ... connect user, pass, host, port, database, socket, flags, conn_attrs end mysq2/client.rb initializeメソッドで database.yml の情報をもとに接続要求
  12. mysql2 Application DB adapter Mysql2::Client Mysql2::Client Mysql2::Client Mysql2Adapter Mysql2Adapter Mysql2Adapter

    Mysql2::Client オブジェクトの作成 C APIで接続要求 database.yml
  13. 接続作成 module ActiveRecord module ConnectionAdapters ... ... class ConnectionPool ...

    def new_connection connection = Base.public_send(db_config.adapter_method, db_config.configuration_hash) connection.pool = self connection.check_version connection end end end
  14. 接続作成 module ActiveRecord module ConnectionAdapters ... ... class ConnectionPool ...

    def new_connection connection = Base.public_send(db_config.adapter_method, db_config.configuration_hash) connection.pool = self connection.check_version connection end end end database.ymlのadapterに記載されて るアダプターに対応する接続を作成 今回の場合はMysql2Adapter作成
  15. 接続作成 module ActiveRecord module ConnectionAdapters ... ... class ConnectionPool ...

    def new_connection connection = Base.public_send(db_config.adapter_method, db_config.configuration_hash) connection.pool = self connection.check_version connection end end end データベースのバージョンを確認する このタイミングで接続要求をする (先ほどのMysql2::Client作成に繋がる)
  16. 接続プールからの接続取得 module ActiveRecord module ConnectionAdapters ... ... class ConnectionPool ...

    def checkout(checkout_timeout = @checkout_timeout) connection = checkout_and_verify(acquire_connection(checkout_timeout)) connection.lock_thread = @lock_thread connection end end end
  17. module ActiveRecord module ConnectionAdapters ... ... class ConnectionPool ... def

    checkout(checkout_timeout = @checkout_timeout) connection = checkout_and_verify(acquire_connection(checkout_timeout)) connection.lock_thread = @lock_thread connection end end end 接続プールから接続を取り出す または新規接続を作成する 接続プールからの接続取得
  18. module ActiveRecord module ConnectionAdapters ... ... class ConnectionPool ... def

    acquire_connection(checkout_timeout)     if conn = @available.poll || try_to_checkout_new_connection       conn     else       reap       @available.poll(checkout_timeout)     end   end end end 接続プールからの接続取得
  19. module ActiveRecord module ConnectionAdapters ... ... class ConnectionPool ... def

    acquire_connection(checkout_timeout)     if conn = @available.poll || try_to_checkout_new_connection       conn     else       reap       @available.poll(checkout_timeout)     end   end end end まず接続キュー(プール)を確認する 接続がなければ新規接続を試みる (先ほどのnew_connectionメソッドに繋がる) 接続プールからの接続取得
  20. ConnectionPoolの作成 def establish_connection(config, owner_name: Base, role: ActiveRecord::Base.current_role, shard: Base.current_shard) owner_name

    = determine_owner_name(owner_name, config) pool_config = resolve_pool_config(config, owner_name, role, shard) db_config = pool_config.db_config pool_manager = set_pool_manager(pool_config.connection_name) existing_pool_config = pool_manager.get_pool_config(role, shard) if existing_pool_config && existing_pool_config.db_config == db_config if owner_name.primary_class? && (existing_pool_config.connection_class != owner_name) existing_pool_config.connection_class = owner_name end existing_pool_config.pool else ・・・    ・・・
  21. def establish_connection(config, owner_name: Base, role: ActiveRecord::Base.current_role, shard: Base.current_shard) owner_name =

    determine_owner_name(owner_name, config) pool_config = resolve_pool_config(config, owner_name, role, shard) db_config = pool_config.db_config pool_manager = set_pool_manager(pool_config.connection_name) existing_pool_config = pool_manager.get_pool_config(role, shard) if existing_pool_config && existing_pool_config.db_config == db_config if owner_name.primary_class? && (existing_pool_config.connection_class != owner_name) existing_pool_config.connection_class = owner_name end existing_pool_config.pool else ・・・    ・・・ database.ymlに記載のどのDBに接続するか を表す抽象クラス名を取得 ConnectionPoolの作成
  22. owner_nameとは primary:   database: my_primary_database   username: app   password: <%= ENV['ROOT_PASSWORD'] %>

      adapter: mysql2 world:   database: world   username: app   password: <%= ENV['WORLDS_ROOT_PASSWORD'] %>   adapter: mysql2 database.yml
  23. primary:   database: my_primary_database   username: app   password: <%= ENV['ROOT_PASSWORD'] %>   adapter:

    mysql2 world:   database: world   username: app   password: <%= ENV['WORLDS_ROOT_PASSWORD'] %>   adapter: mysql2 class WorldRecord < ApplicationRecord   self.abstract_class = true   connects_to database: { writing: :world } end class ApplicationRecord < ActiveRecord::Base self.abstract_class = true connects_to database: { writing: :primary} end ↑primaryのDBに繋ぐように定義したコネクションモデル ↑worldのDBに繋ぐように定義したコネクションモデル owner_nameとは database.yml
  24. primary:   database: my_primary_database   username: app   password: <%= ENV['ROOT_PASSWORD'] %>   adapter:

    mysql2 world:   database: world   username: app   password: <%= ENV['WORLDS_ROOT_PASSWORD'] %>   adapter: mysql2 class WorldRecord < ApplicationRecord   self.abstract_class = true   connects_to database: { writing: :world } end class ApplicationRecord < ActiveRecord::Base self.abstract_class = true connects_to database: { writing: :primary} end owner_name owner_nameとは database.yml
  25. ConnectionPoolの作成 def establish_connection(config, owner_name: Base, role: ActiveRecord::Base.current_role, shard: Base.current_shard) owner_name

    = determine_owner_name(owner_name, config) pool_config = resolve_pool_config(config, owner_name, role, shard) db_config = pool_config.db_config pool_manager = set_pool_manager(pool_config.connection_name) existing_pool_config = pool_manager.get_pool_config(role, shard) if existing_pool_config && existing_pool_config.db_config == db_config if owner_name.primary_class? && (existing_pool_config.connection_class != owner_name) existing_pool_config.connection_class = owner_name end existing_pool_config.pool else ・・・    ・・・
  26. def establish_connection(config, owner_name: Base, role: ActiveRecord::Base.current_role, shard: Base.current_shard) owner_name =

    determine_owner_name(owner_name, config) pool_config = resolve_pool_config(config, owner_name, role, shard) db_config = pool_config.db_config pool_manager = set_pool_manager(pool_config.connection_name) existing_pool_config = pool_manager.get_pool_config(role, shard) if existing_pool_config && existing_pool_config.db_config == db_config if owner_name.primary_class? && (existing_pool_config.connection_class != owner_name) existing_pool_config.connection_class = owner_name end existing_pool_config.pool else ・・・    ・・・ PoolConfigという接続設定クラス作成 それを管理するPoolManagerクラスを作成もしくは既存の再利用 PoolManagerはowner_nameで一意となる ConnectionPoolの作成
  27. ConnectionPoolの作成 def establish_connection(config, owner_name: Base, role: ActiveRecord::Base.current_role, shard: Base.current_shard) owner_name

    = determine_owner_name(owner_name, config) pool_config = resolve_pool_config(config, owner_name, role, shard) db_config = pool_config.db_config pool_manager = set_pool_manager(pool_config.connection_name) existing_pool_config = pool_manager.get_pool_config(role, shard) if existing_pool_config && existing_pool_config.db_config == db_config if owner_name.primary_class? && (existing_pool_config.connection_class != owner_name) existing_pool_config.connection_class = owner_name end existing_pool_config.pool else ・・・    ・・・
  28. def establish_connection(config, owner_name: Base, role: ActiveRecord::Base.current_role, shard: Base.current_shard) owner_name =

    determine_owner_name(owner_name, config) pool_config = resolve_pool_config(config, owner_name, role, shard) db_config = pool_config.db_config pool_manager = set_pool_manager(pool_config.connection_name) existing_pool_config = pool_manager.get_pool_config(role, shard) if existing_pool_config && existing_pool_config.db_config == db_config if owner_name.primary_class? && (existing_pool_config.connection_class != owner_name) existing_pool_config.connection_class = owner_name end existing_pool_config.pool else ・・・    ・・・ 既存のPoolConfigがないか確認 PoolConfigはroleとshardで一意になる ConnectionPoolの作成
  29. world:   database: world   username: app   password: <%= ENV['ANIMALS_ROOT_PASSWORD'] %>   adapter:

    mysql2   host: hogehoge world_replica: database: world username: app_readonly password: <%= ENV['ANIMALS_READONLY_PASSWORD'] %> adapter: mysql2 host: hugahuga replica: true roleとは database.yml
  30. class WorldRecord < ApplicationRecord   self.abstract_class = true   connects_to database: {

    writing: :world, reading: :world_replica } end ↑ ライターとリーダーで別々のDBに接続するように 定義したコネクションモデル world:   database: world   username: app   password: <%= ENV['ANIMALS_ROOT_PASSWORD'] %>   adapter: mysql2   host: hogehoge world_replica: database: world username: app_readonly password: <%= ENV['ANIMALS_READONLY_PASSWORD'] %> adapter: mysql2 host: hugahuga replica: true roleとは database.yml
  31. roleとは world:   database: world   username: app   password: <%= ENV['ANIMALS_ROOT_PASSWORD'] %>

      adapter: mysql2   host: hogehoge world_replica: database: world username: app_readonly password: <%= ENV['ANIMALS_READONLY_PASSWORD'] %> adapter: mysql2 host: hugahuga replica: true class WorldRecord < ApplicationRecord   self.abstract_class = true   connects_to database: { writing: :world, reading: :world_replica } end role database.yml
  32. shardとは primary: database: my_primary_database adapter: mysql2 primary_replica: database: my_primary_database adapter:

    mysql2 replica: true primary_shard_one: database: my_primary_shard_one adapter: mysql2 primary_shard_one_replica: database: my_primary_shard_one adapter: mysql2 replica: true database.yml
  33. class ApplicationRecord < ActiveRecord::Base self.abstract_class = true connects_to shards: {

    default: { writing: :primary, reading: :primary_replica }, shard_one: { writing: :primary_shard_one, reading: :primary_shard_one_replica } } end primary: database: my_primary_database adapter: mysql2 primary_replica: database: my_primary_database adapter: mysql2 replica: true primary_shard_one: database: my_primary_shard_one adapter: mysql2 primary_shard_one_replica: database: my_primary_shard_one adapter: mysql2 replica: true ↑ シャードごとに接続先DBを定義したコネクションモデル shardとは database.yml
  34. primary: database: my_primary_database adapter: mysql2 primary_replica: database: my_primary_database adapter: mysql2

    replica: true primary_shard_one: database: my_primary_shard_one adapter: mysql2 primary_shard_one_replica: database: my_primary_shard_one adapter: mysql2 replica: true class ApplicationRecord < ActiveRecord::Base self.abstract_class = true connects_to shards: { default: { writing: :primary, reading: :primary_replica }, shard_one: { writing: :primary_shard_one, reading: :primary_shard_one_replica } } end shard shardとは database.yml
  35. Application ConnectionHandler DB PoolManager PoolManager PoolConfig PoolConfig PoolConfig PoolConfig role:

    writer role, shard ごとに PoolConfigを作成 role: reader role: reader role: writer
  36. ConnectionPoolの作成 def establish_connection(config, owner_name: Base, role: ActiveRecord::Base.current_role, shard: Base.current_shard) owner_name

    = determine_owner_name(owner_name, config) pool_config = resolve_pool_config(config, owner_name, role, shard) db_config = pool_config.db_config pool_manager = set_pool_manager(pool_config.connection_name) existing_pool_config = pool_manager.get_pool_config(role, shard) if existing_pool_config && existing_pool_config.db_config == db_config if owner_name.primary_class? && (existing_pool_config.connection_class != owner_name) existing_pool_config.connection_class = owner_name end existing_pool_config.pool else ・・・    ・・・
  37. def establish_connection(config, owner_name: Base, role: ActiveRecord::Base.current_role, shard: Base.current_shard) owner_name =

    determine_owner_name(owner_name, config) pool_config = resolve_pool_config(config, owner_name, role, shard) db_config = pool_config.db_config pool_manager = set_pool_manager(pool_config.connection_name) existing_pool_config = pool_manager.get_pool_config(role, shard) if existing_pool_config && existing_pool_config.db_config == db_config if owner_name.primary_class? && (existing_pool_config.connection_class != owner_name) existing_pool_config.connection_class = owner_name end existing_pool_config.pool else ・・・    ・・・ 新規もしくは既存のPoolConfig に基づいたConnectionPool作成 ConnectionPoolの作成
  38. Application ConnectionPool ConnectionHandler DB PoolManager PoolManager PoolConfig PoolConfig PoolConfig PoolConfig

    ConnectionPool ConnectionPool ConnectionPool owner ごとに PoolManagerを作成 role, shard ごとに PoolConfigを作成 ConnectionPool作成
  39. まとめ Copyright © 2020 Present ANDPAD Inc. This information is

    confidential and was prepared by ANDPAD Inc. for the use of our client. It is not to be relied on by and 3rd party. Proprietary & Confidential 無断転載・無断複製の禁止
  40. 全体図(再掲) Active Record mysql2 Application DB C API (libmysqlclient) ORM

    adapter mysql_real_connect(..) • イニシャルハンド シェイクパケット の送信 • 認証方式の決定 • 決定した方式で認 証 • OKパケットかERR パケットの送信
  41. Application ConnectionPool ConnectionHandler DB PoolManager PoolManager PoolConfig PoolConfig PoolConfig PoolConfig

    ConnectionPool ConnectionPool ConnectionPool owner ごとに PoolManagerを作成 role, shard ごとに PoolConfigを作成 ConnectionPool作成
  42. 知見その1(エラー原因の切り分け方法) • エラーメッセージをパッと見ても原因箇所の想像がつかない場合があった ◦ 例:ActiveRecord::ConnectionNotEstablished: No connection pool with hogehoge

    for the hugahuga role • 注目すべきポイントがどこなのか原因を切り分けることができれば、問題 解決の難しさが軽減される • Mysql2Adapterのnew_clientメソッド時点で接続確立を行うので、バック トレースを見て、それ以前・その時点・その以後かどうかで原因の切り分 けができる
  43. 知見その1(エラー原因の切り分け) • new_clientメソッド以前の時点でのエラーは 「接続確立する前のアプリケーション側の処理」の段階 ◦ role/shardの記載が適切でない ◦ 接続の新規作成or再利用の待ち時間超過 • new_clientメソッド時点のエラーは「接続確立する」段階

    ◦ host/user/passwordの記載が適切でない ◦ MySQL側の接続上限に達する • new_clientメソッド以後の時点でのエラーは 「接続確立後のMySQLとのやりとり」の段階 ◦ ロック待ち時間の超過 ◦ デッドロック発生 接続前 接続後 接続時
  44. 知見その2(DBホスト変更の注意点) • Mysql2Adaptersのクライアントインスタンスはdatabase.ymlの 情報を保持している • 接続プールで保持しているのはこのMysql2Adaptersのクライア ントインスタンスである • DBホストの変更を行う際は、プールで保持されている変更以前 のホストの接続が利用される可能性がある

    • 特にフェイルオーバーした時は、降格したリーダーに更新クエリ が投げられることが起こりかねない • プールから接続を得る際にバリデーションクエリ(read_onlyの確 認など)を流す方法があったりする https://qiita.com/hmatsu47/items/605f7edaf390d52ec828
  45. ご清聴ありがとうございました Copyright © 2020 Present ANDPAD Inc. This information is

    confidential and was prepared by ANDPAD Inc. for the use of our client. It is not to be relied on by and 3rd party. Proprietary & Confidential 無断転載・無断複製の禁止
  46. Appendix Copyright © 2020 Present ANDPAD Inc. This information is

    confidential and was prepared by ANDPAD Inc. for the use of our client. It is not to be relied on by and 3rd party. Proprietary & Confidential 無断転載・無断複製の禁止