Sharding para Rails apps

Palestra apresentada na RubyConf Brasil 2016.

Transcript

1. Sharding para Rails apps github.com/lenon linkedin.com/in/lenonmarcel Lenon Marcel
2. • Problemas enfrentados com uma arquitetura single-tenant • Como migramos para uma arquitetura multi-tenant • Problemas de escalabilidade • Estratégias de sharding • Implementando sharding no Rails Agenda
3. Email Marketing Locaweb em 2013
4. Email Marketing Locaweb (~2013) models workers web bounces opt-in mongo db Rails engine Rails apps Main database Sidekiq
5. Problemas Tempo de resposta muito alto Downtimes constantes Difícil de escalar, apesar do hardware caro Clientes insatisfeitos
6. Raízes dos Problemas Dado relacional em um banco de documentos Coleções enormes e muitos relacionamentos Arquitetura multi-tenant fazia mais sentido
7.  PostgreSQL
8. PostgreSQL Suporta características de bancos NoSQL ACID no nível de database Know-how na equipe
9.  PostgreSQL Schemas
10. PostgreSQL Schemas database accounts senders contacts messages várias tabelas
11. database PostgreSQL Schemas public accounts senders contacts messages default postgresql schema
12. PostgreSQL Schemas Um schema é uma coleção de tabelas e funciona como um namespace. https://www.postgresql.org/docs/current/static/ddl-schemas.html
13. create schema schema1; create table schema1.products (name text); insert into schema1.products (name) values ('product on schema 1'); create schema schema2; create table schema2.products (name text); insert into schema2.products (name) values ('product on schema 2'); PostgreSQL Schemas
14. select * from schema1.products; name --------------------- product on schema 1 (1 row) select * from schema2.products; name --------------------- product on schema 2 (1 row) PostgreSQL Schemas
15. create schema schema1; set search_path = schema1; create table products (name text); insert into products (name) values ('product on schema 1'); create schema schema2; set search_path = schema2; create table products (name text); insert into products (name) values ('product on schema 2'); PostgreSQL Schemas
16. set search_path = schema1; select * from products; name --------------------- product on schema 1 (1 row) set search_path = schema2; select * from products; name --------------------- product on schema 2 (1 row) PostgreSQL Schemas
17. PostgreSQL Schemas (benefícios) Dados ficam isolados Ajudam na migração do MongoDB Particionamento lógico
18. PostgreSQL Schemas database public accounts domains senders account_1 contacts messages reports account_2 contacts messages reports account_3 contacts messages reports account_n contacts messages reports
19. Ǹ ActiveRecord & Schemas
20. puts ActiveRecord::Base.connection.schema_search_path #=> "$user", public ActiveRecord & Schemas
21. ApplicationRecord.connection.schema_search_path = 'schema1' Product.create(name: 'product on schema1') Product.all.pluck(:name) #=> ["product on schema1"] ApplicationRecord.connection.schema_search_path = 'schema2' Product.create(name: 'product on schema2') Product.all.pluck(:name) #=> ["product on schema2"] ActiveRecord & Schemas
22. class ApplicationRecord < ActiveRecord::Base self.abstract_class = true def self.using(schema_name) previous_search_path = connection.schema_search_path connection.schema_search_path = schema_name yield ensure connection.schema_search_path = previous_search_path end end ActiveRecord & Schemas
23. ApplicationRecord.using('schema1') do Product.pluck(:name) end #=> ["product on schema1"] ApplicationRecord.using('schema2') do Product.pluck(:name) end #=> ["product on schema2"] ActiveRecord & Schemas
24. class ApplicationController < ActionController::Base around_action :set_user_schema def set_user_schema ApplicationRecord.using(current_user.schema_name) do yield end end end ActiveRecord & Schemas
25. Resultados
26. Resultados Baixo tempo de resposta Fim dos downtimes Clientes satisfeitos
27. Problemas Banco chegando nos terabytes de dados Ponto único de falha
28.  Sharding
29. Antes de fazer Sharding + memória + cpu + disco
30. Antes de fazer Sharding Sharding adiciona mais complexidade
31. Sharding large db small db 1 small db 2 small db 3 small db 4
32. Benefícios com Sharding Sharding mantém falhas isoladas Databases menores são mais rápidas Sharding permite escalar linearmente
33. Estratégias de Sharding Via particionamento de tabelas ? Através de uma coluna da tabela (key based) ? Através de um serviço de "lookup" ?
34. database accounts senders contacts messages Sharding via particionamento de tabelas database 2 messages database 3 contacts
35. # config/database.yml default: &default adapter: postgresql encoding: unicode development: # [...] other_db: <<: *default database: other_db Sharding via particionamento de tabelas
36. class Contact < ActiveRecord::Base establish_connection :other_db end Sharding via particionamento de tabelas
37. contacts name email account_id created_at Sharding através de uma coluna da tabela Shard A Shard B Shard C 0 ≤ account_id < 10 10 ≤ account_id < 20 20 ≤ account_id < 30
38. Sharding através de uma coluna da tabela MySQL, MongoDB, etc suportam PostgreSQL com pg_shard/citus Transparente para a app
39. Requer código no lado da app Sharding através de um lookup service Funciona através de um mapeamento usuário x shard ė Altamente escalável e permite migração entre os shards
40. Sharding através de um lookup service Migrar dados novamente não seria viável PostgreSQL tem suporte limitado para escala horizontal pg_shard/citus não suporta shard por schema
41. Ǹ Sharding no ActiveRecord
43. Sharding for ActiveRecord github.com/thiagopradi/octopus github.com/kovyrin/db-charmer github.com/fiveruns/data_fabric
44. Sharding for ActiveRecord github.com/thiagopradi/octopus github.com/kovyrin/db-charmer github.com/fiveruns/data_fabric ¯\_(ϑ)_/¯
45. github.com/thiagopradi/octopus Suporta replicação (master/slave) Suporta sharding (mesmo model com diferentes DBs) Suporta replicação + sharding
46. github.com/thiagopradi/octopus Contact.where(email: 'foo@example.com').using(:shard1) Octopus.using(:shard1) do Contact.where(email: 'foo@example.com') end
47. Replicação master/slave Message.find(123) slave Message.create(subject: 'Hello world', content: 'Test message') master class Message < ActiveRecord::Base replicated_model end
48. github.com/thiagopradi/octopus Várias issues abertas Problemas com as versões mais recentes do Rails
49. Ǹ Conexões no ActiveRecord
50. Conexões no ActiveRecord O Rails e outras libs assumem 1 database apenas É possível 1 modelo conectado em outro database Não é possível um mesmo modelo conectado em vários databases
51. # config/database.yml default: &default adapter: postgresql encoding: unicode development: # [...] other_db: <<: *default database: other_db Conexões no ActiveRecord
52. class Contact < ActiveRecord::Base establish_connection :other_db end Conexões no ActiveRecord
53. Ǹ Connection Pool
54. Connection pool A connection pool synchronizes thread access to a limited number of database connections. http://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/Connection
55. # config/database.yml production: adapter: postgresql encoding: unicode pool: 3 Connection pool
56. Connection pool connection pool connection 1 connection 2 connection 3 thread 1 thread 2
57. Connection pool connection pool connection 2 connection 3 thread 1 thread 2 request 1 connection 1
58. Connection pool connection pool connection 3 thread 1 request 1 connection 1 request 2 thread 2 connection 2
59. Connection pool connection pool connection 3 thread 1 connection 1 request 2 thread 2 connection 2
60. Connection pool connection pool connection 3 thread 1 connection 1 thread 2 connection 2
61. Connection pool connection pool connection 1 connection 2 connection 3 thread 1 thread 2
62. Ǹ Connection Handler no Rails 4.2
63. Connection Handler (Rails 4.2) Contact.find(123) # active_record/connection_handling.rb:90 def connection retrieve_connection end # activerecord/lib/active_record/core.rb:129 self.default_connection_handler = ConnectionAdapters::ConnectionHandler.new mágica + mágica
64. class ConnectionHandler def initialize @owner_to_pool = ThreadSafe::Cache.new(:initial_capacity => 2) do |h,k| h[k] = ThreadSafe::Cache.new(:initial_capacity => 2) end @class_to_pool = ThreadSafe::Cache.new(:initial_capacity => 2) do |h,k| h[k] = ThreadSafe::Cache.new end end def connection_pool_list owner_to_pool.values.compact end def connection_pools ActiveSupport::Deprecation.warn(<<-MSG.squish) In the next release, this will return the same as `#connection_pool_list`. (An array of pools, rather than a hash mapping specs to pools.) MSG Hash[connection_pool_list.map { |pool| [pool.spec, pool] }] end def establish_connection(owner, spec) @class_to_pool.clear raise RuntimeError, "Anonymous class is not allowed." unless owner.name owner_to_pool[owner.name] = ConnectionAdapters::ConnectionPool.new(spec) end def active_connections? connection_pool_list.any?(&:active_connection?) end def clear_active_connections! connection_pool_list.each(&:release_connection) end def clear_reloadable_connections! connection_pool_list.each(&:clear_reloadable_connections!) end def clear_all_connections! connection_pool_list.each(&:disconnect!) end def retrieve_connection(klass) pool = retrieve_connection_pool(klass) raise ConnectionNotEstablished, "No connection pool for #{klass}" unless pool conn = pool.connection raise ConnectionNotEstablished, "No connection for #{klass} in connection pool" unless conn conn end def connected?(klass) conn = retrieve_connection_pool(klass) conn && conn.connected? end def remove_connection(owner) if pool = owner_to_pool.delete(owner.name) @class_to_pool.clear pool.automatic_reconnect = false pool.disconnect! pool.spec.config end end def retrieve_connection_pool(klass) class_to_pool[klass.name] ||= begin until pool = pool_for(klass) klass = klass.superclass break unless klass <= Base end class_to_pool[klass.name] = pool end end private def owner_to_pool @owner_to_pool[Process.pid] end def class_to_pool @class_to_pool[Process.pid] end def pool_for(owner) owner_to_pool.fetch(owner.name) { if ancestor_pool = pool_from_any_process_for(owner) establish_connection owner, ancestor_pool.spec else owner_to_pool[owner.name] = nil end } end def pool_from_any_process_for(owner) owner_to_pool = @owner_to_pool.values.find { |v| v[owner.name] } owner_to_pool && owner_to_pool[owner.name] end end Connection Handler (Rails 4.2)
65. def retrieve_connection(klass) pool = retrieve_connection_pool(klass) raise ConnectionNotEstablished unless pool conn = pool.connection raise ConnectionNotEstablished unless conn conn end Connection Handler (Rails 4.2)
66. def retrieve_connection_pool(klass) class_to_pool[klass.name] ||= begin until pool = pool_for(klass) klass = klass.superclass break unless klass <= Base end class_to_pool[klass.name] = pool end end Connection Handler (Rails 4.2)
67. class Author < ActiveRecord::Base; end class BankAccount < ActiveRecord::Base; end class Book < ActiveRecord::Base establish_connection 'library_db' end Connection Handler (Rails 4.2) connection pool do ActiveRecord::Base connection pool da classe Book
68. Ǹ Connection Switcher
69. Connection Switcher connection pool connection 1 connection 2 connection 3 thread 1 thread 2
70. pool shard 1 connection 1 connection 2 connection 3 Connection Switcher connection pool connection 1 connection 2 connection 3 thread n pool shard 2 connection 1 connection 2 connection 3 master shards
71. class Handler def setup @cache = {} database_configs.each do |name, _| name = name.to_sym handler = ConnectionHandler.new handler.establish_connection(self, spec(name)) @cache[name] = handler end end def connection_handler_for(name) @cache.fetch(name.to_sym) end end Connection Switcher
72. module ShardHandler def using_shard(shard) old_shard = ThreadRegistry.current_shard ThreadRegistry.current_shard = shard yield ensure clear_active_connections! ThreadRegistry.current_shard = old_shard end def connection_handler return super if use_master_connection? handler.connection_handler_for(current_shard) end end Connection Switcher
73. class ShardedModel < ActiveRecord::Base self.abstract_class = true extend ShardHandler end class Contact < ShardedModel end class Message < ShardedModel end Connection Switcher
74. troca do schema account = Account.find(123) ShardedModel.using_shard(account.shard_name) do ApplicationRecord.using(account.schema_name) do Contact.find_by(email: 'foo@example.com') end end request identificação da conta troca do database execução da ação restaura p/ schema anterior restaura p/ database anterior resposta
75. Ǹ Rails Tasks
76. desc "Migrate the database" task :migrate => [:environment, :load_config] do using('master', 'public') do ActiveRecord::Migrator.migrate('db/public') end Account.all.each do |acc| using(acc.shard_name, acc.schema_name) do ActiveRecord::Migrator.migrate('db/accounts') end end end Rails Tasks
77. Ǹ Connection Handler no Rails 5
78. Connection Handler (Rails 5)
79. class ConnectionHandler def initialize @owner_to_pool = Concurrent::Map.new(:initial_capacity => 2) do |h,k| h[k] = Concurrent::Map.new(:initial_capacity => 2) end end def connection_pool_list owner_to_pool.values.compact end alias :connection_pools :connection_pool_list def establish_connection(spec) owner_to_pool[spec.name] = ConnectionAdapters::ConnectionPool.new(spec) end def active_connections? connection_pool_list.any?(&:active_connection?) end def clear_active_connections! connection_pool_list.each(&:release_connection) end def clear_reloadable_connections! connection_pool_list.each(&:clear_reloadable_connections!) end def clear_all_connections! connection_pool_list.each(&:disconnect!) end def retrieve_connection(spec_name) pool = retrieve_connection_pool(spec_name) raise ConnectionNotEstablished, "No connection pool with id #{spec_name} found." unless pool conn = pool.connection raise ConnectionNotEstablished, "No connection for #{spec_name} in connection pool" unless conn conn end def connected?(spec_name) conn = retrieve_connection_pool(spec_name) conn && conn.connected? end def remove_connection(spec_name) if pool = owner_to_pool.delete(spec_name) pool.automatic_reconnect = false pool.disconnect! pool.spec.config end end def retrieve_connection_pool(spec_name) owner_to_pool.fetch(spec_name) do if ancestor_pool = pool_from_any_process_for(spec_name) establish_connection(ancestor_pool.spec).tap do |pool| pool.schema_cache = ancestor_pool.schema_cache if ancestor_pool.schema_cache end else owner_to_pool[spec_name] = nil end end end private def owner_to_pool @owner_to_pool[Process.pid] end def pool_from_any_process_for(spec_name) owner_to_pool = @owner_to_pool.values.find { |v| v[spec_name] } owner_to_pool && owner_to_pool[spec_name] end end Connection Handler (Rails 5)
80. def retrieve_connection(spec_name) pool = retrieve_connection_pool(spec_name) raise ConnectionNotEstablished unless pool conn = pool.connection raise ConnectionNotEstablished unless conn conn end Connection Handler (Rails 5)
81. def retrieve_connection_pool(spec_name) owner_to_pool.fetch(spec_name) do if ancestor_pool = pool_from_any_process_for(spec_name) establish_connection(ancestor_pool.spec).tap do |pool| pool.schema_cache = ancestor_pool.schema_cache if ancestor_pool.schema_cache end else owner_to_pool[spec_name] = nil end end end Connection Handler (Rails 5)
82. class Contact < ActiveRecord::Base def self.connection_specification_name # alguma specification name declarada no arquivo # config/database.yml end end Connection Handler (Rails 5)
83. Arquitetura atual
84. Arquitetura atual core workers web bounces opt-in shard A Rails engine Rails apps Sidekiq master shard B PostgreSQL shard C
85. Arquitetura atual Em produção há mais de 1 ano Escalabilidade linear
86. Arquitetura atual github.com/locaweb/multitenancy_tools github.com/locaweb/shard_handler
87. Ŏ Obrigado! Perguntas?