Slide 1
Slide 1 text
Data Mapping
The Rails Way
Дмитрий Цепелев, Злые марсиане
фото: Bryan Birdwell, https://bit.ly/2Tpy3mU
Slide 2
Slide 2 text
DmitryTsepelev
@dmitrytsepelev
RUBY DAY 2020 2
Slide 3
Slide 3 text
DmitryTsepelev
@dmitrytsepelev
RUBY DAY 2020 3
evilmartians.com
Slide 4
Slide 4 text
DmitryTsepelev
@dmitrytsepelev
RUBY DAY 2020 4
evilmartians.com
Slide 5
Slide 5 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 5
Data mapping — процесс преобразования данных от одного
формата к другому
Slide 6
Slide 6 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 6
Преобразование типов
SELECT "created_at" FROM "users" LIMIT 1
┌──────────────────────────┐
│ created_at │
├──────────────────────────┤
│ 2019-11-22 09:29:17.5634 │
└──────────────────────────┘
User.first.created_at
# "=> Fri, 22 Nov 2019 12:29:17 MSK +03:00
Slide 7
Slide 7 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 7
Построение запросов
# date
Order.where("created_at ">= ?", Date.today.beginning_of_day)
#"=> select * from orders where created_at ">= '20-02-06'
# enum
Order.where(status: :complete)
#"=> select * from orders where status = 3
Slide 8
Slide 8 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev
Data Mapping в Rails
Зачем разработчику знать про Attributes API
JSON–колонки как модели
8
Slide 9
Slide 9 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 9
Атрибуты модели
User.attribute_types
# "=> {"id""=>#,
# "created_at""=>#,
# "updated_at""=>#,
# ""...
# }
Slide 10
Slide 10 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 10
Автоматическое определение атрибутов
module ActiveRecord
module ModelSchema
def load_schema!
@columns_hash = connection.schema_cache
.columns_hash(table_name)
.except(*ignored_columns)
@columns_hash.each do |name, column|
define_attribute(
name,
connection.lookup_cast_type_from_column(column)
)
end
end
end
end
Slide 11
Slide 11 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 11
ignored_columns
class Post < ActiveRecord"::Base
self.ignored_columns = %w(meta)
end
Slide 12
Slide 12 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 12
Определение атрибутов вручную
class Configuration
include ActiveModel"::Attributes
attribute :model, :string
attribute :color, :string, default: "red"
end
config = Configuration.new
config.model = 42
puts config.model.inspect # "=> "42"
puts config.color.inspect # "=> "red"
Slide 13
Slide 13 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 13
Что будет?
class TestModel
include ActiveModel"::Model
include ActiveModel"::Attributes
attribute :list, default: []
end
model1 = TestModel.new
model1.list "<< 1
model2 = TestModel.new
model2.list "<< 2
model2.list "== [2] # "=> false
Slide 14
Slide 14 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 14
Будет ерунда
class TestModel
include ActiveModel"::Model
include ActiveModel"::Attributes
attribute :list, default: []
end
model1 = TestModel.new
model1.list "<< 1
model2 = TestModel.new
model2.list "<< 2
model2.list # "=> [1, 2]
Slide 15
Slide 15 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 15
Нужно использовать lambda
class TestModel
include ActiveModel"::Model
include ActiveModel"::Attributes
attribute :list, default: "-> { [] }
end
model1 = TestModel.new
model1.list "<< 1
model2 = TestModel.new
model2.list "<< 2
model2.list # "=> [2]
Slide 16
Slide 16 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 16
ActiveModel"::Type"::BigInteger
ActiveModel"::Type"::Binary
ActiveModel"::Type"::Boolean
ActiveModel"::Type"::DateTime
ActiveModel"::Type"::Date
ActiveModel"::Type"::Decimal
ActiveModel"::Type"::"Float
ActiveModel"::Type"::ImmutableString
ActiveModel"::Type"::Integer
ActiveModel"::Type"::String
ActiveModel"::Type"::Time
Типы «из коробки»
ActiveRecord"::Type"::DateTime
ActiveRecord"::Type"::Date
ActiveRecord"::Type"::DecimalWithoutScale
ActiveRecord"::Type"::Json
ActiveRecord"::Type"::Time
ActiveRecord"::Type"::UnsignedInteger
Slide 17
Slide 17 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 17
class ActiveModel"::Type"::Value
# DB "-> Ruby
def deserialize(value)
cast(value)
end
# User "-> Ruby
def cast(value)
cast_value(value) unless value.nil?
end
# Ruby "-> DB
def serialize(value)
value
end
end
Что внутри?
Slide 18
Slide 18 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 18
Реестр типов
ActiveModel::Type::Registry или наследник
в ActiveRecord используется AdapterSpecificRegistry
умеет находить класс типа по имени
ActiveRecord"::Type.registry.lookup(:integer)
# "=> #
Slide 19
Slide 19 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 19
Создание собственного типа
class MoneyType < ActiveRecord"::Type"::Integer
def cast(value)
if !value.kind_of?(Numeric) "&& value.include?('$')
price_in_dollars = value.gsub(/\$/, '').to_f
super(price_in_dollars * 70)
else
super
end
end
end
Slide 20
Slide 20 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 20
Использование собственного типа
ActiveRecord"::Type.register(
:money,
MoneyType
)
class User < ApplicationRecord
attribute :balance, :money
end
User.new(balance: '$1').balance
# "=> 70
class User < ApplicationRecord
attribute :balance, MoneyType.new
end
User.new(balance: '$1').balance
# "=> 70
через реестр напрямую
Slide 21
Slide 21 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 21
⚠ Type cast происходит при чтении ⚠
class IntegerRangeType < ActiveRecord"::Type"::Integer
def initialize(range)
@range = range
end
def cast(value)
raise ArgumentError unless range.cover?(value)
super
end
end
class User
include ActiveModel"::Attributes
attribute :age, IntegerRangeType.new(0"..100)
end
user = User.new
user.age = 120
puts user.age # "=> ArgumentError
Slide 22
Slide 22 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev
Зачем разработчику знать
про Attributes API
22
Data Mapping в Rails
Зачем разработчику знать про Attributes API
JSON–колонки как модели
Slide 23
Slide 23 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 23
Виртуальные атрибуты
class User < ApplicationRecord
attribute :skip_confirmation, :boolean, default: false
validate :confirmation, unless: :skip_confirmation?
end
user = User.new(skip_confirmation: '0')
Slide 24
Slide 24 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 24
Битовые операции
IN_APP = 0
EMAIL = 1
PUSH = 2
settings = 0
0 0 0
Slide 25
Slide 25 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 25
Битовые операции
IN_APP = 0
EMAIL = 1
PUSH = 2
settings = 0
settings "|= (1 "<< EMAIL)
0 0 0
0 1 0
OR
Slide 26
Slide 26 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 26
Битовые операции
IN_APP = 0
EMAIL = 1
PUSH = 2
settings = 0
settings "|= (1 "<< EMAIL) # "=> 2
settings.to_s(2) # "=> "10"
0 1 0
Slide 27
Slide 27 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 27
Битовые операции
IN_APP = 0
EMAIL = 1
PUSH = 2
settings = 0
settings "|= (1 "<< EMAIL) # "=> 2
settings.to_s(2) # "=> "10"
settings & (1 "<< EMAIL) > 0
0 1 0
0 1 0
AND
Slide 28
Slide 28 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 28
Битовые операции
IN_APP = 0
EMAIL = 1
PUSH = 2
settings = 0
settings "|= (1 "<< EMAIL) # "=> 2
settings.to_s(2) # "=> "10"
settings & (1 "<< EMAIL) > 0 # "=> true
0 1 0
0 1 0
AND
Slide 29
Slide 29 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 29
Битовые операции
IN_APP = 0
EMAIL = 1
PUSH = 2
settings = 0
settings "|= (1 "<< EMAIL) # "=> 2
settings.to_s(2) # "=> "10"
settings & (1 "<< EMAIL) > 0 # "=> true
settings & (1 "<< PUSH) > 0
0 1 0
1 0 0
AND
Slide 30
Slide 30 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 30
Битовые операции
IN_APP = 0
EMAIL = 1
PUSH = 2
settings = 0
settings "|= (1 "<< EMAIL) # "=> 2
settings.to_s(2) # "=> "10"
settings & (1 "<< EMAIL) > 0 # "=> true
settings & (1 "<< PUSH) > 0 # "=> false
0 1 0
1 0 0
AND
Slide 31
Slide 31 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 31
Битовые операции
IN_APP = 0
EMAIL = 1
PUSH = 2
settings = 0
settings "|= (1 "<< EMAIL) # "=> 2
settings.to_s(2) # "=> "10"
settings & (1 "<< EMAIL) > 0 # "=> true
settings & (1 "<< PUSH) > 0 # "=> false
settings "|= (1 "<< PUSH) # "=> 6
settings.to_s(2)
0 1 0
1 0 0
OR
Slide 32
Slide 32 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 32
Битовые операции
IN_APP = 0
EMAIL = 1
PUSH = 2
settings = 0
settings "|= (1 "<< EMAIL) # "=> 2
settings.to_s(2) # "=> "10"
settings & (1 "<< EMAIL) > 0 # "=> true
settings & (1 "<< PUSH) > 0 # "=> false
settings "|= (1 "<< PUSH) # "=> 6
settings.to_s(2) # "=> "110"
settings & (1 "<< PUSH) > 0 # "=> true
1 1 0
Slide 33
Slide 33 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 33
Value Object
class NotificationSettings < ActiveRecord"::Type"::Value
IN_APP = 0
EMAIL = 1
PUSH = 2
…
end
Slide 34
Slide 34 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 34
Value Object
class NotificationSettings < ActiveRecord"::Type"::Value
…
def cast(value)
Value.new(value)
end
end
Slide 35
Slide 35 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 35
Value Object
class NotificationSettings < ActiveRecord"::Type"::Value
…
class Value
attr_reader :settings
def initialize(settings)
@settings = settings
end
def subscribe(subscription)
@settings "|= (1 "<< subscription)
end
def receives_email?
@settings & (1 "<< EMAIL) > 0
end
end
end
Slide 36
Slide 36 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 36
Value Object
class User < ActiveRecord+::Base
attribute :notification_settings,
NotificationSettings.new,
default: 0
end
user = User.new
puts user.notification_settings.receives_email? # "=> false
user.notification_settings.subscribe(NotificationSettings"::EMAIL)
puts user.notification_settings.receives_email? # "=> true
Slide 37
Slide 37 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 37
Работа с legacy колонками
class Post < ActiveRecord"::Base
attribute :published_at, StringDateType.new
end
Post.insert_all([{ published_at: "2012-10-17 14:35" }])
puts Post.last.published_at.inspect # "=> 2012-10-17 14:35:00 +0400
Post.create(published_at: Time.now)
puts Post.last.published_at # "=> 2020-02-12 20:15:00 +0300
Slide 38
Slide 38 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 38
Работа с legacy колонками
class StringDateType < ActiveRecord"::Type"::Value
LEGACY_FORMAT = "%Y-%m-%d %H:%M"
def serialize(value)
value.strftime(LEGACY_FORMAT)
end
def deserialize(value)
Time.strptime(value, LEGACY_FORMAT)
end
end
Slide 39
Slide 39 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 39
Работа с JSON колонками
class Post < ActiveRecord"::Base
attribute :meta, PostMetaType.new
end
post = Post.new(meta: { published_at: 1.week.ago })
puts post.meta.published_at # "=> 2020-02-05
post.meta.published_at = Time.now
post.save
puts post.meta.published_at # "=> 2020-02-12
Slide 40
Slide 40 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 40
Работа с JSON колонками
class PostMetaType < ActiveRecord"::Type"::Value
class Meta
include ActiveModel"::Model
include ActiveModel"::Attributes
attribute :published_at, :date
end
def serialize(value)
ActiveSupport"::JSON.encode(value.attributes)
end
def cast(value)
decoded = value.is_a?(String) ? ActiveSupport"::JSON.decode(value) : value
Meta.new(decoded)
end
end
Slide 41
Slide 41 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 41
Вызываем конструктор 50000 раз:
Бенчмарк по памяти
struct
dry-struct
dry-initializer
ActiveModel
virtus
0 225000 450000 675000 900000
memsize objects strings
https:"//gist.github.com/IvanShamatov/94e78ca52f04f20c6085651345dbdfda
Slide 42
Slide 42 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 42
Вызываем конструктор 50000 раз:
Бенчмарк по производительности
struct
dry-initializer
dry-struct
ActiveModel
virtus
0 1500000 3000000 4500000 6000000
ips
https:"//gist.github.com/IvanShamatov/94e78ca52f04f20c6085651345dbdfda
Slide 43
Slide 43 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 43
Но я хочу использовать PORO!
Slide 44
Slide 44 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 44
class PostMetaType < ActiveRecord"::Type"::Value
class Meta
include ActiveModel"::Model
include ActiveModel"::Attributes
attribute :published_at, :date
end
def serialize(value)
ActiveSupport"::JSON.encode(value.attributes)
end
def cast(value)
decoded = value.is_a?(String) ? ActiveSupport"::JSON.decode(value) : value
Meta.new(decoded)
end
end
Slide 45
Slide 45 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 45
Attributes API + struct = ♥
class PostMetaType < ActiveRecord"::Type"::Value
Meta = Struct.new(:published_at) do
def initialize(published_at=nil)
self.published_at =
if published_at.nil? "|| published_at.is_a?(Date)
published_at
else
Date.parse(published_at)
end
end
end
def serialize(value)
ActiveSupport"::JSON.encode(value)
end
def cast(value)
decoded = (value.is_a?(String) ? ActiveSupport"::JSON.decode(value) : value).symbolize_keys
Meta.new(*Meta.members.map { |name| decoded[name] })
end
end
Slide 46
Slide 46 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 46
Attributes API + dry-struct = ♥
class PostMetaType < ActiveRecord"::Type"::Value
class Meta < Dry"::Struct
attribute :published_at, Dry.Types"::JSON"::Date.optional
end
def serialize(value)
ActiveSupport"::JSON.encode(value.attributes)
end
def cast(value)
decoded = (value.is_a?(String) ? ActiveSupport"::JSON.decode(value) : value).symbolize_keys
values = Meta.attribute_names.each_with_object({}) { |a, h| h[a] = decoded[a] }
Meta.new(values)
end
end
Slide 47
Slide 47 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev
JSON–колонки как модели
47
Data Mapping в Rails
Зачем разработчику знать про Attributes API
JSON–колонки как модели
Slide 48
Slide 48 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 48
Типы для работы с JSON похожи
class PostMetaType < ActiveRecord"::Type"::Value
class Meta
include ActiveModel"::Model
include ActiveModel"::Attributes
attribute :published_at, :date
end
def serialize(value)
ActiveSupport"::JSON.encode(value.attributes)
end
def cast(value)
decoded = value.is_a?(String) ?
ActiveSupport"::JSON.decode(value) :
value
Meta.new(decoded)
end
end
class UserSyncType < ActiveRecord"::Type"::Value
class Sync
include ActiveModel"::Model
include ActiveModel"::Attributes
attribute :last_sync_at, :datetime
attribute :success, :boolean
end
def serialize(value)
ActiveSupport"::JSON.encode(value.attributes)
end
def cast(value)
decoded = value.is_a?(String) ?
ActiveSupport"::JSON.decode(value) :
value
Sync.new(decoded)
end
end
Slide 49
Slide 49 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 49
Очень много скобок!
product = Product.find(params[:id])
if product.configuration[:model] "== "spaceship"
product.configuration[:color] = "red"
end
product.save
Slide 50
Slide 50 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 50
product = Product.find(params[:id])
if product.configuration.model "== "spaceship"
product.configuration.color = "red"
end
product.save
Очень много скобок!
Slide 51
Slide 51 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 51
JSON нужно валидировать
product = Product.new
puts product.valid? # "=> false
puts product.errors.messages # "=> { configuration: ["is invalid"] }
puts product.configuration.errors.messages # "=> { color: ["can't be blank"] }
Slide 52
Slide 52 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 52
store_model
https:"//github.com/DmitryTsepelev/store_model
class Configuration
include StoreModel"::Model
attribute :model, :string
enum :status, %i[active archived], default: :active
validates :status, presence: true
end
class Product < ApplicationRecord
attribute :configuration, Configuration.to_type
validates :configuration, store_model: true
end
Slide 53
Slide 53 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 53
store_model: модель
https:"//github.com/DmitryTsepelev/store_model
module StoreModel
module Model
def self.included(base)
base.include ActiveModel"::Model
base.include ActiveModel"::Attributes
base.include StoreModel"::NestedAttributes
base.extend StoreModel"::Enum
base.extend StoreModel"::TypeBuilders
end
end
end
Slide 54
Slide 54 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 54
store_model: генерация типов
https:"//github.com/DmitryTsepelev/store_model
module StoreModel
module TypeBuilders
def to_type
Types"::JsonType.new(self)
end
end
end
Slide 55
Slide 55 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 55
store_model: тип–обертка
class StoreModel"::Types"::JsonType < ActiveModel"::Type"::Value
def initialize(model_klass)
@model_klass = model_klass
end
…
end
https:"//github.com/DmitryTsepelev/store_model
Slide 56
Slide 56 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 56
store_model: тип–обертка
class StoreModel"::Types"::JsonType < ActiveModel"::Type"::Value
def cast_value(value)
case value
when String
decoded = ActiveSupport"::JSON.decode(value) rescue nil
@model_klass.new(decoded) unless decoded.nil?
when Hash
@model_klass.new(value)
when @model_klass, nil
value
else
raise_cast_error(value)
end
end
…
end https:"//github.com/DmitryTsepelev/store_model
Slide 57
Slide 57 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 57
store_model: тип–обертка
class StoreModel"::Types"::JsonType < ActiveModel"::Type"::Value
…
def serialize(value)
case value
when Hash, @model_klass then ActiveSupport"::JSON.encode(value)
else super
end
end
end
https:"//github.com/DmitryTsepelev/store_model
Slide 58
Slide 58 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 58
валидации
enums
атрибуты–списки
вложенные модели
обработка необъявленных полей
Что еще можно сделать?
https:"//github.com/DmitryTsepelev/store_model
Slide 59
Slide 59 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 59
https:"//bit.ly/3amqMeL
Slide 60
Slide 60 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 60
nested_record
https:"//github.com/marshall-lee/nested_record
class User < ActiveRecord"::Base
include NestedRecord"::Macro
has_one_nested :profile
end
class Profile < NestedRecord"::Base
attribute :age, :integer
attribute :active, :boolean
has_one_nested :contacts
end
class Profile"::Contacts < NestedRecord"::Base
attribute :email, :string
attribute :phone, :string
end
user.profile.age = 39
user.profile.contacts.email = '[email protected]'
Slide 61
Slide 61 text
RUBY DAY 2020 DmitryTsepelev
@dmitrytsepelev 61
вложенность дает накладные расходы
возможно стоит исключить JSON из выборки по умолчанию
⚠ JSON — не серебряная пуля ⚠
Slide 62
Slide 62 text
evl.ms/blog
@dmitrytsepelev
DmitryTsepelev
@evilmartians
evl.ms/telegram
THANK YOU!
RUBY DAY 2020 62