Data Mapping: the Rails Way

1. Data Mapping в Rails
2. Зачем разработчику знать про Attributes API
3. JSON–колонки как модели

Transcript

1. Data Mapping
   The Rails Way
   Дмитрий Цепелев, Злые марсиане
   фото: Bryan Birdwell, https://bit.ly/2Tpy3mU
   

   View Slide

2. DmitryTsepelev
   @dmitrytsepelev
   RUBY DAY 2020 2
   

   View Slide

3. DmitryTsepelev
   @dmitrytsepelev
   RUBY DAY 2020 3
   evilmartians.com
   

   View Slide

4. DmitryTsepelev
   @dmitrytsepelev
   RUBY DAY 2020 4
   evilmartians.com
   

   View Slide

5. RUBY DAY 2020 DmitryTsepelev
   @dmitrytsepelev 5
   Data mapping — процесс преобразования данных от одного
   формата к другому
   

   View Slide

6. RUBY DAY 2020 DmitryTsepelev
   @dmitrytsepelev 6
   Преобразование типов
   SELECT "created_at" FROM "users" LIMIT 1
   ┌──────────────────────────┐
   │ created_at │
   ├──────────────────────────┤
   │ 2019-11-22 09:29:17.5634 │
   └──────────────────────────┘
   User.first.created_at
   # "=> Fri, 22 Nov 2019 12:29:17 MSK +03:00
   

   View Slide

7. RUBY DAY 2020 DmitryTsepelev
   @dmitrytsepelev 7
   Построение запросов
   # date
   Order.where("created_at ">= ?", Date.today.beginning_of_day)
   #"=> select * from orders where created_at ">= '20-02-06'
   # enum
   Order.where(status: :complete)
   #"=> select * from orders where status = 3
   

   View Slide

8. RUBY DAY 2020 DmitryTsepelev
   @dmitrytsepelev
   Data Mapping в Rails
   Зачем разработчику знать про Attributes API
   JSON–колонки как модели
   8
   

   View Slide

9. RUBY DAY 2020 DmitryTsepelev
   @dmitrytsepelev 9
   Атрибуты модели
   User.attribute_types
   # "=> {"id""=>#,
   # "created_at""=>#,
   # "updated_at""=>#,
   # ""...
   # }
   

   View Slide

10. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 10
    Автоматическое определение атрибутов
    module ActiveRecord
    module ModelSchema
    def load_schema!
    @columns_hash = connection.schema_cache
    .columns_hash(table_name)
    .except(*ignored_columns)
    @columns_hash.each do |name, column|
    define_attribute(
    name,
    connection.lookup_cast_type_from_column(column)
    )
    end
    end
    end
    end
    

    View Slide

11. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 11
    ignored_columns
    class Post < ActiveRecord"::Base
    self.ignored_columns = %w(meta)
    end
    

    View Slide

12. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 12
    Определение атрибутов вручную
    class Configuration
    include ActiveModel"::Attributes
    attribute :model, :string
    attribute :color, :string, default: "red"
    end
    config = Configuration.new
    config.model = 42
    puts config.model.inspect # "=> "42"
    puts config.color.inspect # "=> "red"
    

    View Slide

13. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 13
    Что будет?
    class TestModel
    include ActiveModel"::Model
    include ActiveModel"::Attributes
    attribute :list, default: []
    end
    model1 = TestModel.new
    model1.list "<< 1
    model2 = TestModel.new
    model2.list "<< 2
    model2.list "== [2] # "=> false
    

    View Slide

14. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 14
    Будет ерунда
    class TestModel
    include ActiveModel"::Model
    include ActiveModel"::Attributes
    attribute :list, default: []
    end
    model1 = TestModel.new
    model1.list "<< 1
    model2 = TestModel.new
    model2.list "<< 2
    model2.list # "=> [1, 2]
    

    View Slide

15. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 15
    Нужно использовать lambda
    class TestModel
    include ActiveModel"::Model
    include ActiveModel"::Attributes
    attribute :list, default: "-> { [] }
    end
    model1 = TestModel.new
    model1.list "<< 1
    model2 = TestModel.new
    model2.list "<< 2
    model2.list # "=> [2]
    

    View Slide

16. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 16
    ActiveModel"::Type"::BigInteger
    ActiveModel"::Type"::Binary
    ActiveModel"::Type"::Boolean
    ActiveModel"::Type"::DateTime
    ActiveModel"::Type"::Date
    ActiveModel"::Type"::Decimal
    ActiveModel"::Type"::"Float
    ActiveModel"::Type"::ImmutableString
    ActiveModel"::Type"::Integer
    ActiveModel"::Type"::String
    ActiveModel"::Type"::Time
    Типы «из коробки»
    ActiveRecord"::Type"::DateTime
    ActiveRecord"::Type"::Date
    ActiveRecord"::Type"::DecimalWithoutScale
    ActiveRecord"::Type"::Json
    ActiveRecord"::Type"::Time
    ActiveRecord"::Type"::UnsignedInteger
    

    View Slide

17. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 17
    class ActiveModel"::Type"::Value
    # DB "-> Ruby
    def deserialize(value)
    cast(value)
    end
    # User "-> Ruby
    def cast(value)
    cast_value(value) unless value.nil?
    end
    # Ruby "-> DB
    def serialize(value)
    value
    end
    end
    Что внутри?
    

    View Slide

18. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 18
    Реестр типов
    ActiveModel::Type::Registry или наследник
    в ActiveRecord используется AdapterSpecificRegistry
    умеет находить класс типа по имени
    ActiveRecord"::Type.registry.lookup(:integer)
    # "=> #
    

    View Slide

19. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 19
    Создание собственного типа
    class MoneyType < ActiveRecord"::Type"::Integer
    def cast(value)
    if !value.kind_of?(Numeric) "&& value.include?('$')
    price_in_dollars = value.gsub(/\$/, '').to_f
    super(price_in_dollars * 70)
    else
    super
    end
    end
    end
    

    View Slide

20. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 20
    Использование собственного типа
    ActiveRecord"::Type.register(
    :money,
    MoneyType
    )
    class User < ApplicationRecord
    attribute :balance, :money
    end
    User.new(balance: '$1').balance
    # "=> 70
    class User < ApplicationRecord
    attribute :balance, MoneyType.new
    end
    User.new(balance: '$1').balance
    # "=> 70
    через реестр напрямую
    

    View Slide

21. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 21
    ⚠ Type cast происходит при чтении ⚠
    class IntegerRangeType < ActiveRecord"::Type"::Integer
    def initialize(range)
    @range = range
    end
    def cast(value)
    raise ArgumentError unless range.cover?(value)
    super
    end
    end
    class User
    include ActiveModel"::Attributes
    attribute :age, IntegerRangeType.new(0"..100)
    end
    user = User.new
    user.age = 120
    puts user.age # "=> ArgumentError
    

    View Slide

22. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev
    Зачем разработчику знать
    про Attributes API
    22
    Data Mapping в Rails
    Зачем разработчику знать про Attributes API
    JSON–колонки как модели
    

    View Slide

23. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 23
    Виртуальные атрибуты
    class User < ApplicationRecord
    attribute :skip_confirmation, :boolean, default: false
    validate :confirmation, unless: :skip_confirmation?
    end
    user = User.new(skip_confirmation: '0')
    

    View Slide

24. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 24
    Битовые операции
    IN_APP = 0
    EMAIL = 1
    PUSH = 2
    settings = 0
    0 0 0
    

    View Slide

25. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 25
    Битовые операции
    IN_APP = 0
    EMAIL = 1
    PUSH = 2
    settings = 0
    settings "|= (1 "<< EMAIL)
    0 0 0
    0 1 0
    OR
    

    View Slide

26. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 26
    Битовые операции
    IN_APP = 0
    EMAIL = 1
    PUSH = 2
    settings = 0
    settings "|= (1 "<< EMAIL) # "=> 2
    settings.to_s(2) # "=> "10"
    0 1 0
    

    View Slide

27. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 27
    Битовые операции
    IN_APP = 0
    EMAIL = 1
    PUSH = 2
    settings = 0
    settings "|= (1 "<< EMAIL) # "=> 2
    settings.to_s(2) # "=> "10"
    settings & (1 "<< EMAIL) > 0
    0 1 0
    0 1 0
    AND
    

    View Slide

28. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 28
    Битовые операции
    IN_APP = 0
    EMAIL = 1
    PUSH = 2
    settings = 0
    settings "|= (1 "<< EMAIL) # "=> 2
    settings.to_s(2) # "=> "10"
    settings & (1 "<< EMAIL) > 0 # "=> true
    0 1 0
    0 1 0
    AND
    

    View Slide

29. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 29
    Битовые операции
    IN_APP = 0
    EMAIL = 1
    PUSH = 2
    settings = 0
    settings "|= (1 "<< EMAIL) # "=> 2
    settings.to_s(2) # "=> "10"
    settings & (1 "<< EMAIL) > 0 # "=> true
    settings & (1 "<< PUSH) > 0
    0 1 0
    1 0 0
    AND
    

    View Slide

30. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 30
    Битовые операции
    IN_APP = 0
    EMAIL = 1
    PUSH = 2
    settings = 0
    settings "|= (1 "<< EMAIL) # "=> 2
    settings.to_s(2) # "=> "10"
    settings & (1 "<< EMAIL) > 0 # "=> true
    settings & (1 "<< PUSH) > 0 # "=> false
    0 1 0
    1 0 0
    AND
    

    View Slide

31. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 31
    Битовые операции
    IN_APP = 0
    EMAIL = 1
    PUSH = 2
    settings = 0
    settings "|= (1 "<< EMAIL) # "=> 2
    settings.to_s(2) # "=> "10"
    settings & (1 "<< EMAIL) > 0 # "=> true
    settings & (1 "<< PUSH) > 0 # "=> false
    settings "|= (1 "<< PUSH) # "=> 6
    settings.to_s(2)
    0 1 0
    1 0 0
    OR
    

    View Slide

32. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 32
    Битовые операции
    IN_APP = 0
    EMAIL = 1
    PUSH = 2
    settings = 0
    settings "|= (1 "<< EMAIL) # "=> 2
    settings.to_s(2) # "=> "10"
    settings & (1 "<< EMAIL) > 0 # "=> true
    settings & (1 "<< PUSH) > 0 # "=> false
    settings "|= (1 "<< PUSH) # "=> 6
    settings.to_s(2) # "=> "110"
    settings & (1 "<< PUSH) > 0 # "=> true
    1 1 0
    

    View Slide

33. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 33
    Value Object
    class NotificationSettings < ActiveRecord"::Type"::Value
    IN_APP = 0
    EMAIL = 1
    PUSH = 2
    …
    end
    

    View Slide

34. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 34
    Value Object
    class NotificationSettings < ActiveRecord"::Type"::Value
    …
    def cast(value)
    Value.new(value)
    end
    end
    

    View Slide

35. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 35
    Value Object
    class NotificationSettings < ActiveRecord"::Type"::Value
    …
    class Value
    attr_reader :settings
    def initialize(settings)
    @settings = settings
    end
    def subscribe(subscription)
    @settings "|= (1 "<< subscription)
    end
    def receives_email?
    @settings & (1 "<< EMAIL) > 0
    end
    end
    end
    

    View Slide

36. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 36
    Value Object
    class User < ActiveRecord+::Base
    attribute :notification_settings,
    NotificationSettings.new,
    default: 0
    end
    user = User.new
    puts user.notification_settings.receives_email? # "=> false
    user.notification_settings.subscribe(NotificationSettings"::EMAIL)
    puts user.notification_settings.receives_email? # "=> true
    

    View Slide

37. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 37
    Работа с legacy колонками
    class Post < ActiveRecord"::Base
    attribute :published_at, StringDateType.new
    end
    Post.insert_all([{ published_at: "2012-10-17 14:35" }])
    puts Post.last.published_at.inspect # "=> 2012-10-17 14:35:00 +0400
    Post.create(published_at: Time.now)
    puts Post.last.published_at # "=> 2020-02-12 20:15:00 +0300
    

    View Slide

38. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 38
    Работа с legacy колонками
    class StringDateType < ActiveRecord"::Type"::Value
    LEGACY_FORMAT = "%Y-%m-%d %H:%M"
    def serialize(value)
    value.strftime(LEGACY_FORMAT)
    end
    def deserialize(value)
    Time.strptime(value, LEGACY_FORMAT)
    end
    end
    

    View Slide

39. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 39
    Работа с JSON колонками
    class Post < ActiveRecord"::Base
    attribute :meta, PostMetaType.new
    end
    post = Post.new(meta: { published_at: 1.week.ago })
    puts post.meta.published_at # "=> 2020-02-05
    post.meta.published_at = Time.now
    post.save
    puts post.meta.published_at # "=> 2020-02-12
    

    View Slide

40. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 40
    Работа с JSON колонками
    class PostMetaType < ActiveRecord"::Type"::Value
    class Meta
    include ActiveModel"::Model
    include ActiveModel"::Attributes
    attribute :published_at, :date
    end
    def serialize(value)
    ActiveSupport"::JSON.encode(value.attributes)
    end
    def cast(value)
    decoded = value.is_a?(String) ? ActiveSupport"::JSON.decode(value) : value
    Meta.new(decoded)
    end
    end
    

    View Slide

41. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 41
    Вызываем конструктор 50000 раз:
    Бенчмарк по памяти
    struct
    dry-struct
    dry-initializer
    ActiveModel
    virtus
    0 225000 450000 675000 900000
    memsize objects strings
    https:"//gist.github.com/IvanShamatov/94e78ca52f04f20c6085651345dbdfda
    

    View Slide

42. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 42
    Вызываем конструктор 50000 раз:
    Бенчмарк по производительности
    struct
    dry-initializer
    dry-struct
    ActiveModel
    virtus
    0 1500000 3000000 4500000 6000000
    ips
    https:"//gist.github.com/IvanShamatov/94e78ca52f04f20c6085651345dbdfda
    

    View Slide

43. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 43
    
    Но я хочу использовать PORO!
    

    View Slide

44. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 44
    class PostMetaType < ActiveRecord"::Type"::Value
    class Meta
    include ActiveModel"::Model
    include ActiveModel"::Attributes
    attribute :published_at, :date
    end
    def serialize(value)
    ActiveSupport"::JSON.encode(value.attributes)
    end
    def cast(value)
    decoded = value.is_a?(String) ? ActiveSupport"::JSON.decode(value) : value
    Meta.new(decoded)
    end
    end
    

    View Slide

45. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 45
    Attributes API + struct = ♥
    class PostMetaType < ActiveRecord"::Type"::Value
    Meta = Struct.new(:published_at) do
    def initialize(published_at=nil)
    self.published_at =
    if published_at.nil? "|| published_at.is_a?(Date)
    published_at
    else
    Date.parse(published_at)
    end
    end
    end
    def serialize(value)
    ActiveSupport"::JSON.encode(value)
    end
    def cast(value)
    decoded = (value.is_a?(String) ? ActiveSupport"::JSON.decode(value) : value).symbolize_keys
    Meta.new(*Meta.members.map { |name| decoded[name] })
    end
    end
    

    View Slide

46. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 46
    Attributes API + dry-struct = ♥
    class PostMetaType < ActiveRecord"::Type"::Value
    class Meta < Dry"::Struct
    attribute :published_at, Dry.Types"::JSON"::Date.optional
    end
    def serialize(value)
    ActiveSupport"::JSON.encode(value.attributes)
    end
    def cast(value)
    decoded = (value.is_a?(String) ? ActiveSupport"::JSON.decode(value) : value).symbolize_keys
    values = Meta.attribute_names.each_with_object({}) { |a, h| h[a] = decoded[a] }
    Meta.new(values)
    end
    end
    

    View Slide

47. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev
    JSON–колонки как модели
    47
    Data Mapping в Rails
    Зачем разработчику знать про Attributes API
    JSON–колонки как модели
    

    View Slide

48. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 48
    Типы для работы с JSON похожи
    class PostMetaType < ActiveRecord"::Type"::Value
    class Meta
    include ActiveModel"::Model
    include ActiveModel"::Attributes
    attribute :published_at, :date
    end
    def serialize(value)
    ActiveSupport"::JSON.encode(value.attributes)
    end
    def cast(value)
    decoded = value.is_a?(String) ?
    ActiveSupport"::JSON.decode(value) :
    value
    Meta.new(decoded)
    end
    end
    class UserSyncType < ActiveRecord"::Type"::Value
    class Sync
    include ActiveModel"::Model
    include ActiveModel"::Attributes
    attribute :last_sync_at, :datetime
    attribute :success, :boolean
    end
    def serialize(value)
    ActiveSupport"::JSON.encode(value.attributes)
    end
    def cast(value)
    decoded = value.is_a?(String) ?
    ActiveSupport"::JSON.decode(value) :
    value
    Sync.new(decoded)
    end
    end
    

    View Slide

49. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 49
    Очень много скобок!
    product = Product.find(params[:id])
    if product.configuration[:model] "== "spaceship"
    product.configuration[:color] = "red"
    end
    product.save
    

    View Slide

50. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 50
    product = Product.find(params[:id])
    if product.configuration.model "== "spaceship"
    product.configuration.color = "red"
    end
    product.save
    Очень много скобок!
    

    View Slide

51. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 51
    JSON нужно валидировать
    product = Product.new
    puts product.valid? # "=> false
    puts product.errors.messages # "=> { configuration: ["is invalid"] }
    puts product.configuration.errors.messages # "=> { color: ["can't be blank"] }
    

    View Slide

52. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 52
    store_model
    https:"//github.com/DmitryTsepelev/store_model
    class Configuration
    include StoreModel"::Model
    attribute :model, :string
    enum :status, %i[active archived], default: :active
    validates :status, presence: true
    end
    class Product < ApplicationRecord
    attribute :configuration, Configuration.to_type
    validates :configuration, store_model: true
    end
    

    View Slide

53. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 53
    store_model: модель
    https:"//github.com/DmitryTsepelev/store_model
    module StoreModel
    module Model
    def self.included(base)
    base.include ActiveModel"::Model
    base.include ActiveModel"::Attributes
    base.include StoreModel"::NestedAttributes
    base.extend StoreModel"::Enum
    base.extend StoreModel"::TypeBuilders
    end
    end
    end
    

    View Slide

54. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 54
    store_model: генерация типов
    https:"//github.com/DmitryTsepelev/store_model
    module StoreModel
    module TypeBuilders
    def to_type
    Types"::JsonType.new(self)
    end
    end
    end
    

    View Slide

55. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 55
    store_model: тип–обертка
    class StoreModel"::Types"::JsonType < ActiveModel"::Type"::Value
    def initialize(model_klass)
    @model_klass = model_klass
    end
    …
    end
    https:"//github.com/DmitryTsepelev/store_model
    

    View Slide

56. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 56
    store_model: тип–обертка
    class StoreModel"::Types"::JsonType < ActiveModel"::Type"::Value
    def cast_value(value)
    case value
    when String
    decoded = ActiveSupport"::JSON.decode(value) rescue nil
    @model_klass.new(decoded) unless decoded.nil?
    when Hash
    @model_klass.new(value)
    when @model_klass, nil
    value
    else
    raise_cast_error(value)
    end
    end
    …
    end https:"//github.com/DmitryTsepelev/store_model
    

    View Slide

57. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 57
    store_model: тип–обертка
    class StoreModel"::Types"::JsonType < ActiveModel"::Type"::Value
    …
    def serialize(value)
    case value
    when Hash, @model_klass then ActiveSupport"::JSON.encode(value)
    else super
    end
    end
    end
    https:"//github.com/DmitryTsepelev/store_model
    

    View Slide

58. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 58
    валидации
    enums
    атрибуты–списки
    вложенные модели
    обработка необъявленных полей
    Что еще можно сделать?
    https:"//github.com/DmitryTsepelev/store_model
    

    View Slide

59. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 59
    https:"//bit.ly/3amqMeL
    

    View Slide

60. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 60
    nested_record
    https:"//github.com/marshall-lee/nested_record
    class User < ActiveRecord"::Base
    include NestedRecord"::Macro
    has_one_nested :profile
    end
    class Profile < NestedRecord"::Base
    attribute :age, :integer
    attribute :active, :boolean
    has_one_nested :contacts
    end
    class Profile"::Contacts < NestedRecord"::Base
    attribute :email, :string
    attribute :phone, :string
    end
    user.profile.age = 39
    user.profile.contacts.email = 'john@doe.com'
    

    View Slide

61. RUBY DAY 2020 DmitryTsepelev
    @dmitrytsepelev 61
    вложенность дает накладные расходы
    возможно стоит исключить JSON из выборки по умолчанию
    ⚠ JSON — не серебряная пуля ⚠
    

    View Slide

62. evl.ms/blog
    @dmitrytsepelev
    DmitryTsepelev
    @evilmartians
    evl.ms/telegram
    THANK YOU!
    RUBY DAY 2020 62
    

    View Slide