Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Translating with Mobility

Translating with Mobility

Presentation at the Montreal Ruby Meetup on June 20, 2017.

Chris Salzberg

June 20, 2017
Tweet

More Decks by Chris Salzberg

Other Decks in Programming

Transcript

  1. Translating with Mobility
    Chris Salzberg (@shioyama)

    View full-size slide

  2. About Me

    Writer, programmer, translator

    VP of Engineering at Degica

    @shioyama on github, twitter, etc

    I program in Ruby

    I live in Tokyo, Japan

    I am the author of Mobility

    I blog at dejimata.com

    I coined “Module Builder Pattern”

    View full-size slide

  3. How I got here today
    Montreal
    Amsterdam
    Tokyo

    View full-size slide

  4. Translation is something that
    seems easy, but is actually very hard

    View full-size slide

  5. One Problem, Many Solutions

    How many languages?

    How many translated models?

    Growing application? Adding new models?

    What’s your database?

    What are your performance requirements?

    How good are you at Arel?

    View full-size slide

  6. Gems and Strategies

    Translation Strategies and Gems:
    – Translatable Columns (Traco)
    – Model Translation Tables (Globalize)
    – Shared Translation Table(s)
    – Serialized Columns (Multilang)
    – Hstore/Jsonb (json_translate, multilang_hstore, ...)

    Each gem is coupled to a storage strategy

    Strategies are (conceptually) independent of ORM

    View full-size slide

  7. Plan for Today’s Talk
    1. Translation Interface
    – How do you read and write translations?
    – What other features are supported?
    2. Storage Strategies
    – How are translations actually stored?
    – Good and bad of each strategy
    3. Mobility
    4. Looking Ahead

    View full-size slide

  8. Translation Interface

    View full-size slide

  9. Common Features

    Translations are fetched and stored using
    (virtual) attribute accessors

    Language fetched or stored is the value of the
    global locale (I18n.locale or similar)

    Models can be queried using translated
    attributes (like normal column-based attributes)

    View full-size slide

  10. One Attribute, Many Languages
    class Post < ApplicationRecord
    translates :title
    end
    post = Post.new
    I18n.locale = :en
    post.title = "Translating with Mobility"
    I18n.locale = :fr
    post.title = "Traduction avec Mobilité"
    post.save
    post = Post.first
    I18n.locale = :en
    post.title #=> "Translating with Mobility"
    I18n.locale = :fr
    post.title #=> "Traduction avec Mobilité"
    class Post < ApplicationRecord
    translates :title
    end
    post = Post.new
    I18n.locale = :en
    post.title = "Translating with Mobility"
    I18n.locale = :fr
    post.title = "Traduction avec Mobilité"
    post.save
    post = Post.first
    I18n.locale = :en
    post.title #=> "Translating with Mobility"
    I18n.locale = :fr
    post.title #=> "Traduction avec Mobilité"

    View full-size slide

  11. Fallbacks
    I18n.locale = :en
    post.title #=> nil
    I18n.locale = :en
    post.title #=> nil
    I18n.locale = :en
    post.title #=> " タイトル "
    I18n.locale = :en
    post.title #=> " タイトル "
    No Fallbacks :en => :ja
    post = Post.new
    I18n.locale = :ja
    post.title = " タイトル "
    post.save
    post = Post.new
    I18n.locale = :ja
    post.title = " タイトル "
    post.save

    View full-size slide

  12. Translations Cache/Stash
    post = Post.first
    I18n.locale = :ja
    post.title
    #=> " タイトル "
    I18n.locale = :en
    post.title
    #=> "Title"
    I18n.locale = :fr
    post.title
    #=> "Titre"
    post = Post.first
    I18n.locale = :ja
    post.title
    #=> " タイトル "
    I18n.locale = :en
    post.title
    #=> "Title"
    I18n.locale = :fr
    post.title
    #=> "Titre"
    Locale Value
    ja タイトル
    en Title
    fr Titre
    “Title” Cache

    View full-size slide

  13. Locale Accessors
    class Post < ApplicationRecord
    translates :title, locale_accessors: [:en, :ja]
    end
    post = Post.new
    post.title_ja = " タイトル 2"
    post.title_en = "Title 2"
    post.title_ja
    #=> " タイトル 2"
    post.title_en
    #=> "Title 2"
    class Post < ApplicationRecord
    translates :title, locale_accessors: [:en, :ja]
    end
    post = Post.new
    post.title_ja = " タイトル 2"
    post.title_en = "Title 2"
    post.title_ja
    #=> " タイトル 2"
    post.title_en
    #=> "Title 2"

    View full-size slide

  14. Dirty Attribute Tracking

    Tracking of translated attribute changes
    – title_was
    – title_changes
    – title_changed?
    – ...

    View full-size slide

  15. Bonus Points: Query by Translation
    I18n.locale = :en
    Post.find_by(title: "Translating with Mobility")
    #=> (post with English title "Translating...")
    I18n.locale = :fr
    Post.find_by(title: "Traduction avec Mobilité")
    #=> (returns post with French title "Traduction...")
    I18n.locale = :en
    Post.find_by(title: "Translating with Mobility")
    #=> (post with English title "Translating...")
    I18n.locale = :fr
    Post.find_by(title: "Traduction avec Mobilité")
    #=> (returns post with French title "Traduction...")

    Override query methods to return records by
    translated attribute

    Hides details of storage strategy from query interface

    View full-size slide

  16. Storage Strategies

    View full-size slide

  17. What is a Storage Strategy?
    translated attribute
    translated attribute
    Persistence Layer
    describes a mapping between
    translated attributes and persistence layer

    View full-size slide

  18. Translatable Columns

    View full-size slide

  19. Translatable Columns

    Good
    – Easy to understand
    – Easy to construct queries
    – Good performance (no joins)

    Bad
    – Every new language requires a migration for every translated
    model
    – Columns created for every locale even if there is no
    translation for a locale

    View full-size slide

  20. Model Translation Tables

    View full-size slide

  21. Model Translation Tables

    Translation table for each model, with:
    – locale column
    – foreign key pointing to model table (post_id)
    – one or more translated columns

    has_many association between model and translations

    Translation table records are created/updated when attribute
    is set and model is saved
    has_many :translations, class_name: PostTranslation
    has_many :translations, class_name: PostTranslation

    View full-size slide

  22. Model Translation Tables

    Good
    – Easily scales with number of languages (no new migrations to add
    locale)
    – Space usage is more efficient (only create record if translation is
    stored)

    Bad
    – Querying may be less performant (need to join translation table)
    – Code is more complex due to joining
    – Migration to create translation table/column for every model/attribute

    View full-size slide

  23. Shared Translation Table(s)

    View full-size slide

  24. Shared Translation Table(s)

    One or more translation tables, with:
    – locale column
    – foreign key pointing to model id (translatable_id)
    – column holding the model class (translatable_type)
    – column holding the name of the attribute (key)
    – column holding the value of the attribute (value)

    One translation table for each value type (string, text, ...)
    has_many :translations,
    as: :translatable,
    class_name: ::Mobility::ActiveRecord::StringTranslation
    has_many :translations,
    as: :translatable,
    class_name: ::Mobility::ActiveRecord::StringTranslation

    View full-size slide

  25. Shared Translation Table(s)

    Good
    – No additional migrations at all, so easy to scale to new
    locales + new models
    – Space usage is relatively efficient, scales with number of
    locales + models

    Bad
    – Very complex querying
    – All translations in the same table, can be difficult to
    maintain a consistent database state

    View full-size slide

  26. Serialized Translations

    Store translations as serialized hash on a single
    database column
    class Post < ApplicationRecord
    serialize :title_translations, Hash
    end
    post = Post.new
    post.title_translations = { "en" => "Translating with Mobility" }
    post.save
    class Post < ApplicationRecord
    serialize :title_translations, Hash
    end
    post = Post.new
    post.title_translations = { "en" => "Translating with Mobility" }
    post.save

    View full-size slide

  27. Serialized Translations

    Good
    – Scales well with locales (no additional migrations)
    – Easy to add translations to model (simply add text column for
    each translated attribute)
    – Simple to access

    Bad
    – Querying is not possible
    – Loading one translation requires loading all translations at
    once (serialized column)

    View full-size slide

  28. Hstore/Jsonb (PostgreSQL)

    Similar to serialized strategy, except that querying is possible
    and can be fast

    Database-dependent strategy, highly coupled to persistence layer

    (Jsonb generally preferred over Hstore)
    Post.where(title: "foo").to_sql
    #=> SELECT "posts".* FROM "posts"
    WHERE ("posts"."title" @> '{"en":"foo"}')
    Post.where(title: "foo").to_sql
    #=> SELECT "posts".* FROM "posts"
    WHERE ("posts"."title" @> '{"en":"foo"}')

    View full-size slide

  29. Other Possibilities

    Stored in external service, accessed via API

    File storage

    NoSQL database

    Cache storage (Redis/Memcache/etc)

    New forms of storage we cannot yet imagine

    View full-size slide

  30. Translation with Mobility

    Existing solutions
    – couple implementation to storage strategy
    – couple implementation to ORM (ActiveRecord) and
    framework (Rails)
    – have similar but not identical interfaces

    What I wanted:
    – Support customizable, swappable storage solutions
    – No hard dependency on AR or Rails
    – Interface that is consistent and unified

    View full-size slide

  31. Mobility: A Translation Framework

    New approach to translation problem:
    – One translation interface, swappable storage backends
    – Supports all existing strategies, plus shared tables (“key value”)
    – Supports both ActiveRecord and Sequel ORM

    Like the OmniAuth of Translation

    View full-size slide

  32. Getting Started
    bundle exec rails g mobility:install
    bundle exec rails g mobility:install
    class Post < ApplicationRecord
    include Mobility
    translates :title, :body
    end
    class Post < ApplicationRecord
    include Mobility
    translates :title, :body
    end
    Mobility.configure do |config|
    config.default_backend = :key_value
    config.accessor_method = :translates
    config.query_method = :i18n
    end
    Mobility.configure do |config|
    config.default_backend = :key_value
    config.accessor_method = :translates
    config.query_method = :i18n
    end
    :key_value
    :table
    :column
    :jsonb
    :hstore
    :serialized

    View full-size slide

  33. Separation of Responsibilities
    Storage Layer

    Implements atomic
    methods for reading,
    writing

    Sets up model

    Extends query scope to
    support backend
    Storage Layer

    Implements atomic
    methods for reading,
    writing

    Sets up model

    Extends query scope to
    support backend
    Interface Layer

    Reading and writing
    translations

    Fallbacks, cache, dirty
    tracking

    Locale accessors
    Interface Layer

    Reading and writing
    translations

    Fallbacks, cache, dirty
    tracking

    Locale accessors

    View full-size slide

  34. “Pluggable” Backends

    Backend encapsulates a storage strategy for a
    single attribute (or set of attributes)

    All backends automatically support locale
    accessors, fallbacks, dirty tracking, cache

    Backends are tested against shared test
    suite to ensure consistency

    View full-size slide

  35. What’s a Backend?
    class MyBackend
    include Mobility::Backend
    # attr_reader :attribute, :model
    def read(locale, **options)
    # Read attribute in locale.
    end
    def write(locale, value, **options)
    # Write value to attribute in locale.
    end
    def self.configure(options)
    # Normalize options (e.g. set any defaults)
    end
    setup do |attributes, options|
    # Do something with attributes and options in context of model class.
    end
    end
    class MyBackend
    include Mobility::Backend
    # attr_reader :attribute, :model
    def read(locale, **options)
    # Read attribute in locale.
    end
    def write(locale, value, **options)
    # Write value to attribute in locale.
    end
    def self.configure(options)
    # Normalize options (e.g. set any defaults)
    end
    setup do |attributes, options|
    # Do something with attributes and options in context of model class.
    end
    end

    View full-size slide

  36. Mobility in Action
    class Post
    include Mobility
    translates :title, backend: :column, locale_accessors: [:en, :ja]
    translates :body, backend: :key_value, locale_accessors: [:fr]
    end
    post = Post.new
    post.title = “Title”
    post.body = “Lorum Ipsum...”
    post.title #=> “Title”
    post.title_backend #=> #<#
    post.title_en #=> “Title”
    post.title_ja #=> nil
    post.title_fr #=> NoMethodError
    post.body #=> “Lorum Ipsum...”
    post.body_backend #=> #<#
    post.body_en #=> NoMethodError
    post.body_ja #=> NoMethodError
    post.body_fr #=> nil
    class Post
    include Mobility
    translates :title, backend: :column, locale_accessors: [:en, :ja]
    translates :body, backend: :key_value, locale_accessors: [:fr]
    end
    post = Post.new
    post.title = “Title”
    post.body = “Lorum Ipsum...”
    post.title #=> “Title”
    post.title_backend #=> #<#
    post.title_en #=> “Title”
    post.title_ja #=> nil
    post.title_fr #=> NoMethodError
    post.body #=> “Lorum Ipsum...”
    post.body_backend #=> #<#
    post.body_en #=> NoMethodError
    post.body_ja #=> NoMethodError
    post.body_fr #=> nil

    View full-size slide

  37. Query Methods Too (?!)
    class Post
    include Mobility
    translates :title, backend: :column, locale_accessors: [:en, :ja]
    translates :body, backend: :key_value, locale_accessors: [:fr]
    end
    post = Post.new
    post.title = “Title”
    post.body = “Lorum Ipsum...”
    post.save
    Post.i18n.where(title: “Title”, body: “Lorum Ipsum...”)
    # SELECT "posts".* FROM "posts"
    # INNER JOIN "mobility_text_translations" "body_mobility_text_translations"
    # ON "body_mobility_text_translations"."key" = 'body'
    # AND "body_mobility_text_translations"."locale" = 'en'
    # AND "body_mobility_text_translations"."translatable_type" = 'Post'
    # AND "body_mobility_text_translations"."translatable_id" = "posts"."id"
    # WHERE "body_mobility_text_translations"."value" = 'Lorum Ipsum...'
    # AND "posts"."title_en" = 'Title'
    class Post
    include Mobility
    translates :title, backend: :column, locale_accessors: [:en, :ja]
    translates :body, backend: :key_value, locale_accessors: [:fr]
    end
    post = Post.new
    post.title = “Title”
    post.body = “Lorum Ipsum...”
    post.save
    Post.i18n.where(title: “Title”, body: “Lorum Ipsum...”)
    # SELECT "posts".* FROM "posts"
    # INNER JOIN "mobility_text_translations" "body_mobility_text_translations"
    # ON "body_mobility_text_translations"."key" = 'body'
    # AND "body_mobility_text_translations"."locale" = 'en'
    # AND "body_mobility_text_translations"."translatable_type" = 'Post'
    # AND "body_mobility_text_translations"."translatable_id" = "posts"."id"
    # WHERE "body_mobility_text_translations"."value" = 'Lorum Ipsum...'
    # AND "posts"."title_en" = 'Title'

    View full-size slide

  38. Under the Hood
    class Post
    translates :title, backend: :key_value,
    locale_accessors: [:en, :ja]
    end
    class Post
    translates :title, backend: :key_value,
    locale_accessors: [:en, :ja]
    end
    class Post
    include Mobility::Attributes.new(:accessor, [:title],
    backend: :key_value,
    model_class: Post)
    end
    class Post
    include Mobility::Attributes.new(:accessor, [:title],
    backend: :key_value,
    model_class: Post)
    end
    It’s a Module Builder!

    View full-size slide

  39. Module Builder
    class Attributes < Module
    def initialize(method, *attributes)
    # ...
    attributes.each do |attribute|
    define_method attribute do
    # ...
    end
    # ...
    end
    end
    end
    class Post
    include Attributes.new(:accessor,
    [:title])
    end
    class Attributes < Module
    def initialize(method, *attributes)
    # ...
    attributes.each do |attribute|
    define_method attribute do
    # ...
    end
    # ...
    end
    end
    end
    class Post
    include Attributes.new(:accessor,
    [:title])
    end
    1) Subclass Module
    2) Define module methods
    in an initializer
    3) Include instance of
    Module subclass in other
    classes

    View full-size slide

  40. “Plugging in”: Building a Backend
    class Post
    translates :title, fallbacks: true, dirty: true
    translates :subtitle, backend: :column
    translates :body, locale_accessors: [:en, :de], cache: false
    end
    class Post
    translates :title, fallbacks: true, dirty: true
    translates :subtitle, backend: :column
    translates :body, locale_accessors: [:en, :de], cache: false
    end
    backend_class.include Mobility::Backend::Fallbacks
    backend_class.include Mobility::Backend::ActiveModel::Dirty
    @backend_class = Class.new(Mobility::Backend::ActiveRecord::Column)
    include Mobility::LocaleAccessors.new(:body, [:en, :de])
    Another Module Builder!

    View full-size slide

  41. “Plugging in”: Building a Backend
    module Mobility
    module Fallbacks
    # ...
    def read(locale, **options)
    fallback = options.delete(:fallback)
    return super if fallback == false || (fallback.nil? && fallbacks.nil?)
    (fallback ? [locale, *fallback] : fallbacks[locale]).detect do |fallback_locale|
    value = super(fallback_locale, **options)
    break value if value.present?
    end
    end
    end
    end
    module Mobility
    module Fallbacks
    # ...
    def read(locale, **options)
    fallback = options.delete(:fallback)
    return super if fallback == false || (fallback.nil? && fallbacks.nil?)
    (fallback ? [locale, *fallback] : fallbacks[locale]).detect do |fallback_locale|
    value = super(fallback_locale, **options)
    break value if value.present?
    end
    end
    end
    end

    Modules included into backend (fallbacks, cache, dirty)
    override read and/or write methods to add functionality
    Find first non-nil value of attribute

    View full-size slide

  42. “Plugging In”: Model Setup
    class Post
    translates :title, fallbacks: true, dirty: true
    translates :subtitle, backend: :column
    translates :body, locale_accessors: [:en, :de], cache: false
    end
    class Post
    translates :title, fallbacks: true, dirty: true
    translates :subtitle, backend: :column
    translates :body, locale_accessors: [:en, :de], cache: false
    end
    class Post
    has_many :mobility_text_translations,
    where: { key: [:title, :body] },
    as: :translatable,
    class_name: ::Mobility::ActiveRecord::TextTranslation
    end
    class Post
    has_many :mobility_text_translations,
    where: { key: [:title, :body] },
    as: :translatable,
    class_name: ::Mobility::ActiveRecord::TextTranslation
    end
    backend_class.setup_model(Post, :title, fallbacks: true, dirty: true)

    View full-size slide

  43. Reading a Translated Attribute
    I18n.locale = :fr
    post.title
    I18n.locale = :fr
    post.title
    post.title_backend.read(:fr)
    post.title_backend.read(:fr)
    post.read_attribute(:title_fr)
    post.read_attribute(:title_fr) post.translations.find { |t|
    t.locale == :fr
    }.title
    post.translations.find { |t|
    t.locale == :fr
    }.title
    Translatable Columns Backend Model Translations Backend

    View full-size slide

  44. Recap (The “Sales Pitch”)

    Mobility:
    – encapsulates storage strategy into a single class
    – supports every storage strategy
    – supports multiple strategies in a single model
    – supports all features (fallbacks, dirty tracking, caching,
    locale accessors, etc) at the attribute level
    – can be used to design new backends using flexible interface

    Mobility keeps all your doors open

    View full-size slide

  45. Looking Ahead

    View full-size slide

  46. The Situation Today

    I18n is ubiquitous

    Translation is essential, but (almost) always an
    afterthought

    Gems lock you into a specific storage strategy:
    – Globalize? Locked to model translation tables
    – Traco? Locked to translatable columns
    – json_translate? Locked to PostgreSQL jsonb

    View full-size slide

  47. Mobility as a Translation Layer

    For application developers:
    – simplifies and standardizes content translation
    – does not lock you into a single storage strategy

    For developers of other gems:
    – A single Mobility integration and will support all
    current and future storage strategies
    – Example: friendly_id-mobility

    View full-size slide

  48. One Integration to Rule them All
    Translatable Columns
    Model Translation Tables
    Shared Translation Tables
    Serialized Column
    Hstore/Jsonb
    Storage Strategies Gem Integrations
    ...
    Mobility
    Mobility
    FriendlyId
    Spree / Solidus
    ActiveAdmin
    Paperclip
    ...

    View full-size slide

  49. Roadmap

    Bug fixing, Rails 5.1 full support

    Performance Testing

    Integrations
    – Spree/Solidus

    Support Hanami as ORM

    API for Mobility extensions
    – Machine translation as fallback

    View full-size slide