Translating with Mobility Chris Salzberg (@shioyama)

About Me ● Writer, programmer, translator ● VP of Engineering at Degica ● @shioyama on github, twitter, etc ● I program in Ruby ● I live in Tokyo, Japan ● I am the author of Mobility ● I blog at dejimata.com ● I coined “Module Builder Pattern”

How I got here today Montreal Amsterdam Tokyo

Translation is something that seems easy, but is actually very hard

One Problem, Many Solutions ● How many languages? ● How many translated models? ● Growing application? Adding new models? ● What’s your database? ● What are your performance requirements? ● How good are you at Arel?

Gems and Strategies ● Translation Strategies and Gems: – Translatable Columns (Traco) – Model Translation Tables (Globalize) – Shared Translation Table(s) – Serialized Columns (Multilang) – Hstore/Jsonb (json_translate, multilang_hstore, ...) ● Each gem is coupled to a storage strategy ● Strategies are (conceptually) independent of ORM

Plan for Today’s Talk 1. Translation Interface – How do you read and write translations? – What other features are supported? 2. Storage Strategies – How are translations actually stored? – Good and bad of each strategy 3. Mobility 4. Looking Ahead

Translation Interface

Common Features ● Translations are fetched and stored using (virtual) attribute accessors ● Language fetched or stored is the value of the global locale (I18n.locale or similar) ● Models can be queried using translated attributes (like normal column-based attributes)

One Attribute, Many Languages class Post < ApplicationRecord translates :title end post = Post.new I18n.locale = :en post.title = "Translating with Mobility" I18n.locale = :fr post.title = "Traduction avec Mobilité" post.save post = Post.first I18n.locale = :en post.title #=> "Translating with Mobility" I18n.locale = :fr post.title #=> "Traduction avec Mobilité" class Post < ApplicationRecord translates :title end post = Post.new I18n.locale = :en post.title = "Translating with Mobility" I18n.locale = :fr post.title = "Traduction avec Mobilité" post.save post = Post.first I18n.locale = :en post.title #=> "Translating with Mobility" I18n.locale = :fr post.title #=> "Traduction avec Mobilité"

Fallbacks I18n.locale = :en post.title #=> nil I18n.locale = :en post.title #=> nil I18n.locale = :en post.title #=> " タイトル " I18n.locale = :en post.title #=> " タイトル " No Fallbacks :en => :ja post = Post.new I18n.locale = :ja post.title = " タイトル " post.save post = Post.new I18n.locale = :ja post.title = " タイトル " post.save

Translations Cache/Stash post = Post.first I18n.locale = :ja post.title #=> " タイトル " I18n.locale = :en post.title #=> "Title" I18n.locale = :fr post.title #=> "Titre" post = Post.first I18n.locale = :ja post.title #=> " タイトル " I18n.locale = :en post.title #=> "Title" I18n.locale = :fr post.title #=> "Titre" Locale Value ja タイトル en Title fr Titre “Title” Cache

Locale Accessors class Post < ApplicationRecord translates :title, locale_accessors: [:en, :ja] end post = Post.new post.title_ja = " タイトル 2" post.title_en = "Title 2" post.title_ja #=> " タイトル 2" post.title_en #=> "Title 2" class Post < ApplicationRecord translates :title, locale_accessors: [:en, :ja] end post = Post.new post.title_ja = " タイトル 2" post.title_en = "Title 2" post.title_ja #=> " タイトル 2" post.title_en #=> "Title 2"

Dirty Attribute Tracking ● Tracking of translated attribute changes – title_was – title_changes – title_changed? – ...

Bonus Points: Query by Translation I18n.locale = :en Post.find_by(title: "Translating with Mobility") #=> (post with English title "Translating...") I18n.locale = :fr Post.find_by(title: "Traduction avec Mobilité") #=> (returns post with French title "Traduction...") I18n.locale = :en Post.find_by(title: "Translating with Mobility") #=> (post with English title "Translating...") I18n.locale = :fr Post.find_by(title: "Traduction avec Mobilité") #=> (returns post with French title "Traduction...") ● Override query methods to return records by translated attribute ● Hides details of storage strategy from query interface

Storage Strategies

What is a Storage Strategy? translated attribute translated attribute Persistence Layer describes a mapping between translated attributes and persistence layer

Translatable Columns

Translatable Columns ● Good – Easy to understand – Easy to construct queries – Good performance (no joins) ● Bad – Every new language requires a migration for every translated model – Columns created for every locale even if there is no translation for a locale

Model Translation Tables

Model Translation Tables ● Translation table for each model, with: – locale column – foreign key pointing to model table (post_id) – one or more translated columns ● has_many association between model and translations ● Translation table records are created/updated when attribute is set and model is saved has_many :translations, class_name: PostTranslation has_many :translations, class_name: PostTranslation

Model Translation Tables ● Good – Easily scales with number of languages (no new migrations to add locale) – Space usage is more efficient (only create record if translation is stored) ● Bad – Querying may be less performant (need to join translation table) – Code is more complex due to joining – Migration to create translation table/column for every model/attribute

Shared Translation Table(s)

Shared Translation Table(s) ● One or more translation tables, with: – locale column – foreign key pointing to model id (translatable_id) – column holding the model class (translatable_type) – column holding the name of the attribute (key) – column holding the value of the attribute (value) ● One translation table for each value type (string, text, ...) has_many :translations, as: :translatable, class_name: ::Mobility::ActiveRecord::StringTranslation has_many :translations, as: :translatable, class_name: ::Mobility::ActiveRecord::StringTranslation

Shared Translation Table(s) ● Good – No additional migrations at all, so easy to scale to new locales + new models – Space usage is relatively efficient, scales with number of locales + models ● Bad – Very complex querying – All translations in the same table, can be difficult to maintain a consistent database state

Serialized Translations ● Store translations as serialized hash on a single database column class Post < ApplicationRecord serialize :title_translations, Hash end post = Post.new post.title_translations = { "en" => "Translating with Mobility" } post.save class Post < ApplicationRecord serialize :title_translations, Hash end post = Post.new post.title_translations = { "en" => "Translating with Mobility" } post.save

Serialized Translations ● Good – Scales well with locales (no additional migrations) – Easy to add translations to model (simply add text column for each translated attribute) – Simple to access ● Bad – Querying is not possible – Loading one translation requires loading all translations at once (serialized column)

Hstore/Jsonb (PostgreSQL) ● Similar to serialized strategy, except that querying is possible and can be fast ● Database-dependent strategy, highly coupled to persistence layer ● (Jsonb generally preferred over Hstore) Post.where(title: "foo").to_sql #=> SELECT "posts".* FROM "posts" WHERE ("posts"."title" @> '{"en":"foo"}') Post.where(title: "foo").to_sql #=> SELECT "posts".* FROM "posts" WHERE ("posts"."title" @> '{"en":"foo"}')

Other Possibilities ● Stored in external service, accessed via API ● File storage ● NoSQL database ● Cache storage (Redis/Memcache/etc) ● New forms of storage we cannot yet imagine

Mobility

Translation with Mobility ● Existing solutions – couple implementation to storage strategy – couple implementation to ORM (ActiveRecord) and framework (Rails) – have similar but not identical interfaces ● What I wanted: – Support customizable, swappable storage solutions – No hard dependency on AR or Rails – Interface that is consistent and unified

Mobility: A Translation Framework ● New approach to translation problem: – One translation interface, swappable storage backends – Supports all existing strategies, plus shared tables (“key value”) – Supports both ActiveRecord and Sequel ORM ● Like the OmniAuth of Translation

Getting Started bundle exec rails g mobility:install bundle exec rails g mobility:install class Post < ApplicationRecord include Mobility translates :title, :body end class Post < ApplicationRecord include Mobility translates :title, :body end Mobility.configure do |config| config.default_backend = :key_value config.accessor_method = :translates config.query_method = :i18n end Mobility.configure do |config| config.default_backend = :key_value config.accessor_method = :translates config.query_method = :i18n end :key_value :table :column :jsonb :hstore :serialized

Separation of Responsibilities Storage Layer ● Implements atomic methods for reading, writing ● Sets up model ● Extends query scope to support backend Storage Layer ● Implements atomic methods for reading, writing ● Sets up model ● Extends query scope to support backend Interface Layer ● Reading and writing translations ● Fallbacks, cache, dirty tracking ● Locale accessors Interface Layer ● Reading and writing translations ● Fallbacks, cache, dirty tracking ● Locale accessors

“Pluggable” Backends ● Backend encapsulates a storage strategy for a single attribute (or set of attributes) ● All backends automatically support locale accessors, fallbacks, dirty tracking, cache ● Backends are tested against shared test suite to ensure consistency

What’s a Backend? class MyBackend include Mobility::Backend # attr_reader :attribute, :model def read(locale, **options) # Read attribute in locale. end def write(locale, value, **options) # Write value to attribute in locale. end def self.configure(options) # Normalize options (e.g. set any defaults) end setup do |attributes, options| # Do something with attributes and options in context of model class. end end class MyBackend include Mobility::Backend # attr_reader :attribute, :model def read(locale, **options) # Read attribute in locale. end def write(locale, value, **options) # Write value to attribute in locale. end def self.configure(options) # Normalize options (e.g. set any defaults) end setup do |attributes, options| # Do something with attributes and options in context of model class. end end

Mobility in Action class Post include Mobility translates :title, backend: :column, locale_accessors: [:en, :ja] translates :body, backend: :key_value, locale_accessors: [:fr] end post = Post.new post.title = “Title” post.body = “Lorum Ipsum...” post.title #=> “Title” post.title_backend #=> #<# post.title_en #=> “Title” post.title_ja #=> nil post.title_fr #=> NoMethodError post.body #=> “Lorum Ipsum...” post.body_backend #=> #<# post.body_en #=> NoMethodError post.body_ja #=> NoMethodError post.body_fr #=> nil class Post include Mobility translates :title, backend: :column, locale_accessors: [:en, :ja] translates :body, backend: :key_value, locale_accessors: [:fr] end post = Post.new post.title = “Title” post.body = “Lorum Ipsum...” post.title #=> “Title” post.title_backend #=> #<# post.title_en #=> “Title” post.title_ja #=> nil post.title_fr #=> NoMethodError post.body #=> “Lorum Ipsum...” post.body_backend #=> #<# post.body_en #=> NoMethodError post.body_ja #=> NoMethodError post.body_fr #=> nil

Query Methods Too (?!) class Post include Mobility translates :title, backend: :column, locale_accessors: [:en, :ja] translates :body, backend: :key_value, locale_accessors: [:fr] end post = Post.new post.title = “Title” post.body = “Lorum Ipsum...” post.save Post.i18n.where(title: “Title”, body: “Lorum Ipsum...”) # SELECT "posts".* FROM "posts" # INNER JOIN "mobility_text_translations" "body_mobility_text_translations" # ON "body_mobility_text_translations"."key" = 'body' # AND "body_mobility_text_translations"."locale" = 'en' # AND "body_mobility_text_translations"."translatable_type" = 'Post' # AND "body_mobility_text_translations"."translatable_id" = "posts"."id" # WHERE "body_mobility_text_translations"."value" = 'Lorum Ipsum...' # AND "posts"."title_en" = 'Title' class Post include Mobility translates :title, backend: :column, locale_accessors: [:en, :ja] translates :body, backend: :key_value, locale_accessors: [:fr] end post = Post.new post.title = “Title” post.body = “Lorum Ipsum...” post.save Post.i18n.where(title: “Title”, body: “Lorum Ipsum...”) # SELECT "posts".* FROM "posts" # INNER JOIN "mobility_text_translations" "body_mobility_text_translations" # ON "body_mobility_text_translations"."key" = 'body' # AND "body_mobility_text_translations"."locale" = 'en' # AND "body_mobility_text_translations"."translatable_type" = 'Post' # AND "body_mobility_text_translations"."translatable_id" = "posts"."id" # WHERE "body_mobility_text_translations"."value" = 'Lorum Ipsum...' # AND "posts"."title_en" = 'Title'

Under the Hood class Post translates :title, backend: :key_value, locale_accessors: [:en, :ja] end class Post translates :title, backend: :key_value, locale_accessors: [:en, :ja] end class Post include Mobility::Attributes.new(:accessor, [:title], backend: :key_value, model_class: Post) end class Post include Mobility::Attributes.new(:accessor, [:title], backend: :key_value, model_class: Post) end It’s a Module Builder!

Module Builder class Attributes < Module def initialize(method, *attributes) # ... attributes.each do |attribute| define_method attribute do # ... end # ... end end end class Post include Attributes.new(:accessor, [:title]) end class Attributes < Module def initialize(method, *attributes) # ... attributes.each do |attribute| define_method attribute do # ... end # ... end end end class Post include Attributes.new(:accessor, [:title]) end 1) Subclass Module 2) Define module methods in an initializer 3) Include instance of Module subclass in other classes

No content

“Plugging in”: Building a Backend class Post translates :title, fallbacks: true, dirty: true translates :subtitle, backend: :column translates :body, locale_accessors: [:en, :de], cache: false end class Post translates :title, fallbacks: true, dirty: true translates :subtitle, backend: :column translates :body, locale_accessors: [:en, :de], cache: false end backend_class.include Mobility::Backend::Fallbacks backend_class.include Mobility::Backend::ActiveModel::Dirty @backend_class = Class.new(Mobility::Backend::ActiveRecord::Column) include Mobility::LocaleAccessors.new(:body, [:en, :de]) Another Module Builder!

“Plugging in”: Building a Backend module Mobility module Fallbacks # ... def read(locale, **options) fallback = options.delete(:fallback) return super if fallback == false || (fallback.nil? && fallbacks.nil?) (fallback ? [locale, *fallback] : fallbacks[locale]).detect do |fallback_locale| value = super(fallback_locale, **options) break value if value.present? end end end end module Mobility module Fallbacks # ... def read(locale, **options) fallback = options.delete(:fallback) return super if fallback == false || (fallback.nil? && fallbacks.nil?) (fallback ? [locale, *fallback] : fallbacks[locale]).detect do |fallback_locale| value = super(fallback_locale, **options) break value if value.present? end end end end ● Modules included into backend (fallbacks, cache, dirty) override read and/or write methods to add functionality Find first non-nil value of attribute

“Plugging In”: Model Setup class Post translates :title, fallbacks: true, dirty: true translates :subtitle, backend: :column translates :body, locale_accessors: [:en, :de], cache: false end class Post translates :title, fallbacks: true, dirty: true translates :subtitle, backend: :column translates :body, locale_accessors: [:en, :de], cache: false end class Post has_many :mobility_text_translations, where: { key: [:title, :body] }, as: :translatable, class_name: ::Mobility::ActiveRecord::TextTranslation end class Post has_many :mobility_text_translations, where: { key: [:title, :body] }, as: :translatable, class_name: ::Mobility::ActiveRecord::TextTranslation end backend_class.setup_model(Post, :title, fallbacks: true, dirty: true)

Reading a Translated Attribute I18n.locale = :fr post.title I18n.locale = :fr post.title post.title_backend.read(:fr) post.title_backend.read(:fr) post.read_attribute(:title_fr) post.read_attribute(:title_fr) post.translations.find { |t| t.locale == :fr }.title post.translations.find { |t| t.locale == :fr }.title Translatable Columns Backend Model Translations Backend

Recap (The “Sales Pitch”) ● Mobility: – encapsulates storage strategy into a single class – supports every storage strategy – supports multiple strategies in a single model – supports all features (fallbacks, dirty tracking, caching, locale accessors, etc) at the attribute level – can be used to design new backends using flexible interface ● Mobility keeps all your doors open

Looking Ahead

The Situation Today ● I18n is ubiquitous ● Translation is essential, but (almost) always an afterthought ● Gems lock you into a specific storage strategy: – Globalize? Locked to model translation tables – Traco? Locked to translatable columns – json_translate? Locked to PostgreSQL jsonb

Mobility as a Translation Layer ● For application developers: – simplifies and standardizes content translation – does not lock you into a single storage strategy ● For developers of other gems: – A single Mobility integration and will support all current and future storage strategies – Example: friendly_id-mobility

One Integration to Rule them All Translatable Columns Model Translation Tables Shared Translation Tables Serialized Column Hstore/Jsonb Storage Strategies Gem Integrations ... Mobility Mobility FriendlyId Spree / Solidus ActiveAdmin Paperclip ...

Roadmap ● Bug fixing, Rails 5.1 full support ● Performance Testing ● Integrations – Spree/Solidus ● Support Hanami as ORM ● API for Mobility extensions – Machine translation as fallback

Thanks!