Share
Embed
Translating with Mobility
Presentation at the Montreal Ruby Meetup on June 20, 2017.
Chris Salzberg
June 20, 2017
Transcript
52 slides
-
About Me • Writer, programmer, translator • VP of Engineering
at Degica • @shioyama on github, twitter, etc • I program in Ruby • I live in Tokyo, Japan • I am the author of Mobility • I blog at dejimata.com • I coined “Module Builder Pattern” -
One Problem, Many Solutions • How many languages? • How
many translated models? • Growing application? Adding new models? • What’s your database? • What are your performance requirements? • How good are you at Arel? -
Gems and Strategies • Translation Strategies and Gems: – Translatable
Columns (Traco) – Model Translation Tables (Globalize) – Shared Translation Table(s) – Serialized Columns (Multilang) – Hstore/Jsonb (json_translate, multilang_hstore, ...) • Each gem is coupled to a storage strategy • Strategies are (conceptually) independent of ORM -
Plan for Today’s Talk 1. Translation Interface – How do
you read and write translations? – What other features are supported? 2. Storage Strategies – How are translations actually stored? – Good and bad of each strategy 3. Mobility 4. Looking Ahead -
Common Features • Translations are fetched and stored using (virtual)
attribute accessors • Language fetched or stored is the value of the global locale (I18n.locale or similar) • Models can be queried using translated attributes (like normal column-based attributes) -
One Attribute, Many Languages class Post < ApplicationRecord translates :title
end post = Post.new I18n.locale = :en post.title = "Translating with Mobility" I18n.locale = :fr post.title = "Traduction avec Mobilité" post.save post = Post.first I18n.locale = :en post.title #=> "Translating with Mobility" I18n.locale = :fr post.title #=> "Traduction avec Mobilité" class Post < ApplicationRecord translates :title end post = Post.new I18n.locale = :en post.title = "Translating with Mobility" I18n.locale = :fr post.title = "Traduction avec Mobilité" post.save post = Post.first I18n.locale = :en post.title #=> "Translating with Mobility" I18n.locale = :fr post.title #=> "Traduction avec Mobilité" -
Fallbacks I18n.locale = :en post.title #=> nil I18n.locale = :en
post.title #=> nil I18n.locale = :en post.title #=> " タイトル " I18n.locale = :en post.title #=> " タイトル " No Fallbacks :en => :ja post = Post.new I18n.locale = :ja post.title = " タイトル " post.save post = Post.new I18n.locale = :ja post.title = " タイトル " post.save -
Translations Cache/Stash post = Post.first I18n.locale = :ja post.title #=>
" タイトル " I18n.locale = :en post.title #=> "Title" I18n.locale = :fr post.title #=> "Titre" post = Post.first I18n.locale = :ja post.title #=> " タイトル " I18n.locale = :en post.title #=> "Title" I18n.locale = :fr post.title #=> "Titre" Locale Value ja タイトル en Title fr Titre “Title” Cache -
Locale Accessors class Post < ApplicationRecord translates :title, locale_accessors: [:en,
:ja] end post = Post.new post.title_ja = " タイトル 2" post.title_en = "Title 2" post.title_ja #=> " タイトル 2" post.title_en #=> "Title 2" class Post < ApplicationRecord translates :title, locale_accessors: [:en, :ja] end post = Post.new post.title_ja = " タイトル 2" post.title_en = "Title 2" post.title_ja #=> " タイトル 2" post.title_en #=> "Title 2" -
Dirty Attribute Tracking • Tracking of translated attribute changes –
title_was – title_changes – title_changed? – ... -
Bonus Points: Query by Translation I18n.locale = :en Post.find_by(title: "Translating
with Mobility") #=> (post with English title "Translating...") I18n.locale = :fr Post.find_by(title: "Traduction avec Mobilité") #=> (returns post with French title "Traduction...") I18n.locale = :en Post.find_by(title: "Translating with Mobility") #=> (post with English title "Translating...") I18n.locale = :fr Post.find_by(title: "Traduction avec Mobilité") #=> (returns post with French title "Traduction...") • Override query methods to return records by translated attribute • Hides details of storage strategy from query interface -
What is a Storage Strategy? translated attribute translated attribute Persistence
Layer describes a mapping between translated attributes and persistence layer -
Translatable Columns • Good – Easy to understand – Easy
to construct queries – Good performance (no joins) • Bad – Every new language requires a migration for every translated model – Columns created for every locale even if there is no translation for a locale -
Model Translation Tables • Translation table for each model, with:
– locale column – foreign key pointing to model table (post_id) – one or more translated columns • has_many association between model and translations • Translation table records are created/updated when attribute is set and model is saved has_many :translations, class_name: PostTranslation has_many :translations, class_name: PostTranslation -
Model Translation Tables • Good – Easily scales with number
of languages (no new migrations to add locale) – Space usage is more efficient (only create record if translation is stored) • Bad – Querying may be less performant (need to join translation table) – Code is more complex due to joining – Migration to create translation table/column for every model/attribute -
Shared Translation Table(s) • One or more translation tables, with:
– locale column – foreign key pointing to model id (translatable_id) – column holding the model class (translatable_type) – column holding the name of the attribute (key) – column holding the value of the attribute (value) • One translation table for each value type (string, text, ...) has_many :translations, as: :translatable, class_name: ::Mobility::ActiveRecord::StringTranslation has_many :translations, as: :translatable, class_name: ::Mobility::ActiveRecord::StringTranslation -
Shared Translation Table(s) • Good – No additional migrations at
all, so easy to scale to new locales + new models – Space usage is relatively efficient, scales with number of locales + models • Bad – Very complex querying – All translations in the same table, can be difficult to maintain a consistent database state -
Serialized Translations • Store translations as serialized hash on a
single database column class Post < ApplicationRecord serialize :title_translations, Hash end post = Post.new post.title_translations = { "en" => "Translating with Mobility" } post.save class Post < ApplicationRecord serialize :title_translations, Hash end post = Post.new post.title_translations = { "en" => "Translating with Mobility" } post.save -
Serialized Translations • Good – Scales well with locales (no
additional migrations) – Easy to add translations to model (simply add text column for each translated attribute) – Simple to access • Bad – Querying is not possible – Loading one translation requires loading all translations at once (serialized column) -
Hstore/Jsonb (PostgreSQL) • Similar to serialized strategy, except that querying
is possible and can be fast • Database-dependent strategy, highly coupled to persistence layer • (Jsonb generally preferred over Hstore) Post.where(title: "foo").to_sql #=> SELECT "posts".* FROM "posts" WHERE ("posts"."title" @> '{"en":"foo"}') Post.where(title: "foo").to_sql #=> SELECT "posts".* FROM "posts" WHERE ("posts"."title" @> '{"en":"foo"}') -
Other Possibilities • Stored in external service, accessed via API
• File storage • NoSQL database • Cache storage (Redis/Memcache/etc) • New forms of storage we cannot yet imagine -
Translation with Mobility • Existing solutions – couple implementation to
storage strategy – couple implementation to ORM (ActiveRecord) and framework (Rails) – have similar but not identical interfaces • What I wanted: – Support customizable, swappable storage solutions – No hard dependency on AR or Rails – Interface that is consistent and unified -
Mobility: A Translation Framework • New approach to translation problem:
– One translation interface, swappable storage backends – Supports all existing strategies, plus shared tables (“key value”) – Supports both ActiveRecord and Sequel ORM • Like the OmniAuth of Translation -
Getting Started bundle exec rails g mobility:install bundle exec rails
g mobility:install class Post < ApplicationRecord include Mobility translates :title, :body end class Post < ApplicationRecord include Mobility translates :title, :body end Mobility.configure do |config| config.default_backend = :key_value config.accessor_method = :translates config.query_method = :i18n end Mobility.configure do |config| config.default_backend = :key_value config.accessor_method = :translates config.query_method = :i18n end :key_value :table :column :jsonb :hstore :serialized -
Separation of Responsibilities Storage Layer • Implements atomic methods for
reading, writing • Sets up model • Extends query scope to support backend Storage Layer • Implements atomic methods for reading, writing • Sets up model • Extends query scope to support backend Interface Layer • Reading and writing translations • Fallbacks, cache, dirty tracking • Locale accessors Interface Layer • Reading and writing translations • Fallbacks, cache, dirty tracking • Locale accessors -
“Pluggable” Backends • Backend encapsulates a storage strategy for a
single attribute (or set of attributes) • All backends automatically support locale accessors, fallbacks, dirty tracking, cache • Backends are tested against shared test suite to ensure consistency -
What’s a Backend? class MyBackend include Mobility::Backend # attr_reader :attribute,
:model def read(locale, **options) # Read attribute in locale. end def write(locale, value, **options) # Write value to attribute in locale. end def self.configure(options) # Normalize options (e.g. set any defaults) end setup do |attributes, options| # Do something with attributes and options in context of model class. end end class MyBackend include Mobility::Backend # attr_reader :attribute, :model def read(locale, **options) # Read attribute in locale. end def write(locale, value, **options) # Write value to attribute in locale. end def self.configure(options) # Normalize options (e.g. set any defaults) end setup do |attributes, options| # Do something with attributes and options in context of model class. end end -
Mobility in Action class Post include Mobility translates :title, backend:
:column, locale_accessors: [:en, :ja] translates :body, backend: :key_value, locale_accessors: [:fr] end post = Post.new post.title = “Title” post.body = “Lorum Ipsum...” post.title #=> “Title” post.title_backend #=> #<#<Class:... @fallbacks=nil @cache={...} > post.title_en #=> “Title” post.title_ja #=> nil post.title_fr #=> NoMethodError post.body #=> “Lorum Ipsum...” post.body_backend #=> #<#<Class:... @fallbacks=nil @cache={...} > post.body_en #=> NoMethodError post.body_ja #=> NoMethodError post.body_fr #=> nil class Post include Mobility translates :title, backend: :column, locale_accessors: [:en, :ja] translates :body, backend: :key_value, locale_accessors: [:fr] end post = Post.new post.title = “Title” post.body = “Lorum Ipsum...” post.title #=> “Title” post.title_backend #=> #<#<Class:... @fallbacks=nil @cache={...} > post.title_en #=> “Title” post.title_ja #=> nil post.title_fr #=> NoMethodError post.body #=> “Lorum Ipsum...” post.body_backend #=> #<#<Class:... @fallbacks=nil @cache={...} > post.body_en #=> NoMethodError post.body_ja #=> NoMethodError post.body_fr #=> nil -
Query Methods Too (?!) class Post include Mobility translates :title,
backend: :column, locale_accessors: [:en, :ja] translates :body, backend: :key_value, locale_accessors: [:fr] end post = Post.new post.title = “Title” post.body = “Lorum Ipsum...” post.save Post.i18n.where(title: “Title”, body: “Lorum Ipsum...”) # SELECT "posts".* FROM "posts" # INNER JOIN "mobility_text_translations" "body_mobility_text_translations" # ON "body_mobility_text_translations"."key" = 'body' # AND "body_mobility_text_translations"."locale" = 'en' # AND "body_mobility_text_translations"."translatable_type" = 'Post' # AND "body_mobility_text_translations"."translatable_id" = "posts"."id" # WHERE "body_mobility_text_translations"."value" = 'Lorum Ipsum...' # AND "posts"."title_en" = 'Title' class Post include Mobility translates :title, backend: :column, locale_accessors: [:en, :ja] translates :body, backend: :key_value, locale_accessors: [:fr] end post = Post.new post.title = “Title” post.body = “Lorum Ipsum...” post.save Post.i18n.where(title: “Title”, body: “Lorum Ipsum...”) # SELECT "posts".* FROM "posts" # INNER JOIN "mobility_text_translations" "body_mobility_text_translations" # ON "body_mobility_text_translations"."key" = 'body' # AND "body_mobility_text_translations"."locale" = 'en' # AND "body_mobility_text_translations"."translatable_type" = 'Post' # AND "body_mobility_text_translations"."translatable_id" = "posts"."id" # WHERE "body_mobility_text_translations"."value" = 'Lorum Ipsum...' # AND "posts"."title_en" = 'Title' -
Under the Hood class Post translates :title, backend: :key_value, locale_accessors:
[:en, :ja] end class Post translates :title, backend: :key_value, locale_accessors: [:en, :ja] end class Post include Mobility::Attributes.new(:accessor, [:title], backend: :key_value, model_class: Post) end class Post include Mobility::Attributes.new(:accessor, [:title], backend: :key_value, model_class: Post) end It’s a Module Builder! -
Module Builder class Attributes < Module def initialize(method, *attributes) #
... attributes.each do |attribute| define_method attribute do # ... end # ... end end end class Post include Attributes.new(:accessor, [:title]) end class Attributes < Module def initialize(method, *attributes) # ... attributes.each do |attribute| define_method attribute do # ... end # ... end end end class Post include Attributes.new(:accessor, [:title]) end 1) Subclass Module 2) Define module methods in an initializer 3) Include instance of Module subclass in other classes -
“Plugging in”: Building a Backend class Post translates :title, fallbacks:
true, dirty: true translates :subtitle, backend: :column translates :body, locale_accessors: [:en, :de], cache: false end class Post translates :title, fallbacks: true, dirty: true translates :subtitle, backend: :column translates :body, locale_accessors: [:en, :de], cache: false end backend_class.include Mobility::Backend::Fallbacks backend_class.include Mobility::Backend::ActiveModel::Dirty @backend_class = Class.new(Mobility::Backend::ActiveRecord::Column) include Mobility::LocaleAccessors.new(:body, [:en, :de]) Another Module Builder! -
“Plugging in”: Building a Backend module Mobility module Fallbacks #
... def read(locale, **options) fallback = options.delete(:fallback) return super if fallback == false || (fallback.nil? && fallbacks.nil?) (fallback ? [locale, *fallback] : fallbacks[locale]).detect do |fallback_locale| value = super(fallback_locale, **options) break value if value.present? end end end end module Mobility module Fallbacks # ... def read(locale, **options) fallback = options.delete(:fallback) return super if fallback == false || (fallback.nil? && fallbacks.nil?) (fallback ? [locale, *fallback] : fallbacks[locale]).detect do |fallback_locale| value = super(fallback_locale, **options) break value if value.present? end end end end • Modules included into backend (fallbacks, cache, dirty) override read and/or write methods to add functionality Find first non-nil value of attribute -
“Plugging In”: Model Setup class Post translates :title, fallbacks: true,
dirty: true translates :subtitle, backend: :column translates :body, locale_accessors: [:en, :de], cache: false end class Post translates :title, fallbacks: true, dirty: true translates :subtitle, backend: :column translates :body, locale_accessors: [:en, :de], cache: false end class Post has_many :mobility_text_translations, where: { key: [:title, :body] }, as: :translatable, class_name: ::Mobility::ActiveRecord::TextTranslation end class Post has_many :mobility_text_translations, where: { key: [:title, :body] }, as: :translatable, class_name: ::Mobility::ActiveRecord::TextTranslation end backend_class.setup_model(Post, :title, fallbacks: true, dirty: true) -
Reading a Translated Attribute I18n.locale = :fr post.title I18n.locale =
:fr post.title post.title_backend.read(:fr) post.title_backend.read(:fr) post.read_attribute(:title_fr) post.read_attribute(:title_fr) post.translations.find { |t| t.locale == :fr }.title post.translations.find { |t| t.locale == :fr }.title Translatable Columns Backend Model Translations Backend -
Recap (The “Sales Pitch”) • Mobility: – encapsulates storage strategy
into a single class – supports every storage strategy – supports multiple strategies in a single model – supports all features (fallbacks, dirty tracking, caching, locale accessors, etc) at the attribute level – can be used to design new backends using flexible interface • Mobility keeps all your doors open -
The Situation Today • I18n is ubiquitous • Translation is
essential, but (almost) always an afterthought • Gems lock you into a specific storage strategy: – Globalize? Locked to model translation tables – Traco? Locked to translatable columns – json_translate? Locked to PostgreSQL jsonb -
Mobility as a Translation Layer • For application developers: –
simplifies and standardizes content translation – does not lock you into a single storage strategy • For developers of other gems: – A single Mobility integration and will support all current and future storage strategies – Example: friendly_id-mobility -
One Integration to Rule them All Translatable Columns Model Translation
Tables Shared Translation Tables Serialized Column Hstore/Jsonb Storage Strategies Gem Integrations ... Mobility Mobility FriendlyId Spree / Solidus ActiveAdmin Paperclip ... -
Roadmap • Bug fixing, Rails 5.1 full support • Performance
Testing • Integrations – Spree/Solidus • Support Hanami as ORM • API for Mobility extensions – Machine translation as fallback