$30 off During Our Annual Pro Sale. View Details »

Translating with Mobility

Translating with Mobility

Presentation at the Montreal Ruby Meetup on June 20, 2017.

Chris Salzberg

June 20, 2017
Tweet

More Decks by Chris Salzberg

Other Decks in Programming

Transcript

  1. Translating with Mobility Chris Salzberg (@shioyama)

  2. About Me • Writer, programmer, translator • VP of Engineering

    at Degica • @shioyama on github, twitter, etc • I program in Ruby • I live in Tokyo, Japan • I am the author of Mobility • I blog at dejimata.com • I coined “Module Builder Pattern”
  3. How I got here today Montreal Amsterdam Tokyo

  4. Translation is something that seems easy, but is actually very

    hard
  5. One Problem, Many Solutions • How many languages? • How

    many translated models? • Growing application? Adding new models? • What’s your database? • What are your performance requirements? • How good are you at Arel?
  6. Gems and Strategies • Translation Strategies and Gems: – Translatable

    Columns (Traco) – Model Translation Tables (Globalize) – Shared Translation Table(s) – Serialized Columns (Multilang) – Hstore/Jsonb (json_translate, multilang_hstore, ...) • Each gem is coupled to a storage strategy • Strategies are (conceptually) independent of ORM
  7. Plan for Today’s Talk 1. Translation Interface – How do

    you read and write translations? – What other features are supported? 2. Storage Strategies – How are translations actually stored? – Good and bad of each strategy 3. Mobility 4. Looking Ahead
  8. Translation Interface

  9. Common Features • Translations are fetched and stored using (virtual)

    attribute accessors • Language fetched or stored is the value of the global locale (I18n.locale or similar) • Models can be queried using translated attributes (like normal column-based attributes)
  10. One Attribute, Many Languages class Post < ApplicationRecord translates :title

    end post = Post.new I18n.locale = :en post.title = "Translating with Mobility" I18n.locale = :fr post.title = "Traduction avec Mobilité" post.save post = Post.first I18n.locale = :en post.title #=> "Translating with Mobility" I18n.locale = :fr post.title #=> "Traduction avec Mobilité" class Post < ApplicationRecord translates :title end post = Post.new I18n.locale = :en post.title = "Translating with Mobility" I18n.locale = :fr post.title = "Traduction avec Mobilité" post.save post = Post.first I18n.locale = :en post.title #=> "Translating with Mobility" I18n.locale = :fr post.title #=> "Traduction avec Mobilité"
  11. Fallbacks I18n.locale = :en post.title #=> nil I18n.locale = :en

    post.title #=> nil I18n.locale = :en post.title #=> " タイトル " I18n.locale = :en post.title #=> " タイトル " No Fallbacks :en => :ja post = Post.new I18n.locale = :ja post.title = " タイトル " post.save post = Post.new I18n.locale = :ja post.title = " タイトル " post.save
  12. Translations Cache/Stash post = Post.first I18n.locale = :ja post.title #=>

    " タイトル " I18n.locale = :en post.title #=> "Title" I18n.locale = :fr post.title #=> "Titre" post = Post.first I18n.locale = :ja post.title #=> " タイトル " I18n.locale = :en post.title #=> "Title" I18n.locale = :fr post.title #=> "Titre" Locale Value ja タイトル en Title fr Titre “Title” Cache
  13. Locale Accessors class Post < ApplicationRecord translates :title, locale_accessors: [:en,

    :ja] end post = Post.new post.title_ja = " タイトル 2" post.title_en = "Title 2" post.title_ja #=> " タイトル 2" post.title_en #=> "Title 2" class Post < ApplicationRecord translates :title, locale_accessors: [:en, :ja] end post = Post.new post.title_ja = " タイトル 2" post.title_en = "Title 2" post.title_ja #=> " タイトル 2" post.title_en #=> "Title 2"
  14. Dirty Attribute Tracking • Tracking of translated attribute changes –

    title_was – title_changes – title_changed? – ...
  15. Bonus Points: Query by Translation I18n.locale = :en Post.find_by(title: "Translating

    with Mobility") #=> (post with English title "Translating...") I18n.locale = :fr Post.find_by(title: "Traduction avec Mobilité") #=> (returns post with French title "Traduction...") I18n.locale = :en Post.find_by(title: "Translating with Mobility") #=> (post with English title "Translating...") I18n.locale = :fr Post.find_by(title: "Traduction avec Mobilité") #=> (returns post with French title "Traduction...") • Override query methods to return records by translated attribute • Hides details of storage strategy from query interface
  16. Storage Strategies

  17. What is a Storage Strategy? translated attribute translated attribute Persistence

    Layer describes a mapping between translated attributes and persistence layer
  18. Translatable Columns

  19. Translatable Columns • Good – Easy to understand – Easy

    to construct queries – Good performance (no joins) • Bad – Every new language requires a migration for every translated model – Columns created for every locale even if there is no translation for a locale
  20. Model Translation Tables

  21. Model Translation Tables • Translation table for each model, with:

    – locale column – foreign key pointing to model table (post_id) – one or more translated columns • has_many association between model and translations • Translation table records are created/updated when attribute is set and model is saved has_many :translations, class_name: PostTranslation has_many :translations, class_name: PostTranslation
  22. Model Translation Tables • Good – Easily scales with number

    of languages (no new migrations to add locale) – Space usage is more efficient (only create record if translation is stored) • Bad – Querying may be less performant (need to join translation table) – Code is more complex due to joining – Migration to create translation table/column for every model/attribute
  23. Shared Translation Table(s)

  24. Shared Translation Table(s) • One or more translation tables, with:

    – locale column – foreign key pointing to model id (translatable_id) – column holding the model class (translatable_type) – column holding the name of the attribute (key) – column holding the value of the attribute (value) • One translation table for each value type (string, text, ...) has_many :translations, as: :translatable, class_name: ::Mobility::ActiveRecord::StringTranslation has_many :translations, as: :translatable, class_name: ::Mobility::ActiveRecord::StringTranslation
  25. Shared Translation Table(s) • Good – No additional migrations at

    all, so easy to scale to new locales + new models – Space usage is relatively efficient, scales with number of locales + models • Bad – Very complex querying – All translations in the same table, can be difficult to maintain a consistent database state
  26. Serialized Translations • Store translations as serialized hash on a

    single database column class Post < ApplicationRecord serialize :title_translations, Hash end post = Post.new post.title_translations = { "en" => "Translating with Mobility" } post.save class Post < ApplicationRecord serialize :title_translations, Hash end post = Post.new post.title_translations = { "en" => "Translating with Mobility" } post.save
  27. Serialized Translations • Good – Scales well with locales (no

    additional migrations) – Easy to add translations to model (simply add text column for each translated attribute) – Simple to access • Bad – Querying is not possible – Loading one translation requires loading all translations at once (serialized column)
  28. Hstore/Jsonb (PostgreSQL) • Similar to serialized strategy, except that querying

    is possible and can be fast • Database-dependent strategy, highly coupled to persistence layer • (Jsonb generally preferred over Hstore) Post.where(title: "foo").to_sql #=> SELECT "posts".* FROM "posts" WHERE ("posts"."title" @> '{"en":"foo"}') Post.where(title: "foo").to_sql #=> SELECT "posts".* FROM "posts" WHERE ("posts"."title" @> '{"en":"foo"}')
  29. Other Possibilities • Stored in external service, accessed via API

    • File storage • NoSQL database • Cache storage (Redis/Memcache/etc) • New forms of storage we cannot yet imagine
  30. Mobility

  31. Translation with Mobility • Existing solutions – couple implementation to

    storage strategy – couple implementation to ORM (ActiveRecord) and framework (Rails) – have similar but not identical interfaces • What I wanted: – Support customizable, swappable storage solutions – No hard dependency on AR or Rails – Interface that is consistent and unified
  32. Mobility: A Translation Framework • New approach to translation problem:

    – One translation interface, swappable storage backends – Supports all existing strategies, plus shared tables (“key value”) – Supports both ActiveRecord and Sequel ORM • Like the OmniAuth of Translation
  33. Getting Started bundle exec rails g mobility:install bundle exec rails

    g mobility:install class Post < ApplicationRecord include Mobility translates :title, :body end class Post < ApplicationRecord include Mobility translates :title, :body end Mobility.configure do |config| config.default_backend = :key_value config.accessor_method = :translates config.query_method = :i18n end Mobility.configure do |config| config.default_backend = :key_value config.accessor_method = :translates config.query_method = :i18n end :key_value :table :column :jsonb :hstore :serialized
  34. Separation of Responsibilities Storage Layer • Implements atomic methods for

    reading, writing • Sets up model • Extends query scope to support backend Storage Layer • Implements atomic methods for reading, writing • Sets up model • Extends query scope to support backend Interface Layer • Reading and writing translations • Fallbacks, cache, dirty tracking • Locale accessors Interface Layer • Reading and writing translations • Fallbacks, cache, dirty tracking • Locale accessors
  35. “Pluggable” Backends • Backend encapsulates a storage strategy for a

    single attribute (or set of attributes) • All backends automatically support locale accessors, fallbacks, dirty tracking, cache • Backends are tested against shared test suite to ensure consistency
  36. What’s a Backend? class MyBackend include Mobility::Backend # attr_reader :attribute,

    :model def read(locale, **options) # Read attribute in locale. end def write(locale, value, **options) # Write value to attribute in locale. end def self.configure(options) # Normalize options (e.g. set any defaults) end setup do |attributes, options| # Do something with attributes and options in context of model class. end end class MyBackend include Mobility::Backend # attr_reader :attribute, :model def read(locale, **options) # Read attribute in locale. end def write(locale, value, **options) # Write value to attribute in locale. end def self.configure(options) # Normalize options (e.g. set any defaults) end setup do |attributes, options| # Do something with attributes and options in context of model class. end end
  37. Mobility in Action class Post include Mobility translates :title, backend:

    :column, locale_accessors: [:en, :ja] translates :body, backend: :key_value, locale_accessors: [:fr] end post = Post.new post.title = “Title” post.body = “Lorum Ipsum...” post.title #=> “Title” post.title_backend #=> #<#<Class:... @fallbacks=nil @cache={...} > post.title_en #=> “Title” post.title_ja #=> nil post.title_fr #=> NoMethodError post.body #=> “Lorum Ipsum...” post.body_backend #=> #<#<Class:... @fallbacks=nil @cache={...} > post.body_en #=> NoMethodError post.body_ja #=> NoMethodError post.body_fr #=> nil class Post include Mobility translates :title, backend: :column, locale_accessors: [:en, :ja] translates :body, backend: :key_value, locale_accessors: [:fr] end post = Post.new post.title = “Title” post.body = “Lorum Ipsum...” post.title #=> “Title” post.title_backend #=> #<#<Class:... @fallbacks=nil @cache={...} > post.title_en #=> “Title” post.title_ja #=> nil post.title_fr #=> NoMethodError post.body #=> “Lorum Ipsum...” post.body_backend #=> #<#<Class:... @fallbacks=nil @cache={...} > post.body_en #=> NoMethodError post.body_ja #=> NoMethodError post.body_fr #=> nil
  38. Query Methods Too (?!) class Post include Mobility translates :title,

    backend: :column, locale_accessors: [:en, :ja] translates :body, backend: :key_value, locale_accessors: [:fr] end post = Post.new post.title = “Title” post.body = “Lorum Ipsum...” post.save Post.i18n.where(title: “Title”, body: “Lorum Ipsum...”) # SELECT "posts".* FROM "posts" # INNER JOIN "mobility_text_translations" "body_mobility_text_translations" # ON "body_mobility_text_translations"."key" = 'body' # AND "body_mobility_text_translations"."locale" = 'en' # AND "body_mobility_text_translations"."translatable_type" = 'Post' # AND "body_mobility_text_translations"."translatable_id" = "posts"."id" # WHERE "body_mobility_text_translations"."value" = 'Lorum Ipsum...' # AND "posts"."title_en" = 'Title' class Post include Mobility translates :title, backend: :column, locale_accessors: [:en, :ja] translates :body, backend: :key_value, locale_accessors: [:fr] end post = Post.new post.title = “Title” post.body = “Lorum Ipsum...” post.save Post.i18n.where(title: “Title”, body: “Lorum Ipsum...”) # SELECT "posts".* FROM "posts" # INNER JOIN "mobility_text_translations" "body_mobility_text_translations" # ON "body_mobility_text_translations"."key" = 'body' # AND "body_mobility_text_translations"."locale" = 'en' # AND "body_mobility_text_translations"."translatable_type" = 'Post' # AND "body_mobility_text_translations"."translatable_id" = "posts"."id" # WHERE "body_mobility_text_translations"."value" = 'Lorum Ipsum...' # AND "posts"."title_en" = 'Title'
  39. Under the Hood class Post translates :title, backend: :key_value, locale_accessors:

    [:en, :ja] end class Post translates :title, backend: :key_value, locale_accessors: [:en, :ja] end class Post include Mobility::Attributes.new(:accessor, [:title], backend: :key_value, model_class: Post) end class Post include Mobility::Attributes.new(:accessor, [:title], backend: :key_value, model_class: Post) end It’s a Module Builder!
  40. Module Builder class Attributes < Module def initialize(method, *attributes) #

    ... attributes.each do |attribute| define_method attribute do # ... end # ... end end end class Post include Attributes.new(:accessor, [:title]) end class Attributes < Module def initialize(method, *attributes) # ... attributes.each do |attribute| define_method attribute do # ... end # ... end end end class Post include Attributes.new(:accessor, [:title]) end 1) Subclass Module 2) Define module methods in an initializer 3) Include instance of Module subclass in other classes
  41. None
  42. “Plugging in”: Building a Backend class Post translates :title, fallbacks:

    true, dirty: true translates :subtitle, backend: :column translates :body, locale_accessors: [:en, :de], cache: false end class Post translates :title, fallbacks: true, dirty: true translates :subtitle, backend: :column translates :body, locale_accessors: [:en, :de], cache: false end backend_class.include Mobility::Backend::Fallbacks backend_class.include Mobility::Backend::ActiveModel::Dirty @backend_class = Class.new(Mobility::Backend::ActiveRecord::Column) include Mobility::LocaleAccessors.new(:body, [:en, :de]) Another Module Builder!
  43. “Plugging in”: Building a Backend module Mobility module Fallbacks #

    ... def read(locale, **options) fallback = options.delete(:fallback) return super if fallback == false || (fallback.nil? && fallbacks.nil?) (fallback ? [locale, *fallback] : fallbacks[locale]).detect do |fallback_locale| value = super(fallback_locale, **options) break value if value.present? end end end end module Mobility module Fallbacks # ... def read(locale, **options) fallback = options.delete(:fallback) return super if fallback == false || (fallback.nil? && fallbacks.nil?) (fallback ? [locale, *fallback] : fallbacks[locale]).detect do |fallback_locale| value = super(fallback_locale, **options) break value if value.present? end end end end • Modules included into backend (fallbacks, cache, dirty) override read and/or write methods to add functionality Find first non-nil value of attribute
  44. “Plugging In”: Model Setup class Post translates :title, fallbacks: true,

    dirty: true translates :subtitle, backend: :column translates :body, locale_accessors: [:en, :de], cache: false end class Post translates :title, fallbacks: true, dirty: true translates :subtitle, backend: :column translates :body, locale_accessors: [:en, :de], cache: false end class Post has_many :mobility_text_translations, where: { key: [:title, :body] }, as: :translatable, class_name: ::Mobility::ActiveRecord::TextTranslation end class Post has_many :mobility_text_translations, where: { key: [:title, :body] }, as: :translatable, class_name: ::Mobility::ActiveRecord::TextTranslation end backend_class.setup_model(Post, :title, fallbacks: true, dirty: true)
  45. Reading a Translated Attribute I18n.locale = :fr post.title I18n.locale =

    :fr post.title post.title_backend.read(:fr) post.title_backend.read(:fr) post.read_attribute(:title_fr) post.read_attribute(:title_fr) post.translations.find { |t| t.locale == :fr }.title post.translations.find { |t| t.locale == :fr }.title Translatable Columns Backend Model Translations Backend
  46. Recap (The “Sales Pitch”) • Mobility: – encapsulates storage strategy

    into a single class – supports every storage strategy – supports multiple strategies in a single model – supports all features (fallbacks, dirty tracking, caching, locale accessors, etc) at the attribute level – can be used to design new backends using flexible interface • Mobility keeps all your doors open
  47. Looking Ahead

  48. The Situation Today • I18n is ubiquitous • Translation is

    essential, but (almost) always an afterthought • Gems lock you into a specific storage strategy: – Globalize? Locked to model translation tables – Traco? Locked to translatable columns – json_translate? Locked to PostgreSQL jsonb
  49. Mobility as a Translation Layer • For application developers: –

    simplifies and standardizes content translation – does not lock you into a single storage strategy • For developers of other gems: – A single Mobility integration and will support all current and future storage strategies – Example: friendly_id-mobility
  50. One Integration to Rule them All Translatable Columns Model Translation

    Tables Shared Translation Tables Serialized Column Hstore/Jsonb Storage Strategies Gem Integrations ... Mobility Mobility FriendlyId Spree / Solidus ActiveAdmin Paperclip ...
  51. Roadmap • Bug fixing, Rails 5.1 full support • Performance

    Testing • Integrations – Spree/Solidus • Support Hanami as ORM • API for Mobility extensions – Machine translation as fallback
  52. Thanks!