Translating with Mobility

Translating with Mobility

05fdba1ae381f24512e977f8fe2697b4?s=128

Chris Salzberg

August 08, 2017
Tweet

Transcript

  1. Translating with Mobility Translating with Mobility Chris Salzberg (@shioyama)

  2. About Me • Writer, programmer, translator • @shioyama on github,

    twitter, etc • I program in Ruby • I live in Tokyo • I am the author of Mobility • I blog at dejimata.com • I coined “Module Builder Pattern”
  3. None
  4. We’re Hiring!

  5. Amsterdam How I got here today Montreal Tokyo

  6. Translation is something that seems easy, but is actually very

    hard
  7. One Problem, Many Solutions • How many languages? • How

    many translated models? • Growing application? Adding new models? • What’s your database? Are your translations in your database? • What are your performance requirements? • How good are you at Arel?
  8. Gems and Strategies • Translation Strategies and Gems: – Translatable

    Columns (Traco) – Model Translation Tables (Globalize) – Shared Translation Table(s) – Serialized Columns (Multilang) – Hstore/Jsonb (json_translate, multilang_hstore, ...) • Each gem is coupled to a storage strategy • Strategies are (conceptually) independent of ORM
  9. Plan for Today’s Talk 1. Translation Interface – How do

    you read and write translations? – What other features are supported? 2. Storage Strategies – How are translations actually stored? – Good and bad of each strategy 3. Mobility – Translated Attributes, Backends and Plugins 4. Looking Ahead – Mobility as Translation Layer
  10. Translation Interface Translation Interface

  11. Common Features • Translations are fetched and stored using (virtual)

    attribute accessors • Language fetched or stored is the value of the global locale (I18n.locale or similar) • Models can be queried using translated attributes (like normal column-based attributes)
  12. One Attribute, Many Languages class Post < ApplicationRecord translates :title

    end post = Post.new I18n.locale = :en post.title = "Translating with Mobility" I18n.locale = :fr post.title = "Traduction avec Mobilité" post.save post = Post.first I18n.locale = :en post.title #=> "Translating with Mobility" I18n.locale = :fr post.title #=> "Traduction avec Mobilité" class Post < ApplicationRecord translates :title end post = Post.new I18n.locale = :en post.title = "Translating with Mobility" I18n.locale = :fr post.title = "Traduction avec Mobilité" post.save post = Post.first I18n.locale = :en post.title #=> "Translating with Mobility" I18n.locale = :fr post.title #=> "Traduction avec Mobilité"
  13. :en => :ja Fallbacks I18n.locale = :en post.title #=> nil

    I18n.locale = :en post.title #=> nil I18n.locale = :en post.title #=> " タイトル " I18n.locale = :en post.title #=> " タイトル " No Fallbacks post = Post.new I18n.locale = :ja post.title = " タイトル " post.save post = Post.new I18n.locale = :ja post.title = " タイトル " post.save
  14. Translations Cache/Stash post = Post.first I18n.locale = :ja post.title #=>

    " タイトル " I18n.locale = :en post.title #=> "Title" I18n.locale = :fr post.title #=> "Titre" post = Post.first I18n.locale = :ja post.title #=> " タイトル " I18n.locale = :en post.title #=> "Title" I18n.locale = :fr post.title #=> "Titre" Locale Value ja タイトル en Title fr Titre “Title” Cache
  15. Locale (Easy) Accessors class Post < ApplicationRecord translates :title, locale_accessors:

    [:en, :ja] end post = Post.new post.title_ja = " タイトル 2" post.title_en = "Title 2" post.title_ja #=> " タイトル 2" post.title_en #=> "Title 2" class Post < ApplicationRecord translates :title, locale_accessors: [:en, :ja] end post = Post.new post.title_ja = " タイトル 2" post.title_en = "Title 2" post.title_ja #=> " タイトル 2" post.title_en #=> "Title 2"
  16. Dirty Attribute Tracking • Tracking of translated attribute changes –

    title_was – title_changes – title_changed? – ...
  17. Bonus Points: Query by Translation I18n.locale = :en Post.find_by(title: "Translating

    with Mobility") #=> (post with English title "Translating...") I18n.locale = :fr Post.find_by(title: "Traduction avec Mobilité") #=> (returns post with French title "Traduction...") I18n.locale = :en Post.find_by(title: "Translating with Mobility") #=> (post with English title "Translating...") I18n.locale = :fr Post.find_by(title: "Traduction avec Mobilité") #=> (returns post with French title "Traduction...") • Override query methods to return records by translated attribute • Hides details of storage strategy from query interface
  18. Storage Strategies Storage Strategies

  19. What is a Storage Strategy? translated attribute translated attribute Persistence

    Layer describes a mapping between translated attributes and persistence layer
  20. Translatable Columns

  21. Translatable Columns • Good – Easy to understand – Easy

    to construct queries – Good performance (no joins) • Bad – Every new language requires a migration for every translated model – Columns created for every locale even if there is no translation for a locale
  22. Model Translation Tables

  23. Model Translation Tables • Translation table for each model, with:

    – locale column – foreign key pointing to model table (post_id) – one or more translated columns • has_many association between model and translations • Translation table records are created/updated when attribute is set and model is saved has_many :translations, class_name: PostTranslation has_many :translations, class_name: PostTranslation
  24. Model Translation Tables • Good – Easily scales with number

    of languages (no new migrations to add locale) – Space usage is more efficient (only create record if translation is stored) • Bad – Querying may be less performant (need to join translation table) – Code is more complex due to joining – Migration to create translation table/column for every model/attribute
  25. Shared Translation Tables

  26. Shared Translation Tables • One or more translation tables, with:

    – locale column – foreign key pointing to model id (translatable_id) – column holding the model class (translatable_type) – column holding the name of the attribute (key) – column holding the value of the attribute (value) • One translation table for each value type (string, text, ...) has_many :translations, as: :translatable, class_name: ::Mobility::ActiveRecord::StringTranslation has_many :translations, as: :translatable, class_name: ::Mobility::ActiveRecord::StringTranslation
  27. Shared Translation Tables • Good – No additional migrations at

    all, so easy to scale to new locales + new models – Space usage is relatively efficient, scales with number of locales + models • Bad – Very complex querying (know your Arel?) – All translations in the same table, can be difficult to maintain a consistent database state
  28. Serialized Translations • Store translations as serialized hash on a

    single database column class Post < ApplicationRecord serialize :title_translations, Hash end post = Post.new post.title_translations = { "en" => "Translating with Mobility" } post.save class Post < ApplicationRecord serialize :title_translations, Hash end post = Post.new post.title_translations = { "en" => "Translating with Mobility" } post.save
  29. Serialized Translations • Good – Scales well with locales (no

    additional migrations) – Easy to add translations to model (simply add text column for each translated attribute) – Simple to access • Bad – Querying is not possible – Loading one translation requires loading all translations at once (serialized column)
  30. Hstore/Jsonb (PostgreSQL) • Similar to serialized strategy, except that querying

    is possible and can be fast • Database-dependent strategy, highly coupled to persistence layer • (Jsonb generally preferred over Hstore) Post.where(title: "foo").to_sql #=> SELECT "posts".* FROM "posts" WHERE ("posts"."title" @> '{"en":"foo"}') Post.where(title: "foo").to_sql #=> SELECT "posts".* FROM "posts" WHERE ("posts"."title" @> '{"en":"foo"}')
  31. Other Possibilities • Stored in external service, accessed via API

    (Contentful, Firebase, etc) • File storage • NoSQL database • Cache storage (Redis/Memcache/etc) • New forms of storage we cannot yet imagine
  32. Mobility Mobility

  33. Translating with Mobility • Existing solutions – couple implementation to

    storage strategy – couple implementation to ORM (ActiveRecord) and framework (Rails) – have similar but not identical interfaces • What I wanted: – Support customizable, swappable storage solutions – No hard dependency on ActiveRecord or Rails – Interface that is consistent and unified, but flexible
  34. A Translation Framework • New approach to translation problem: –

    One translation interface, swappable storage backends – Supports all existing strategies, plus more – Supports both ActiveRecord and Sequel ORM • Like the OmniAuth of Translation
  35. Getting Started rails g mobility:install rails g mobility:install class Post

    < ApplicationRecord include Mobility translates :title, :body end class Post < ApplicationRecord include Mobility translates :title, :body end Mobility.configure do |config| config.default_backend = :key_value config.accessor_method = :translates config.query_method = :i18n end Mobility.configure do |config| config.default_backend = :key_value config.accessor_method = :translates config.query_method = :i18n end :key_value :table :column :jsonb :hstore :serialized rake db:migrate rake db:migrate
  36. Separation of Responsibilities Storage Layer • Atomic methods for reading

    and writing • Model setup • Querying extension Storage Layer • Atomic methods for reading and writing • Model setup • Querying extension Interface Layer • Defines translation accessors • Fallbacks, cache, dirty tracking • Locale accessors Interface Layer • Defines translation accessors • Fallbacks, cache, dirty tracking • Locale accessors
  37. Pluggable Backends • Backend encapsulates a storage strategy for a

    single attribute (or set of attributes) • Automatically support locale accessors, fallbacks, dirty tracking, cache, etc (called “plugins”) • Backends are tested against shared test suite to ensure consistency translates :title, backend: :column, fallbacks: true, dirty: true translates :body, backend: :key_value, locale_accessors: [:fr], ... translates :title, backend: :column, fallbacks: true, dirty: true translates :body, backend: :key_value, locale_accessors: [:fr], ...
  38. A Backend is a Ruby Class class MyBackend include Mobility::Backend

    # attr_reader :attribute, :model def read(locale, options = {}) # Read attribute in locale. end def write(locale, value, options = {}) # Write value to attribute in locale. end def self.configure(options) # Configure/normalize options (e.g. set any defaults) end setup do |attributes, options| # Do something with attributes and options in context of model class. end end class MyBackend include Mobility::Backend # attr_reader :attribute, :model def read(locale, options = {}) # Read attribute in locale. end def write(locale, value, options = {}) # Write value to attribute in locale. end def self.configure(options) # Configure/normalize options (e.g. set any defaults) end setup do |attributes, options| # Do something with attributes and options in context of model class. end end
  39. A Mobility Model class Post include Mobility translates :title, backend:

    :column, locale_accessors: [:en, :ja] translates :body, backend: :key_value, locale_accessors: [:fr] end post = Post.new post.title = “Title” post.body = “Lorum Ipsum...” post.title #=> “Title” post.title_backend #=> #<#<Class:... @fallbacks=nil @cache={...} > post.title_en #=> “Title” post.title_ja #=> nil post.title_fr #=> NoMethodError post.body #=> “Lorum Ipsum...” post.body_backend #=> #<#<Class:... @fallbacks=nil @cache={...} > post.body_en #=> NoMethodError post.body_ja #=> NoMethodError post.body_fr #=> nil class Post include Mobility translates :title, backend: :column, locale_accessors: [:en, :ja] translates :body, backend: :key_value, locale_accessors: [:fr] end post = Post.new post.title = “Title” post.body = “Lorum Ipsum...” post.title #=> “Title” post.title_backend #=> #<#<Class:... @fallbacks=nil @cache={...} > post.title_en #=> “Title” post.title_ja #=> nil post.title_fr #=> NoMethodError post.body #=> “Lorum Ipsum...” post.body_backend #=> #<#<Class:... @fallbacks=nil @cache={...} > post.body_en #=> NoMethodError post.body_ja #=> NoMethodError post.body_fr #=> nil
  40. Query Methods Too class Post include Mobility translates :title, backend:

    :column, locale_accessors: [:en, :ja] translates :body, backend: :key_value, locale_accessors: [:fr] end post = Post.new post.title = “Title” post.body = “Lorum Ipsum...” post.save Post.i18n.where(title: “Title”, body: “Lorum Ipsum...”) # SELECT "posts".* FROM "posts" # INNER JOIN "mobility_text_translations" "body_mobility_text_translations" # ON "body_mobility_text_translations"."key" = 'body' # AND "body_mobility_text_translations"."locale" = 'en' # AND "body_mobility_text_translations"."translatable_type" = 'Post' # AND "body_mobility_text_translations"."translatable_id" = "posts"."id" # WHERE "body_mobility_text_translations"."value" = 'Lorum Ipsum...' # AND "posts"."title_en" = 'Title' class Post include Mobility translates :title, backend: :column, locale_accessors: [:en, :ja] translates :body, backend: :key_value, locale_accessors: [:fr] end post = Post.new post.title = “Title” post.body = “Lorum Ipsum...” post.save Post.i18n.where(title: “Title”, body: “Lorum Ipsum...”) # SELECT "posts".* FROM "posts" # INNER JOIN "mobility_text_translations" "body_mobility_text_translations" # ON "body_mobility_text_translations"."key" = 'body' # AND "body_mobility_text_translations"."locale" = 'en' # AND "body_mobility_text_translations"."translatable_type" = 'Post' # AND "body_mobility_text_translations"."translatable_id" = "posts"."id" # WHERE "body_mobility_text_translations"."value" = 'Lorum Ipsum...' # AND "posts"."title_en" = 'Title' title body
  41. Attributes as a Module class Post translates :title, backend: :key_value

    end class Post translates :title, backend: :key_value end class Post include Mobility::Attributes.new(:accessor, [:title], backend: :key_value) end class Post include Mobility::Attributes.new(:accessor, [:title], backend: :key_value) end It’s a Module Builder!
  42. Elements of a Module Builder class Attributes < Module def

    initialize(method, *attributes) @attributes = attributes # ... end def included(klass) @attributes.each do |attribute| define_method attribute do # ... end # ... end end end class Post include Attributes.new(:accessor, [:title]) end class Attributes < Module def initialize(method, *attributes) @attributes = attributes # ... end def included(klass) @attributes.each do |attribute| define_method attribute do # ... end # ... end end end class Post include Attributes.new(:accessor, [:title]) end Subclass Module Define module methods in an included hook Include instance of Module subclass in other classes
  43. None
  44. “Plugging in”: Mobility Plugins class Post translates :title, fallbacks: true,

    dirty: true translates :subtitle, backend: :column translates :body, locale_accessors: [:en, :de], cache: false end class Post translates :title, fallbacks: true, dirty: true translates :subtitle, backend: :column translates :body, locale_accessors: [:en, :de], cache: false end backend_class.include Mobility::Plugins::Fallbacks.new(true) More Module Builders! @backend_class = Class.new(Mobility::Backends::ActiveRecord::Column) backend_class.include Mobility::Plugins::ActiveModel::Dirty include Mobility::Plugins::LocaleAccessors.new(:body, [:en, :de])
  45. Fallbacks as a Plugin module Mobility module Fallbacks # ...

    def read(locale, options = {}) fallback = options.delete(:fallback) return super if fallback == false || (fallback.nil? && fallbacks.nil?) (fallback ? [locale, *fallback] : fallbacks[locale]).detect do |fallback_locale| value = super(fallback_locale, options) break value if value.present? end end end end module Mobility module Fallbacks # ... def read(locale, options = {}) fallback = options.delete(:fallback) return super if fallback == false || (fallback.nil? && fallbacks.nil?) (fallback ? [locale, *fallback] : fallbacks[locale]).detect do |fallback_locale| value = super(fallback_locale, options) break value if value.present? end end end end • Plugins applied to backend (fallbacks, cache, dirty) override read and/or write methods to add functionality Find first non-nil value of attribute
  46. Backend Model Setup class Post translates :title, fallbacks: true, dirty:

    true translates :subtitle, backend: :column translates :body, locale_accessors: [:en, :de], cache: false end class Post translates :title, fallbacks: true, dirty: true translates :subtitle, backend: :column translates :body, locale_accessors: [:en, :de], cache: false end class Post has_many :mobility_text_translations, where: { key: [:title, :body] }, as: :translatable, class_name: ::Mobility::ActiveRecord::TextTranslation end class Post has_many :mobility_text_translations, where: { key: [:title, :body] }, as: :translatable, class_name: ::Mobility::ActiveRecord::TextTranslation end backend_class.setup_model(Post, :title, fallbacks: true, dirty: true)
  47. Reading a Translated Attribute I18n.locale = :fr post.title I18n.locale =

    :fr post.title post.title_backend.read(:fr) post.title_backend.read(:fr) post.read_attribute(:title_fr) post.read_attribute(:title_fr) post.translations.find { |t| t.locale == :fr }.title post.translations.find { |t| t.locale == :fr }.title Translatable Columns Backend Model Translations Backend
  48. Recap (Sales Pitch) • Mobility: – supports every storage strategy,

    each in its own class – supports multiple strategies in a single model – supports all common features, at the attribute level • Mobility can be: – as simple as you want it to be – as complex as you need it to be
  49. Looking Ahead Looking Ahead

  50. Mobility as a Translation Layer • For application developers: –

    simplifies and standardizes content translation – does not lock you into a single storage strategy • For developers of other gems: – A single Mobility integration and will support all current and future storage strategies – Example: friendly_id-mobility
  51. One Translation Layer Translatable Columns Model Translation Tables Shared Translation

    Tables Serialized Column Hstore/Jsonb Storage Strategies Gem Integrations ... Mobility Mobility FriendlyId Spree / Solidus Ransack Locomotive CMS ...
  52. Roadmap • Extensions & Plugins – Form Helpers – Machine

    Translation as fallback • Performance Testing • Integrations – Spree/Solidus, ... • Extend ORM support (Hanami, etc)
  53. Thanks!