$30 off During Our Annual Pro Sale. View details »

Translating with Mobility

Translating with Mobility

Chris Salzberg

August 08, 2017
Tweet

More Decks by Chris Salzberg

Other Decks in Technology

Transcript

  1. Translating with Mobility Translating with Mobility Chris Salzberg (@shioyama)

  2. About Me • Writer, programmer, translator • @shioyama on github,

    twitter, etc • I program in Ruby • I live in Tokyo • I am the author of Mobility • I blog at dejimata.com • I coined “Module Builder Pattern”
  3. None
  4. We’re Hiring!

  5. Amsterdam How I got here today Montreal Tokyo

  6. Translation is something that seems easy, but is actually very

    hard
  7. One Problem, Many Solutions • How many languages? • How

    many translated models? • Growing application? Adding new models? • What’s your database? Are your translations in your database? • What are your performance requirements? • How good are you at Arel?
  8. Gems and Strategies • Translation Strategies and Gems: – Translatable

    Columns (Traco) – Model Translation Tables (Globalize) – Shared Translation Table(s) – Serialized Columns (Multilang) – Hstore/Jsonb (json_translate, multilang_hstore, ...) • Each gem is coupled to a storage strategy • Strategies are (conceptually) independent of ORM
  9. Plan for Today’s Talk 1. Translation Interface – How do

    you read and write translations? – What other features are supported? 2. Storage Strategies – How are translations actually stored? – Good and bad of each strategy 3. Mobility – Translated Attributes, Backends and Plugins 4. Looking Ahead – Mobility as Translation Layer
  10. Translation Interface Translation Interface

  11. Common Features • Translations are fetched and stored using (virtual)

    attribute accessors • Language fetched or stored is the value of the global locale (I18n.locale or similar) • Models can be queried using translated attributes (like normal column-based attributes)
  12. One Attribute, Many Languages class Post < ApplicationRecord translates :title

    end post = Post.new I18n.locale = :en post.title = "Translating with Mobility" I18n.locale = :fr post.title = "Traduction avec Mobilité" post.save post = Post.first I18n.locale = :en post.title #=> "Translating with Mobility" I18n.locale = :fr post.title #=> "Traduction avec Mobilité" class Post < ApplicationRecord translates :title end post = Post.new I18n.locale = :en post.title = "Translating with Mobility" I18n.locale = :fr post.title = "Traduction avec Mobilité" post.save post = Post.first I18n.locale = :en post.title #=> "Translating with Mobility" I18n.locale = :fr post.title #=> "Traduction avec Mobilité"
  13. :en => :ja Fallbacks I18n.locale = :en post.title #=> nil

    I18n.locale = :en post.title #=> nil I18n.locale = :en post.title #=> " タイトル " I18n.locale = :en post.title #=> " タイトル " No Fallbacks post = Post.new I18n.locale = :ja post.title = " タイトル " post.save post = Post.new I18n.locale = :ja post.title = " タイトル " post.save
  14. Translations Cache/Stash post = Post.first I18n.locale = :ja post.title #=>

    " タイトル " I18n.locale = :en post.title #=> "Title" I18n.locale = :fr post.title #=> "Titre" post = Post.first I18n.locale = :ja post.title #=> " タイトル " I18n.locale = :en post.title #=> "Title" I18n.locale = :fr post.title #=> "Titre" Locale Value ja タイトル en Title fr Titre “Title” Cache
  15. Locale (Easy) Accessors class Post < ApplicationRecord translates :title, locale_accessors:

    [:en, :ja] end post = Post.new post.title_ja = " タイトル 2" post.title_en = "Title 2" post.title_ja #=> " タイトル 2" post.title_en #=> "Title 2" class Post < ApplicationRecord translates :title, locale_accessors: [:en, :ja] end post = Post.new post.title_ja = " タイトル 2" post.title_en = "Title 2" post.title_ja #=> " タイトル 2" post.title_en #=> "Title 2"
  16. Dirty Attribute Tracking • Tracking of translated attribute changes –

    title_was – title_changes – title_changed? – ...
  17. Bonus Points: Query by Translation I18n.locale = :en Post.find_by(title: "Translating

    with Mobility") #=> (post with English title "Translating...") I18n.locale = :fr Post.find_by(title: "Traduction avec Mobilité") #=> (returns post with French title "Traduction...") I18n.locale = :en Post.find_by(title: "Translating with Mobility") #=> (post with English title "Translating...") I18n.locale = :fr Post.find_by(title: "Traduction avec Mobilité") #=> (returns post with French title "Traduction...") • Override query methods to return records by translated attribute • Hides details of storage strategy from query interface
  18. Storage Strategies Storage Strategies

  19. What is a Storage Strategy? translated attribute translated attribute Persistence

    Layer describes a mapping between translated attributes and persistence layer
  20. Translatable Columns

  21. Translatable Columns • Good – Easy to understand – Easy

    to construct queries – Good performance (no joins) • Bad – Every new language requires a migration for every translated model – Columns created for every locale even if there is no translation for a locale
  22. Model Translation Tables

  23. Model Translation Tables • Translation table for each model, with:

    – locale column – foreign key pointing to model table (post_id) – one or more translated columns • has_many association between model and translations • Translation table records are created/updated when attribute is set and model is saved has_many :translations, class_name: PostTranslation has_many :translations, class_name: PostTranslation
  24. Model Translation Tables • Good – Easily scales with number

    of languages (no new migrations to add locale) – Space usage is more efficient (only create record if translation is stored) • Bad – Querying may be less performant (need to join translation table) – Code is more complex due to joining – Migration to create translation table/column for every model/attribute
  25. Shared Translation Tables

  26. Shared Translation Tables • One or more translation tables, with:

    – locale column – foreign key pointing to model id (translatable_id) – column holding the model class (translatable_type) – column holding the name of the attribute (key) – column holding the value of the attribute (value) • One translation table for each value type (string, text, ...) has_many :translations, as: :translatable, class_name: ::Mobility::ActiveRecord::StringTranslation has_many :translations, as: :translatable, class_name: ::Mobility::ActiveRecord::StringTranslation
  27. Shared Translation Tables • Good – No additional migrations at

    all, so easy to scale to new locales + new models – Space usage is relatively efficient, scales with number of locales + models • Bad – Very complex querying (know your Arel?) – All translations in the same table, can be difficult to maintain a consistent database state
  28. Serialized Translations • Store translations as serialized hash on a

    single database column class Post < ApplicationRecord serialize :title_translations, Hash end post = Post.new post.title_translations = { "en" => "Translating with Mobility" } post.save class Post < ApplicationRecord serialize :title_translations, Hash end post = Post.new post.title_translations = { "en" => "Translating with Mobility" } post.save
  29. Serialized Translations • Good – Scales well with locales (no

    additional migrations) – Easy to add translations to model (simply add text column for each translated attribute) – Simple to access • Bad – Querying is not possible – Loading one translation requires loading all translations at once (serialized column)
  30. Hstore/Jsonb (PostgreSQL) • Similar to serialized strategy, except that querying

    is possible and can be fast • Database-dependent strategy, highly coupled to persistence layer • (Jsonb generally preferred over Hstore) Post.where(title: "foo").to_sql #=> SELECT "posts".* FROM "posts" WHERE ("posts"."title" @> '{"en":"foo"}') Post.where(title: "foo").to_sql #=> SELECT "posts".* FROM "posts" WHERE ("posts"."title" @> '{"en":"foo"}')
  31. Other Possibilities • Stored in external service, accessed via API

    (Contentful, Firebase, etc) • File storage • NoSQL database • Cache storage (Redis/Memcache/etc) • New forms of storage we cannot yet imagine
  32. Mobility Mobility

  33. Translating with Mobility • Existing solutions – couple implementation to

    storage strategy – couple implementation to ORM (ActiveRecord) and framework (Rails) – have similar but not identical interfaces • What I wanted: – Support customizable, swappable storage solutions – No hard dependency on ActiveRecord or Rails – Interface that is consistent and unified, but flexible
  34. A Translation Framework • New approach to translation problem: –

    One translation interface, swappable storage backends – Supports all existing strategies, plus more – Supports both ActiveRecord and Sequel ORM • Like the OmniAuth of Translation
  35. Getting Started rails g mobility:install rails g mobility:install class Post

    < ApplicationRecord include Mobility translates :title, :body end class Post < ApplicationRecord include Mobility translates :title, :body end Mobility.configure do |config| config.default_backend = :key_value config.accessor_method = :translates config.query_method = :i18n end Mobility.configure do |config| config.default_backend = :key_value config.accessor_method = :translates config.query_method = :i18n end :key_value :table :column :jsonb :hstore :serialized rake db:migrate rake db:migrate
  36. Separation of Responsibilities Storage Layer • Atomic methods for reading

    and writing • Model setup • Querying extension Storage Layer • Atomic methods for reading and writing • Model setup • Querying extension Interface Layer • Defines translation accessors • Fallbacks, cache, dirty tracking • Locale accessors Interface Layer • Defines translation accessors • Fallbacks, cache, dirty tracking • Locale accessors
  37. Pluggable Backends • Backend encapsulates a storage strategy for a

    single attribute (or set of attributes) • Automatically support locale accessors, fallbacks, dirty tracking, cache, etc (called “plugins”) • Backends are tested against shared test suite to ensure consistency translates :title, backend: :column, fallbacks: true, dirty: true translates :body, backend: :key_value, locale_accessors: [:fr], ... translates :title, backend: :column, fallbacks: true, dirty: true translates :body, backend: :key_value, locale_accessors: [:fr], ...
  38. A Backend is a Ruby Class class MyBackend include Mobility::Backend

    # attr_reader :attribute, :model def read(locale, options = {}) # Read attribute in locale. end def write(locale, value, options = {}) # Write value to attribute in locale. end def self.configure(options) # Configure/normalize options (e.g. set any defaults) end setup do |attributes, options| # Do something with attributes and options in context of model class. end end class MyBackend include Mobility::Backend # attr_reader :attribute, :model def read(locale, options = {}) # Read attribute in locale. end def write(locale, value, options = {}) # Write value to attribute in locale. end def self.configure(options) # Configure/normalize options (e.g. set any defaults) end setup do |attributes, options| # Do something with attributes and options in context of model class. end end
  39. A Mobility Model class Post include Mobility translates :title, backend:

    :column, locale_accessors: [:en, :ja] translates :body, backend: :key_value, locale_accessors: [:fr] end post = Post.new post.title = “Title” post.body = “Lorum Ipsum...” post.title #=> “Title” post.title_backend #=> #<#<Class:... @fallbacks=nil @cache={...} > post.title_en #=> “Title” post.title_ja #=> nil post.title_fr #=> NoMethodError post.body #=> “Lorum Ipsum...” post.body_backend #=> #<#<Class:... @fallbacks=nil @cache={...} > post.body_en #=> NoMethodError post.body_ja #=> NoMethodError post.body_fr #=> nil class Post include Mobility translates :title, backend: :column, locale_accessors: [:en, :ja] translates :body, backend: :key_value, locale_accessors: [:fr] end post = Post.new post.title = “Title” post.body = “Lorum Ipsum...” post.title #=> “Title” post.title_backend #=> #<#<Class:... @fallbacks=nil @cache={...} > post.title_en #=> “Title” post.title_ja #=> nil post.title_fr #=> NoMethodError post.body #=> “Lorum Ipsum...” post.body_backend #=> #<#<Class:... @fallbacks=nil @cache={...} > post.body_en #=> NoMethodError post.body_ja #=> NoMethodError post.body_fr #=> nil
  40. Query Methods Too class Post include Mobility translates :title, backend:

    :column, locale_accessors: [:en, :ja] translates :body, backend: :key_value, locale_accessors: [:fr] end post = Post.new post.title = “Title” post.body = “Lorum Ipsum...” post.save Post.i18n.where(title: “Title”, body: “Lorum Ipsum...”) # SELECT "posts".* FROM "posts" # INNER JOIN "mobility_text_translations" "body_mobility_text_translations" # ON "body_mobility_text_translations"."key" = 'body' # AND "body_mobility_text_translations"."locale" = 'en' # AND "body_mobility_text_translations"."translatable_type" = 'Post' # AND "body_mobility_text_translations"."translatable_id" = "posts"."id" # WHERE "body_mobility_text_translations"."value" = 'Lorum Ipsum...' # AND "posts"."title_en" = 'Title' class Post include Mobility translates :title, backend: :column, locale_accessors: [:en, :ja] translates :body, backend: :key_value, locale_accessors: [:fr] end post = Post.new post.title = “Title” post.body = “Lorum Ipsum...” post.save Post.i18n.where(title: “Title”, body: “Lorum Ipsum...”) # SELECT "posts".* FROM "posts" # INNER JOIN "mobility_text_translations" "body_mobility_text_translations" # ON "body_mobility_text_translations"."key" = 'body' # AND "body_mobility_text_translations"."locale" = 'en' # AND "body_mobility_text_translations"."translatable_type" = 'Post' # AND "body_mobility_text_translations"."translatable_id" = "posts"."id" # WHERE "body_mobility_text_translations"."value" = 'Lorum Ipsum...' # AND "posts"."title_en" = 'Title' title body
  41. Attributes as a Module class Post translates :title, backend: :key_value

    end class Post translates :title, backend: :key_value end class Post include Mobility::Attributes.new(:accessor, [:title], backend: :key_value) end class Post include Mobility::Attributes.new(:accessor, [:title], backend: :key_value) end It’s a Module Builder!
  42. Elements of a Module Builder class Attributes < Module def

    initialize(method, *attributes) @attributes = attributes # ... end def included(klass) @attributes.each do |attribute| define_method attribute do # ... end # ... end end end class Post include Attributes.new(:accessor, [:title]) end class Attributes < Module def initialize(method, *attributes) @attributes = attributes # ... end def included(klass) @attributes.each do |attribute| define_method attribute do # ... end # ... end end end class Post include Attributes.new(:accessor, [:title]) end Subclass Module Define module methods in an included hook Include instance of Module subclass in other classes
  43. None
  44. “Plugging in”: Mobility Plugins class Post translates :title, fallbacks: true,

    dirty: true translates :subtitle, backend: :column translates :body, locale_accessors: [:en, :de], cache: false end class Post translates :title, fallbacks: true, dirty: true translates :subtitle, backend: :column translates :body, locale_accessors: [:en, :de], cache: false end backend_class.include Mobility::Plugins::Fallbacks.new(true) More Module Builders! @backend_class = Class.new(Mobility::Backends::ActiveRecord::Column) backend_class.include Mobility::Plugins::ActiveModel::Dirty include Mobility::Plugins::LocaleAccessors.new(:body, [:en, :de])
  45. Fallbacks as a Plugin module Mobility module Fallbacks # ...

    def read(locale, options = {}) fallback = options.delete(:fallback) return super if fallback == false || (fallback.nil? && fallbacks.nil?) (fallback ? [locale, *fallback] : fallbacks[locale]).detect do |fallback_locale| value = super(fallback_locale, options) break value if value.present? end end end end module Mobility module Fallbacks # ... def read(locale, options = {}) fallback = options.delete(:fallback) return super if fallback == false || (fallback.nil? && fallbacks.nil?) (fallback ? [locale, *fallback] : fallbacks[locale]).detect do |fallback_locale| value = super(fallback_locale, options) break value if value.present? end end end end • Plugins applied to backend (fallbacks, cache, dirty) override read and/or write methods to add functionality Find first non-nil value of attribute
  46. Backend Model Setup class Post translates :title, fallbacks: true, dirty:

    true translates :subtitle, backend: :column translates :body, locale_accessors: [:en, :de], cache: false end class Post translates :title, fallbacks: true, dirty: true translates :subtitle, backend: :column translates :body, locale_accessors: [:en, :de], cache: false end class Post has_many :mobility_text_translations, where: { key: [:title, :body] }, as: :translatable, class_name: ::Mobility::ActiveRecord::TextTranslation end class Post has_many :mobility_text_translations, where: { key: [:title, :body] }, as: :translatable, class_name: ::Mobility::ActiveRecord::TextTranslation end backend_class.setup_model(Post, :title, fallbacks: true, dirty: true)
  47. Reading a Translated Attribute I18n.locale = :fr post.title I18n.locale =

    :fr post.title post.title_backend.read(:fr) post.title_backend.read(:fr) post.read_attribute(:title_fr) post.read_attribute(:title_fr) post.translations.find { |t| t.locale == :fr }.title post.translations.find { |t| t.locale == :fr }.title Translatable Columns Backend Model Translations Backend
  48. Recap (Sales Pitch) • Mobility: – supports every storage strategy,

    each in its own class – supports multiple strategies in a single model – supports all common features, at the attribute level • Mobility can be: – as simple as you want it to be – as complex as you need it to be
  49. Looking Ahead Looking Ahead

  50. Mobility as a Translation Layer • For application developers: –

    simplifies and standardizes content translation – does not lock you into a single storage strategy • For developers of other gems: – A single Mobility integration and will support all current and future storage strategies – Example: friendly_id-mobility
  51. One Translation Layer Translatable Columns Model Translation Tables Shared Translation

    Tables Serialized Column Hstore/Jsonb Storage Strategies Gem Integrations ... Mobility Mobility FriendlyId Spree / Solidus Ransack Locomotive CMS ...
  52. Roadmap • Extensions & Plugins – Form Helpers – Machine

    Translation as fallback • Performance Testing • Integrations – Spree/Solidus, ... • Extend ORM support (Hanami, etc)
  53. Thanks!