Translating with Mobility

Presentation at the Montreal Ruby Meetup on June 20, 2017.

Transcript

1. Translating with Mobility Chris Salzberg (@shioyama)
2. About Me • Writer, programmer, translator • VP of Engineering at Degica • @shioyama on github, twitter, etc • I program in Ruby • I live in Tokyo, Japan • I am the author of Mobility • I blog at dejimata.com • I coined “Module Builder Pattern”
3. How I got here today Montreal Amsterdam Tokyo
4. Translation is something that seems easy, but is actually very hard
5. One Problem, Many Solutions • How many languages? • How many translated models? • Growing application? Adding new models? • What’s your database? • What are your performance requirements? • How good are you at Arel?
6. Gems and Strategies • Translation Strategies and Gems: – Translatable Columns (Traco) – Model Translation Tables (Globalize) – Shared Translation Table(s) – Serialized Columns (Multilang) – Hstore/Jsonb (json_translate, multilang_hstore, ...) • Each gem is coupled to a storage strategy • Strategies are (conceptually) independent of ORM
7. Plan for Today’s Talk 1. Translation Interface – How do you read and write translations? – What other features are supported? 2. Storage Strategies – How are translations actually stored? – Good and bad of each strategy 3. Mobility 4. Looking Ahead
8. Translation Interface
9. Common Features • Translations are fetched and stored using (virtual) attribute accessors • Language fetched or stored is the value of the global locale (I18n.locale or similar) • Models can be queried using translated attributes (like normal column-based attributes)
10. One Attribute, Many Languages class Post < ApplicationRecord translates :title end post = Post.new I18n.locale = :en post.title = "Translating with Mobility" I18n.locale = :fr post.title = "Traduction avec Mobilité" post.save post = Post.first I18n.locale = :en post.title #=> "Translating with Mobility" I18n.locale = :fr post.title #=> "Traduction avec Mobilité" class Post < ApplicationRecord translates :title end post = Post.new I18n.locale = :en post.title = "Translating with Mobility" I18n.locale = :fr post.title = "Traduction avec Mobilité" post.save post = Post.first I18n.locale = :en post.title #=> "Translating with Mobility" I18n.locale = :fr post.title #=> "Traduction avec Mobilité"
11. Fallbacks I18n.locale = :en post.title #=> nil I18n.locale = :en post.title #=> nil I18n.locale = :en post.title #=> " タイトル " I18n.locale = :en post.title #=> " タイトル " No Fallbacks :en => :ja post = Post.new I18n.locale = :ja post.title = " タイトル " post.save post = Post.new I18n.locale = :ja post.title = " タイトル " post.save
12. Translations Cache/Stash post = Post.first I18n.locale = :ja post.title #=> " タイトル " I18n.locale = :en post.title #=> "Title" I18n.locale = :fr post.title #=> "Titre" post = Post.first I18n.locale = :ja post.title #=> " タイトル " I18n.locale = :en post.title #=> "Title" I18n.locale = :fr post.title #=> "Titre" Locale Value ja タイトル en Title fr Titre “Title” Cache
13. Locale Accessors class Post < ApplicationRecord translates :title, locale_accessors: [:en, :ja] end post = Post.new post.title_ja = " タイトル 2" post.title_en = "Title 2" post.title_ja #=> " タイトル 2" post.title_en #=> "Title 2" class Post < ApplicationRecord translates :title, locale_accessors: [:en, :ja] end post = Post.new post.title_ja = " タイトル 2" post.title_en = "Title 2" post.title_ja #=> " タイトル 2" post.title_en #=> "Title 2"
14. Dirty Attribute Tracking • Tracking of translated attribute changes – title_was – title_changes – title_changed? – ...
15. Bonus Points: Query by Translation I18n.locale = :en Post.find_by(title: "Translating with Mobility") #=> (post with English title "Translating...") I18n.locale = :fr Post.find_by(title: "Traduction avec Mobilité") #=> (returns post with French title "Traduction...") I18n.locale = :en Post.find_by(title: "Translating with Mobility") #=> (post with English title "Translating...") I18n.locale = :fr Post.find_by(title: "Traduction avec Mobilité") #=> (returns post with French title "Traduction...") • Override query methods to return records by translated attribute • Hides details of storage strategy from query interface
16. Storage Strategies
17. What is a Storage Strategy? translated attribute translated attribute Persistence Layer describes a mapping between translated attributes and persistence layer
18. Translatable Columns
19. Translatable Columns • Good – Easy to understand – Easy to construct queries – Good performance (no joins) • Bad – Every new language requires a migration for every translated model – Columns created for every locale even if there is no translation for a locale
20. Model Translation Tables
21. Model Translation Tables • Translation table for each model, with: – locale column – foreign key pointing to model table (post_id) – one or more translated columns • has_many association between model and translations • Translation table records are created/updated when attribute is set and model is saved has_many :translations, class_name: PostTranslation has_many :translations, class_name: PostTranslation
22. Model Translation Tables • Good – Easily scales with number of languages (no new migrations to add locale) – Space usage is more efficient (only create record if translation is stored) • Bad – Querying may be less performant (need to join translation table) – Code is more complex due to joining – Migration to create translation table/column for every model/attribute
23. Shared Translation Table(s)
24. Shared Translation Table(s) • One or more translation tables, with: – locale column – foreign key pointing to model id (translatable_id) – column holding the model class (translatable_type) – column holding the name of the attribute (key) – column holding the value of the attribute (value) • One translation table for each value type (string, text, ...) has_many :translations, as: :translatable, class_name: ::Mobility::ActiveRecord::StringTranslation has_many :translations, as: :translatable, class_name: ::Mobility::ActiveRecord::StringTranslation
25. Shared Translation Table(s) • Good – No additional migrations at all, so easy to scale to new locales + new models – Space usage is relatively efficient, scales with number of locales + models • Bad – Very complex querying – All translations in the same table, can be difficult to maintain a consistent database state
26. Serialized Translations • Store translations as serialized hash on a single database column class Post < ApplicationRecord serialize :title_translations, Hash end post = Post.new post.title_translations = { "en" => "Translating with Mobility" } post.save class Post < ApplicationRecord serialize :title_translations, Hash end post = Post.new post.title_translations = { "en" => "Translating with Mobility" } post.save
27. Serialized Translations • Good – Scales well with locales (no additional migrations) – Easy to add translations to model (simply add text column for each translated attribute) – Simple to access • Bad – Querying is not possible – Loading one translation requires loading all translations at once (serialized column)
28. Hstore/Jsonb (PostgreSQL) • Similar to serialized strategy, except that querying is possible and can be fast • Database-dependent strategy, highly coupled to persistence layer • (Jsonb generally preferred over Hstore) Post.where(title: "foo").to_sql #=> SELECT "posts".* FROM "posts" WHERE ("posts"."title" @> '{"en":"foo"}') Post.where(title: "foo").to_sql #=> SELECT "posts".* FROM "posts" WHERE ("posts"."title" @> '{"en":"foo"}')
29. Other Possibilities • Stored in external service, accessed via API • File storage • NoSQL database • Cache storage (Redis/Memcache/etc) • New forms of storage we cannot yet imagine
30. Mobility
31. Translation with Mobility • Existing solutions – couple implementation to storage strategy – couple implementation to ORM (ActiveRecord) and framework (Rails) – have similar but not identical interfaces • What I wanted: – Support customizable, swappable storage solutions – No hard dependency on AR or Rails – Interface that is consistent and unified
32. Mobility: A Translation Framework • New approach to translation problem: – One translation interface, swappable storage backends – Supports all existing strategies, plus shared tables (“key value”) – Supports both ActiveRecord and Sequel ORM • Like the OmniAuth of Translation
33. Getting Started bundle exec rails g mobility:install bundle exec rails g mobility:install class Post < ApplicationRecord include Mobility translates :title, :body end class Post < ApplicationRecord include Mobility translates :title, :body end Mobility.configure do |config| config.default_backend = :key_value config.accessor_method = :translates config.query_method = :i18n end Mobility.configure do |config| config.default_backend = :key_value config.accessor_method = :translates config.query_method = :i18n end :key_value :table :column :jsonb :hstore :serialized
34. Separation of Responsibilities Storage Layer • Implements atomic methods for reading, writing • Sets up model • Extends query scope to support backend Storage Layer • Implements atomic methods for reading, writing • Sets up model • Extends query scope to support backend Interface Layer • Reading and writing translations • Fallbacks, cache, dirty tracking • Locale accessors Interface Layer • Reading and writing translations • Fallbacks, cache, dirty tracking • Locale accessors
35. “Pluggable” Backends • Backend encapsulates a storage strategy for a single attribute (or set of attributes) • All backends automatically support locale accessors, fallbacks, dirty tracking, cache • Backends are tested against shared test suite to ensure consistency
36. What’s a Backend? class MyBackend include Mobility::Backend # attr_reader :attribute, :model def read(locale, **options) # Read attribute in locale. end def write(locale, value, **options) # Write value to attribute in locale. end def self.configure(options) # Normalize options (e.g. set any defaults) end setup do |attributes, options| # Do something with attributes and options in context of model class. end end class MyBackend include Mobility::Backend # attr_reader :attribute, :model def read(locale, **options) # Read attribute in locale. end def write(locale, value, **options) # Write value to attribute in locale. end def self.configure(options) # Normalize options (e.g. set any defaults) end setup do |attributes, options| # Do something with attributes and options in context of model class. end end
37. Mobility in Action class Post include Mobility translates :title, backend: :column, locale_accessors: [:en, :ja] translates :body, backend: :key_value, locale_accessors: [:fr] end post = Post.new post.title = “Title” post.body = “Lorum Ipsum...” post.title #=> “Title” post.title_backend #=> #<#<Class:... @fallbacks=nil @cache={...} > post.title_en #=> “Title” post.title_ja #=> nil post.title_fr #=> NoMethodError post.body #=> “Lorum Ipsum...” post.body_backend #=> #<#<Class:... @fallbacks=nil @cache={...} > post.body_en #=> NoMethodError post.body_ja #=> NoMethodError post.body_fr #=> nil class Post include Mobility translates :title, backend: :column, locale_accessors: [:en, :ja] translates :body, backend: :key_value, locale_accessors: [:fr] end post = Post.new post.title = “Title” post.body = “Lorum Ipsum...” post.title #=> “Title” post.title_backend #=> #<#<Class:... @fallbacks=nil @cache={...} > post.title_en #=> “Title” post.title_ja #=> nil post.title_fr #=> NoMethodError post.body #=> “Lorum Ipsum...” post.body_backend #=> #<#<Class:... @fallbacks=nil @cache={...} > post.body_en #=> NoMethodError post.body_ja #=> NoMethodError post.body_fr #=> nil
38. Query Methods Too (?!) class Post include Mobility translates :title, backend: :column, locale_accessors: [:en, :ja] translates :body, backend: :key_value, locale_accessors: [:fr] end post = Post.new post.title = “Title” post.body = “Lorum Ipsum...” post.save Post.i18n.where(title: “Title”, body: “Lorum Ipsum...”) # SELECT "posts".* FROM "posts" # INNER JOIN "mobility_text_translations" "body_mobility_text_translations" # ON "body_mobility_text_translations"."key" = 'body' # AND "body_mobility_text_translations"."locale" = 'en' # AND "body_mobility_text_translations"."translatable_type" = 'Post' # AND "body_mobility_text_translations"."translatable_id" = "posts"."id" # WHERE "body_mobility_text_translations"."value" = 'Lorum Ipsum...' # AND "posts"."title_en" = 'Title' class Post include Mobility translates :title, backend: :column, locale_accessors: [:en, :ja] translates :body, backend: :key_value, locale_accessors: [:fr] end post = Post.new post.title = “Title” post.body = “Lorum Ipsum...” post.save Post.i18n.where(title: “Title”, body: “Lorum Ipsum...”) # SELECT "posts".* FROM "posts" # INNER JOIN "mobility_text_translations" "body_mobility_text_translations" # ON "body_mobility_text_translations"."key" = 'body' # AND "body_mobility_text_translations"."locale" = 'en' # AND "body_mobility_text_translations"."translatable_type" = 'Post' # AND "body_mobility_text_translations"."translatable_id" = "posts"."id" # WHERE "body_mobility_text_translations"."value" = 'Lorum Ipsum...' # AND "posts"."title_en" = 'Title'
39. Under the Hood class Post translates :title, backend: :key_value, locale_accessors: [:en, :ja] end class Post translates :title, backend: :key_value, locale_accessors: [:en, :ja] end class Post include Mobility::Attributes.new(:accessor, [:title], backend: :key_value, model_class: Post) end class Post include Mobility::Attributes.new(:accessor, [:title], backend: :key_value, model_class: Post) end It’s a Module Builder!
40. Module Builder class Attributes < Module def initialize(method, *attributes) # ... attributes.each do |attribute| define_method attribute do # ... end # ... end end end class Post include Attributes.new(:accessor, [:title]) end class Attributes < Module def initialize(method, *attributes) # ... attributes.each do |attribute| define_method attribute do # ... end # ... end end end class Post include Attributes.new(:accessor, [:title]) end 1) Subclass Module 2) Define module methods in an initializer 3) Include instance of Module subclass in other classes
42. “Plugging in”: Building a Backend class Post translates :title, fallbacks: true, dirty: true translates :subtitle, backend: :column translates :body, locale_accessors: [:en, :de], cache: false end class Post translates :title, fallbacks: true, dirty: true translates :subtitle, backend: :column translates :body, locale_accessors: [:en, :de], cache: false end backend_class.include Mobility::Backend::Fallbacks backend_class.include Mobility::Backend::ActiveModel::Dirty @backend_class = Class.new(Mobility::Backend::ActiveRecord::Column) include Mobility::LocaleAccessors.new(:body, [:en, :de]) Another Module Builder!
43. “Plugging in”: Building a Backend module Mobility module Fallbacks # ... def read(locale, **options) fallback = options.delete(:fallback) return super if fallback == false || (fallback.nil? && fallbacks.nil?) (fallback ? [locale, *fallback] : fallbacks[locale]).detect do |fallback_locale| value = super(fallback_locale, **options) break value if value.present? end end end end module Mobility module Fallbacks # ... def read(locale, **options) fallback = options.delete(:fallback) return super if fallback == false || (fallback.nil? && fallbacks.nil?) (fallback ? [locale, *fallback] : fallbacks[locale]).detect do |fallback_locale| value = super(fallback_locale, **options) break value if value.present? end end end end • Modules included into backend (fallbacks, cache, dirty) override read and/or write methods to add functionality Find first non-nil value of attribute
44. “Plugging In”: Model Setup class Post translates :title, fallbacks: true, dirty: true translates :subtitle, backend: :column translates :body, locale_accessors: [:en, :de], cache: false end class Post translates :title, fallbacks: true, dirty: true translates :subtitle, backend: :column translates :body, locale_accessors: [:en, :de], cache: false end class Post has_many :mobility_text_translations, where: { key: [:title, :body] }, as: :translatable, class_name: ::Mobility::ActiveRecord::TextTranslation end class Post has_many :mobility_text_translations, where: { key: [:title, :body] }, as: :translatable, class_name: ::Mobility::ActiveRecord::TextTranslation end backend_class.setup_model(Post, :title, fallbacks: true, dirty: true)
45. Reading a Translated Attribute I18n.locale = :fr post.title I18n.locale = :fr post.title post.title_backend.read(:fr) post.title_backend.read(:fr) post.read_attribute(:title_fr) post.read_attribute(:title_fr) post.translations.find { |t| t.locale == :fr }.title post.translations.find { |t| t.locale == :fr }.title Translatable Columns Backend Model Translations Backend
46. Recap (The “Sales Pitch”) • Mobility: – encapsulates storage strategy into a single class – supports every storage strategy – supports multiple strategies in a single model – supports all features (fallbacks, dirty tracking, caching, locale accessors, etc) at the attribute level – can be used to design new backends using flexible interface • Mobility keeps all your doors open
47. Looking Ahead
48. The Situation Today • I18n is ubiquitous • Translation is essential, but (almost) always an afterthought • Gems lock you into a specific storage strategy: – Globalize? Locked to model translation tables – Traco? Locked to translatable columns – json_translate? Locked to PostgreSQL jsonb
49. Mobility as a Translation Layer • For application developers: – simplifies and standardizes content translation – does not lock you into a single storage strategy • For developers of other gems: – A single Mobility integration and will support all current and future storage strategies – Example: friendly_id-mobility
50. One Integration to Rule them All Translatable Columns Model Translation Tables Shared Translation Tables Serialized Column Hstore/Jsonb Storage Strategies Gem Integrations ... Mobility Mobility FriendlyId Spree / Solidus ActiveAdmin Paperclip ...
51. Roadmap • Bug fixing, Rails 5.1 full support • Performance Testing • Integrations – Spree/Solidus • Support Hanami as ORM • API for Mobility extensions – Machine translation as fallback
52. Thanks!