Save 37% off PRO during our Black Friday Sale! »

Meiosis in Rails Apps

Meiosis in Rails Apps

Rails is optimized for getting you up and running quickly. However, while the first steps are so fast and fun (we all built a blog in 15 minutes!), the rapid speed at which they grow can raise some challenges.
At some point, every Rails developer needs to transform a “monster app” into something more manageable.

During this talk, you’ll join me as I share the story of one such app and the meiosis process of splitting it into several apps, services and gems.
You’ll learn about the pitfalls we faced, the bugs that plagued us, as well as get a robust set of pragmatic tips that will explain what to do when facing a daunting task such as this one.

Talk given at Rails Israel 2014

2f7eb8e439ce2d288dcfa240b5210664?s=128

Yonatan Bergman

November 02, 2014
Tweet

Transcript

  1. MEIOSIS IN RAILS APPS @yonbergman

  2. Yonatan Bergman @yonbergman

  3. Meiosis

  4. Meiosis

  5. Tech Lead eBay Israel Innovation Center

  6. eBay Valet

  7. None
  8. $ GET PRICE ESTIMATION SEND ITEM FOR FREE WE SELL

    IT, YOU GET PAID
  9. None
  10. SERVER

  11. SERVER

  12. HISTORY

  13. SERVER

  14. ! SERVER APP SERVERS DATABASE REDIS SIDEKIQ UTILITY SERVERS

  15. SERVER

  16. SERVER

  17. SERVER EBAY SERVICES

  18. SERVER ANALYTICS

  19. EBAY SERVICES ANALYTICS SERVER

  20. EBAY SERVICES ANALYTICS REPORTING AUDITING QA TOOLS CUSTOMER SUPPORT A/B

    TESTING SERVER PACKAGE TRACKING
  21. PACKAGE TRACKING EBAY SERVICES ANALYTICS REPORTING AUDITING QA TOOLS CUSTOMER

    SUPPORT A/B TESTING SERVER
  22. None
  23. SAD PANDA

  24. LAUNCH

  25. None
  26. DATABASE LOAD SERVER LOAD DEPLOYMENTS USER RESPONSIBILITIES COMPLEX MODEL TREE

    STAGING <-> PRODUCTION
  27. DATABASE LOAD SERVER LOAD DEPLOYMENTS USER RESPONSIBILITIES COMPLEX MODEL TREE

    STAGING <-> PRODUCTION
  28. DATABASE LOAD SERVER LOAD DEPLOYMENTS USER RESPONSIBILITIES COMPLEX MODEL TREE

    STAGING <-> PRODUCTION
  29. DATABASE LOAD SERVER LOAD DEPLOYMENTS USER RESPONSIBILITIES COMPLEX MODEL TREE

    STAGING <-> PRODUCTION
  30. 6 LEVELS OF SEPARATION

  31. 6 LEVELS OF SEPARATION

  32. 6 5 4 3 2 MONSTER APP 1

  33. 6 5 4 3 2 1 MONSTER APP /LIB

  34. 6 5 4 3 MONSTER APP /LIB NAMESPACING 2 1

  35. 6 5 4 MONSTER APP /LIB NAMESPACING ENGINES 3 2

    1
  36. 6 5 MONSTER APP /LIB NAMESPACING ENGINES GEMS 4 3

    2 1
  37. 6 MONSTER APP /LIB NAMESPACING ENGINES GEMS SERVICES & SERVERS

    5 4 3 2 1
  38. 6 MONSTER APP /LIB NAMESPACING ENGINES GEMS SOA 5 4

    3 2 1
  39. SEPARATION

  40. SERVER

  41. EBAY GEM SERVER PRICE ESTIMATION SERVER

  42. EBAY GEM SERVER PRICE ESTIMATION SERVER GEM

  43. EBAY GEM SERVER PRICE ESTIMATION SERVER GEM

  44. EBAY GEM SERVER PRICE ESTIMATION SERVER EBAY SERVICES GEM

  45. require 'ebayr'
 require 'rebay'
 require_relative ‘services/super_secret_api' ! ! module EbayServices

    class Shopping def initialize @shopping = Rebay::Shopping.new end end end ! !
  46. EBAY GEM SERVER PRICE ESTIMATION SERVER EBAY SERVICES GEM

  47. API DESIGN

  48. AUTH VERSIONING SERIALIZING PARSING ERRORS

  49. AUTH VERSIONING SERIALIZING PARSING ERRORS

  50. $ curl -H “TOKEN: FOO” https://example.com { ‘status’: ‘ok’ }

  51. $ curl -H “TOKEN: FOO” https://example.com { ‘status’: ‘ok’ }

    ! $ curl https://example.com?token=FOO { ‘status’: ‘ok’ }
  52. AUTH VERSIONING SERIALIZING PARSING ERRORS

  53. $ curl https://example.com/api/v1/status { ‘status’: ‘ok’ }

  54. $ curl https://example.com/api/v1/status { ‘status’: ‘ok’ } ! $ curl

    -H ‘FOO-VERSION: 1’ 
 https://example.com/api/status { ‘status’: ‘ok’ }
  55. $ curl https://example.com/api/v1/status { ‘status’: ‘ok’ } ! $ curl

    -H ‘FOO-VERSION: 1’ 
 https://example.com/api/status { ‘status’: ‘ok’ } ! $ curl
 -H ‘Accept: application/vnd.foo.v1+json’
 https://example.com/api/status { ‘status’: ‘ok’ }
  56. gem ‘versionist’

  57. AUTH VERSIONING SERIALIZING PARSING ERRORS

  58. gem ‘active_model_serializers’

  59. class ItemSerializer < ActiveModel::Serializer attributes :uid, :title, :description, :category has_one

    :response 
 end
  60. class API::ItemSerializer < ActiveModel::Serializer attributes :uid, :title, :description, :category has_one

    :response, 
 serializer: API::ResponseSerializer 
 def include_category?
 scope.api_version > 1
 end 
 end
  61. class API::ItemSerializer < ActiveModel::Serializer attributes :uid, :title, :description, :category has_one

    :response, 
 serializer: API::ResponseSerializer 
 def include_category?
 scope.api_version > 1
 end 
 end
  62. class API::ItemSerializer < ActiveModel::Serializer attributes :uid, :title, :description, :category has_one

    :response, 
 serializer: API::ResponseSerializer 
 def include_category?
 scope.api_version > 1
 end 
 end
  63. class API::ItemSerializer < ActiveModel::Serializer attributes :uid, :title, :description, :category has_one

    :response, 
 serializer: API::ResponseSerializer 
 def include_category?
 scope.api_version > 1
 end 
 end
  64. AUTH VERSIONING SERIALIZING PARSING ERRORS

  65. SERIALIZING OBJ JSON PARSING OBJ JSON

  66. gem ‘hashie’

  67. class API::ItemCreator def create(params)
 Item.create(parsed_params(params))
 end def parsed_params(params)
 ItemParser.new(params).to_hash
 end

    class ItemParser < Hashie::Trash
 property :title, required: true
 property :description
 end end
  68. class API::ItemCreator def create(params)
 Item.create(parsed_params(params))
 end def parsed_params(params)
 ItemParser.new(params).to_hash
 end

    class ItemParser < Hashie::Trash
 property :title, required: true
 property :description
 end end
  69. class API::ItemCreator def create(params)
 Item.create(parsed_params(params))
 end def parsed_params(params)
 ItemParser.new(params).to_hash
 end

    class ItemParser < Hashie::Trash
 property :title, required: true
 property :description
 end end
  70. class ItemParser < Hashie::Trash property :title, required: true
 property :description


    property :category_id, from: ‘category’
 property :price_cents, with: -> (price) {
 (price * 100).to_i
 } end
  71. class ItemParser < Hashie::Trash property :title, required: true
 property :description


    property :category_id, from: ‘category’
 property :price_cents, with: -> (price) {
 (price * 100).to_i
 } end
  72. class ItemParser < Hashie::Trash property :title, required: true
 property :description


    property :category_id, from: ‘category’
 property :price_cents, with: -> (price) {
 (price * 100).to_i
 } end
  73. class ItemParser < Hashie::Trash property :title, required: true
 property :description


    property :category_id, from: ‘category’
 property :price_cents, with: -> (price) {
 (price * 100).to_i
 } end
  74. class ItemParser < Hashie::Trash property :user_attributes, 
 from: ‘seller’,
 with:

    -> (hash) { 
 ItemParser::UserParser.new(hash) 
 } class UserParser < Hashie::Trash
 property :name, required: true
 end end
  75. class ItemParser < Hashie::Trash property :user_attributes, 
 from: ‘seller’,
 with:

    -> (hash) { 
 ItemParser::UserParser.new(hash) 
 } class UserParser < Hashie::Trash
 property :name, required: true
 end end
  76. class ItemParser < Hashie::Trash property :user_attributes, 
 from: ‘seller’,
 with:

    -> (hash) { 
 ItemParser::UserParser.new(hash) 
 } class UserParser < Hashie::Trash
 property :name, required: true
 end end
  77. class ItemParser < Hashie::Trash property :user_attributes, 
 from: ‘seller’,
 with:

    -> (hash) { 
 ItemParser::UserParser.new(hash) 
 } class UserParser < Hashie::Trash
 property :name, required: true
 end end
  78. class ItemParser < Hashie::Trash property :user_attributes, 
 from: ‘seller’,
 with:

    -> (hash) { 
 ItemParser::UserParser.new(hash) 
 } class UserParser < Hashie::Trash
 property :name, required: true
 end end
  79. gem ‘representable’
 gem ‘roar’

  80. AUTH VERSIONING SERIALIZING PARSING ERRORS

  81. ERROR ERROR CODE ERROR MESSAGE HTTP CODE record not found

    1003 The object you are looking for could not be found 404 unauthorized 1004 You are not authorized to do the following action 403 invalid param 1005 You are missing a required parameter 400 invalid value 1006 Why are you reading this slide? 400
  82. module API::ErrorSupport rescue_from Exception, with: :respond_with_error def respond_with_error(e)
 error =

    translate_exception(e)
 render json: error, 
 status: error.status_code
 end end
  83. module API::ErrorSupport rescue_from Exception, with: :respond_with_error def respond_with_error(e)
 error =

    translate_exception(e)
 render json: error, 
 status: error.status_code
 end end
  84. module API::ErrorSupport rescue_from Exception, with: :respond_with_error def respond_with_error(e)
 error =

    translate_exception(e)
 render json: error, 
 status: error.status_code
 end end
  85. module API::ErrorSupport def translate_exception(e)
 case e
 when ActiveRecord::RecordNotFound
 ApiError.new(1003,
 I18n.t(‘errors.api.not_found'),


    :not_found)
 ...
 end
  86. module API::ErrorSupport def translate_exception(e)
 case e
 when ActiveRecord::RecordNotFound
 ApiError.new(1003,
 I18n.t(‘errors.api.not_found'),


    :not_found)
 ...
 end
  87. module API::ErrorSupport def translate_exception(e)
 case e
 when ActiveRecord::RecordNotFound
 ApiError.new(1003,
 I18n.t(‘errors.api.not_found'),


    :not_found)
 ...
 end
  88. module API::ErrorSupport def translate_exception(e)
 case e
 when ActiveRecord::RecordNotFound
 ApiError.new(1003,
 I18n.t(‘errors.api.not_found'),


    :not_found)
 ...
 end
  89. LONG RUNNING TASKS

  90. None
  91. EBAY GEM SERVER PRICE ESTIMATION SERVER

  92. EBAY GEM SERVER PRICE ESTIMATION SERVER

  93. EBAY GEM SERVER PRICE ESTIMATION SERVER

  94. EBAY GEM SERVER PRICE ESTIMATION SERVER

  95. POST /api/estimations { 'estimation': { ... },
 'callback': ‘https://example.com/callback' }

    ! RESPONSE { 'uid': 'Y2z2Q3fOTENx4w',
 'status': ‘added’, 'estimation': { ... }
  96. GET /api/estimations/:uid { 'uid': 'Y2z2Q3fOTENx4w',
 'status': 'pending' 'estimation': { ...

    } }
  97. POST /callback { 'notification': { 'uid': 'Y2z2Q3fOTENx4w', 'event': ‘estimation_completed', 'estimation':

    {...} } }
  98. RESPONSE 500 // Wait 1s
 RESPONSE 500 // Wait 2s


    RESPONSE 500 // Wait 3s
 RESPONSE 500 // Wait 5s
 RESPONSE 500 // Wait 8s
 RESPONSE 200
  99. DATA MIGRATION

  100. None
  101. gem ‘trucker’

  102. None
  103. SERVER PRICE ESTIMATION SERVER DB DB DB

  104. SERVER PRICE ESTIMATION SERVER DB DB DB

  105. development: adapter: 'postgresql' encoding: 'unicode' database: 'legacy_database' host: 'localhost' pool:

    30
  106. class LegacyModel < ActiveRecord::Base ! YAML_FILE = ‘config/legacy_db.yml’ self.abstract_class =

    true ! establish_connection( YAML::load_file(YAML_FILE)[Rails.env] ) ! end
  107. class LegacyItem < LegacyModel ! self.table_name = ‘items’ ! has_one

    :estimation, class_name: ‘LegacyEstimation’ foreign_key: ‘item_id’ ! end
  108. class LegacyMigration ! def run @output = MigrationOutput.new(@file) scope =

    LegacyItem.from_id(@starting_id) scope.find_each do |legacy_item| migrator = ItemMigrator.new(legacy_item) migrator.run @output.add(migrator.output) end end ! end
  109. JSON DB SERVER PRICE ESTIMATION SERVER DB DB

  110. JSON DB SERVER PRICE ESTIMATION SERVER DB DB

  111. JSON DB SERVER PRICE ESTIMATION SERVER DB DB

  112. JSON DB SERVER PRICE ESTIMATION SERVER DB DB

  113. 4h ~300k

  114. EBAY GEM SERVER PRICE ESTIMATION SERVER EBAY SERVICES GEM

  115. EBAY GEM SERVER PRICE ESTIMATION SERVER EBAY SERVICES GEM

  116. None
  117. jobs-il@ebay.com

  118. jobs-il@ebay.com

  119. @yonbergman