Meiosis in Rails Apps

Rails is optimized for getting you up and running quickly. However, while the first steps are so fast and fun (we all built a blog in 15 minutes!), the rapid speed at which they grow can raise some challenges.
At some point, every Rails developer needs to transform a “monster app” into something more manageable.

During this talk, you’ll join me as I share the story of one such app and the meiosis process of splitting it into several apps, services and gems.
You’ll learn about the pitfalls we faced, the bugs that plagued us, as well as get a robust set of pragmatic tips that will explain what to do when facing a daunting task such as this one.

Talk given at Rails Israel 2014

Transcript

1. MEIOSIS IN RAILS APPS @yonbergman
2. Yonatan Bergman @yonbergman
3. Meiosis
4. Meiosis
5. Tech Lead eBay Israel Innovation Center
6. eBay Valet
8. $ GET PRICE ESTIMATION SEND ITEM FOR FREE WE SELL IT, YOU GET PAID
10. SERVER
11. SERVER
12. HISTORY
13. SERVER
14. ! SERVER APP SERVERS DATABASE REDIS SIDEKIQ UTILITY SERVERS
15. SERVER
16. SERVER
17. SERVER EBAY SERVICES
18. SERVER ANALYTICS
19. EBAY SERVICES ANALYTICS SERVER
20. EBAY SERVICES ANALYTICS REPORTING AUDITING QA TOOLS CUSTOMER SUPPORT A/B TESTING SERVER PACKAGE TRACKING
21. PACKAGE TRACKING EBAY SERVICES ANALYTICS REPORTING AUDITING QA TOOLS CUSTOMER SUPPORT A/B TESTING SERVER
23. SAD PANDA
24. LAUNCH
26. DATABASE LOAD SERVER LOAD DEPLOYMENTS USER RESPONSIBILITIES COMPLEX MODEL TREE STAGING <-> PRODUCTION
27. DATABASE LOAD SERVER LOAD DEPLOYMENTS USER RESPONSIBILITIES COMPLEX MODEL TREE STAGING <-> PRODUCTION
28. DATABASE LOAD SERVER LOAD DEPLOYMENTS USER RESPONSIBILITIES COMPLEX MODEL TREE STAGING <-> PRODUCTION
29. DATABASE LOAD SERVER LOAD DEPLOYMENTS USER RESPONSIBILITIES COMPLEX MODEL TREE STAGING <-> PRODUCTION
30. 6 LEVELS OF SEPARATION
31. 6 LEVELS OF SEPARATION
32. 6 5 4 3 2 MONSTER APP 1
33. 6 5 4 3 2 1 MONSTER APP /LIB
34. 6 5 4 3 MONSTER APP /LIB NAMESPACING 2 1
35. 6 5 4 MONSTER APP /LIB NAMESPACING ENGINES 3 2 1
36. 6 5 MONSTER APP /LIB NAMESPACING ENGINES GEMS 4 3 2 1
37. 6 MONSTER APP /LIB NAMESPACING ENGINES GEMS SERVICES & SERVERS 5 4 3 2 1
38. 6 MONSTER APP /LIB NAMESPACING ENGINES GEMS SOA 5 4 3 2 1
39. SEPARATION
40. SERVER
41. EBAY GEM SERVER PRICE ESTIMATION SERVER
42. EBAY GEM SERVER PRICE ESTIMATION SERVER GEM
43. EBAY GEM SERVER PRICE ESTIMATION SERVER GEM
44. EBAY GEM SERVER PRICE ESTIMATION SERVER EBAY SERVICES GEM
45. require 'ebayr'
 require 'rebay'
 require_relative ‘services/super_secret_api' ! ! module EbayServices class Shopping def initialize @shopping = Rebay::Shopping.new end end end ! !
46. EBAY GEM SERVER PRICE ESTIMATION SERVER EBAY SERVICES GEM
47. API DESIGN
48. AUTH VERSIONING SERIALIZING PARSING ERRORS
49. AUTH VERSIONING SERIALIZING PARSING ERRORS
50. $ curl -H “TOKEN: FOO” https://example.com { ‘status’: ‘ok’ }
51. $ curl -H “TOKEN: FOO” https://example.com { ‘status’: ‘ok’ } ! $ curl https://example.com?token=FOO { ‘status’: ‘ok’ }
52. AUTH VERSIONING SERIALIZING PARSING ERRORS
53. $ curl https://example.com/api/v1/status { ‘status’: ‘ok’ }
54. $ curl https://example.com/api/v1/status { ‘status’: ‘ok’ } ! $ curl -H ‘FOO-VERSION: 1’ 
 https://example.com/api/status { ‘status’: ‘ok’ }
55. $ curl https://example.com/api/v1/status { ‘status’: ‘ok’ } ! $ curl -H ‘FOO-VERSION: 1’ 
 https://example.com/api/status { ‘status’: ‘ok’ } ! $ curl
 -H ‘Accept: application/vnd.foo.v1+json’
 https://example.com/api/status { ‘status’: ‘ok’ }
56. gem ‘versionist’
57. AUTH VERSIONING SERIALIZING PARSING ERRORS
58. gem ‘active_model_serializers’
59. class ItemSerializer < ActiveModel::Serializer attributes :uid, :title, :description, :category has_one :response 
 end
60. class API::ItemSerializer < ActiveModel::Serializer attributes :uid, :title, :description, :category has_one :response, 
 serializer: API::ResponseSerializer 
 def include_category?
 scope.api_version > 1
 end 
 end
61. class API::ItemSerializer < ActiveModel::Serializer attributes :uid, :title, :description, :category has_one :response, 
 serializer: API::ResponseSerializer 
 def include_category?
 scope.api_version > 1
 end 
 end
62. class API::ItemSerializer < ActiveModel::Serializer attributes :uid, :title, :description, :category has_one :response, 
 serializer: API::ResponseSerializer 
 def include_category?
 scope.api_version > 1
 end 
 end
63. class API::ItemSerializer < ActiveModel::Serializer attributes :uid, :title, :description, :category has_one :response, 
 serializer: API::ResponseSerializer 
 def include_category?
 scope.api_version > 1
 end 
 end
64. AUTH VERSIONING SERIALIZING PARSING ERRORS
65. SERIALIZING OBJ JSON PARSING OBJ JSON
66. gem ‘hashie’
67. class API::ItemCreator def create(params)
 Item.create(parsed_params(params))
 end def parsed_params(params)
 ItemParser.new(params).to_hash
 end class ItemParser < Hashie::Trash
 property :title, required: true
 property :description
 end end
68. class API::ItemCreator def create(params)
 Item.create(parsed_params(params))
 end def parsed_params(params)
 ItemParser.new(params).to_hash
 end class ItemParser < Hashie::Trash
 property :title, required: true
 property :description
 end end
69. class API::ItemCreator def create(params)
 Item.create(parsed_params(params))
 end def parsed_params(params)
 ItemParser.new(params).to_hash
 end class ItemParser < Hashie::Trash
 property :title, required: true
 property :description
 end end
70. class ItemParser < Hashie::Trash property :title, required: true
 property :description
 property :category_id, from: ‘category’
 property :price_cents, with: -> (price) {
 (price * 100).to_i
 } end
71. class ItemParser < Hashie::Trash property :title, required: true
 property :description
 property :category_id, from: ‘category’
 property :price_cents, with: -> (price) {
 (price * 100).to_i
 } end
72. class ItemParser < Hashie::Trash property :title, required: true
 property :description
 property :category_id, from: ‘category’
 property :price_cents, with: -> (price) {
 (price * 100).to_i
 } end
73. class ItemParser < Hashie::Trash property :title, required: true
 property :description
 property :category_id, from: ‘category’
 property :price_cents, with: -> (price) {
 (price * 100).to_i
 } end
74. class ItemParser < Hashie::Trash property :user_attributes, 
 from: ‘seller’,
 with: -> (hash) { 
 ItemParser::UserParser.new(hash) 
 } class UserParser < Hashie::Trash
 property :name, required: true
 end end
75. class ItemParser < Hashie::Trash property :user_attributes, 
 from: ‘seller’,
 with: -> (hash) { 
 ItemParser::UserParser.new(hash) 
 } class UserParser < Hashie::Trash
 property :name, required: true
 end end
76. class ItemParser < Hashie::Trash property :user_attributes, 
 from: ‘seller’,
 with: -> (hash) { 
 ItemParser::UserParser.new(hash) 
 } class UserParser < Hashie::Trash
 property :name, required: true
 end end
77. class ItemParser < Hashie::Trash property :user_attributes, 
 from: ‘seller’,
 with: -> (hash) { 
 ItemParser::UserParser.new(hash) 
 } class UserParser < Hashie::Trash
 property :name, required: true
 end end
78. class ItemParser < Hashie::Trash property :user_attributes, 
 from: ‘seller’,
 with: -> (hash) { 
 ItemParser::UserParser.new(hash) 
 } class UserParser < Hashie::Trash
 property :name, required: true
 end end
79. gem ‘representable’
 gem ‘roar’
80. AUTH VERSIONING SERIALIZING PARSING ERRORS
81. ERROR ERROR CODE ERROR MESSAGE HTTP CODE record not found 1003 The object you are looking for could not be found 404 unauthorized 1004 You are not authorized to do the following action 403 invalid param 1005 You are missing a required parameter 400 invalid value 1006 Why are you reading this slide? 400
82. module API::ErrorSupport rescue_from Exception, with: :respond_with_error def respond_with_error(e)
 error = translate_exception(e)
 render json: error, 
 status: error.status_code
 end end
83. module API::ErrorSupport rescue_from Exception, with: :respond_with_error def respond_with_error(e)
 error = translate_exception(e)
 render json: error, 
 status: error.status_code
 end end
84. module API::ErrorSupport rescue_from Exception, with: :respond_with_error def respond_with_error(e)
 error = translate_exception(e)
 render json: error, 
 status: error.status_code
 end end
85. module API::ErrorSupport def translate_exception(e)
 case e
 when ActiveRecord::RecordNotFound
 ApiError.new(1003,
 I18n.t(‘errors.api.not_found'),
 :not_found)
 ...
 end
86. module API::ErrorSupport def translate_exception(e)
 case e
 when ActiveRecord::RecordNotFound
 ApiError.new(1003,
 I18n.t(‘errors.api.not_found'),
 :not_found)
 ...
 end
87. module API::ErrorSupport def translate_exception(e)
 case e
 when ActiveRecord::RecordNotFound
 ApiError.new(1003,
 I18n.t(‘errors.api.not_found'),
 :not_found)
 ...
 end
88. module API::ErrorSupport def translate_exception(e)
 case e
 when ActiveRecord::RecordNotFound
 ApiError.new(1003,
 I18n.t(‘errors.api.not_found'),
 :not_found)
 ...
 end
89. LONG RUNNING TASKS
91. EBAY GEM SERVER PRICE ESTIMATION SERVER
92. EBAY GEM SERVER PRICE ESTIMATION SERVER
93. EBAY GEM SERVER PRICE ESTIMATION SERVER
94. EBAY GEM SERVER PRICE ESTIMATION SERVER
95. POST /api/estimations { 'estimation': { ... },
 'callback': ‘https://example.com/callback' } ! RESPONSE { 'uid': 'Y2z2Q3fOTENx4w',
 'status': ‘added’, 'estimation': { ... }
96. GET /api/estimations/:uid { 'uid': 'Y2z2Q3fOTENx4w',
 'status': 'pending' 'estimation': { ... } }
97. POST /callback { 'notification': { 'uid': 'Y2z2Q3fOTENx4w', 'event': ‘estimation_completed', 'estimation': {...} } }
98. RESPONSE 500 // Wait 1s
 RESPONSE 500 // Wait 2s
 RESPONSE 500 // Wait 3s
 RESPONSE 500 // Wait 5s
 RESPONSE 500 // Wait 8s
 RESPONSE 200
99. DATA MIGRATION
101. gem ‘trucker’
103. SERVER PRICE ESTIMATION SERVER DB DB DB
104. SERVER PRICE ESTIMATION SERVER DB DB DB
105. development: adapter: 'postgresql' encoding: 'unicode' database: 'legacy_database' host: 'localhost' pool: 30
106. class LegacyModel < ActiveRecord::Base ! YAML_FILE = ‘config/legacy_db.yml’ self.abstract_class = true ! establish_connection( YAML::load_file(YAML_FILE)[Rails.env] ) ! end
107. class LegacyItem < LegacyModel ! self.table_name = ‘items’ ! has_one :estimation, class_name: ‘LegacyEstimation’ foreign_key: ‘item_id’ ! end
108. class LegacyMigration ! def run @output = MigrationOutput.new(@file) scope = LegacyItem.from_id(@starting_id) scope.find_each do |legacy_item| migrator = ItemMigrator.new(legacy_item) migrator.run @output.add(migrator.output) end end ! end
109. JSON DB SERVER PRICE ESTIMATION SERVER DB DB
110. JSON DB SERVER PRICE ESTIMATION SERVER DB DB
111. JSON DB SERVER PRICE ESTIMATION SERVER DB DB
112. JSON DB SERVER PRICE ESTIMATION SERVER DB DB
113. 4h ~300k
114. EBAY GEM SERVER PRICE ESTIMATION SERVER EBAY SERVICES GEM
115. EBAY GEM SERVER PRICE ESTIMATION SERVER EBAY SERVICES GEM
117. jobs-il@ebay.com
118. jobs-il@ebay.com
119. @yonbergman