Facing the Monolith: Overcoming Monolithic Applications with SOA

For many small sites with a minimal amount of complexity, a single rails application works fine. The problem is that as the application's complexity and scope grow, several problems arise. These include: heavy coupling, increased load and response times, and test complexity. All of these cause your feature development to slow considerably.

The concepts to solving these problems is relatively simple. The primary approach is extracting each concern in your application into its own service. The trick is extricating the data and classes when they are interdepend on the other aspects of the application.

We'll go over how this is done and what it meant when we had successfully teased the application apart. This approach was used on an application that collected, stored and managed millions of leads each month.

Transcript

1. Facing the Monolith Charles Max Wood Tuesday, October 9, 12
2. Charles Max Wood ‣ Lead Developer at Intentional Excellence Productions ‣ Host of Ruby Rogues, Javascript Jabber, and Ruby Freelancers podcasts Tuesday, October 9, 12
3. Tuesday, October 9, 12
4. How it Starts... Tuesday, October 9, 12
5. How it Starts... Tame this! Tuesday, October 9, 12
6. How it Fails... Tuesday, October 9, 12
7. How it Fails... Tame this! Tuesday, October 9, 12
8. How We Started... Tuesday, October 9, 12
9. How We Started... Tuesday, October 9, 12
10. How We Started... Tuesday, October 9, 12
11. How We Started... Tuesday, October 9, 12
12. How We Started... Tuesday, October 9, 12
13. How We Started... Tuesday, October 9, 12
14. How We Started... Tuesday, October 9, 12
15. How We Started... Tuesday, October 9, 12
16. How We Started... Tuesday, October 9, 12
17. How We Started... Tuesday, October 9, 12
18. How We Started... Tuesday, October 9, 12
19. How We Started... Tuesday, October 9, 12
20. How We Started... Tuesday, October 9, 12
21. How We Started... Tuesday, October 9, 12
22. How We Started... Tuesday, October 9, 12
23. How We Started... Tuesday, October 9, 12
24. Tuesday, October 9, 12
25. Tuesday, October 9, 12
26. Tuesday, October 9, 12
27. Tuesday, October 9, 12
28. Tuesday, October 9, 12
29. Tuesday, October 9, 12
30. Tuesday, October 9, 12
31. Tuesday, October 9, 12
32. How it Fails... Tame this! Tuesday, October 9, 12
33. Tuesday, October 9, 12
34. Your Backlog Tuesday, October 9, 12
35. Tuesday, October 9, 12
36. Tuesday, October 9, 12
37. Organization Tuesday, October 9, 12
38. Organization Tuesday, October 9, 12
39. Organization Tuesday, October 9, 12
40. Organization Tuesday, October 9, 12
41. Communication Tuesday, October 9, 12
42. Tuesday, October 9, 12
43. Tuesday, October 9, 12
44. HTTP lead_path = 'http://leads.myapp.com/leads/1.json' response = HTTParty.get(lead_path) #... Send lead to a school ... data = {lead_id: 1, school_id: 3, sent_at: ‘2012-10-2’} reporting_path = 'http://reporting.myapp.com/lead_sent/' HTTParty.post(reporting_path, {body: data}) Tuesday, October 9, 12
45. HTTP lead_path = 'http://leads.myapp.com/leads/1.json' response = HTTParty.get(lead_path) #... Send lead to a school ... data = {lead_id: 1, school_id: 3, sent_at: ‘2012-10-2’} reporting_path = 'http://reporting.myapp.com/lead_sent/' HTTParty.post(reporting_path, {body: data}) Tuesday, October 9, 12
46. HTTP lead_path = 'http://leads.myapp.com/leads/1.json' response = HTTParty.get(lead_path) #... Send lead to a school ... data = {lead_id: 1, school_id: 3, sent_at: ‘2012-10-2’} reporting_path = 'http://reporting.myapp.com/lead_sent/' HTTParty.post(reporting_path, {body: data}) Tuesday, October 9, 12
47. HTTP lead_path = 'http://leads.myapp.com/leads/1.json' response = HTTParty.get(lead_path) #... Send lead to a school ... data = {lead_id: 1, school_id: 3, sent_at: ‘2012-10-2’} reporting_path = 'http://reporting.myapp.com/lead_sent/' HTTParty.post(reporting_path, {body: data}) Tuesday, October 9, 12
48. HTTP Tuesday, October 9, 12
49. HTTP ‣ Synchronous Tuesday, October 9, 12
50. HTTP ‣ Synchronous ‣ Response Needed to Continue Tuesday, October 9, 12
51. HTTP ‣ Synchronous ‣ Response Needed to Continue ‣ Immediate Action Tuesday, October 9, 12
52. HTTP ‣ Synchronous ‣ Response Needed to Continue ‣ Immediate Action ‣ Information Requests Tuesday, October 9, 12
53. Organization Tuesday, October 9, 12
54. Organization Tuesday, October 9, 12
55. Queue/Worker Tuesday, October 9, 12
56. Resque class LeadSubmitter @queue = :sell_leads def perform(lead_id, school_id = nil) lead = Lead.request(lead_id) school = School.find_by_id(school_id) if school_id school ||= SchoolFinder.find_school_for(lead) school.submit(lead) Resque.enqueue(Reporting, :lead_submitted, lead_id, school.id, Time.now) end end class Lead < ActiveRecord::Base after_create :sell_lead def sell_lead Resque.enqueue(LeadSubmitter, self.id, self.school_id) end end Tuesday, October 9, 12
57. Resque class LeadSubmitter @queue = :sell_leads def perform(lead_id, school_id = nil) lead = Lead.request(lead_id) school = School.find_by_id(school_id) if school_id school ||= SchoolFinder.find_school_for(lead) school.submit(lead) Resque.enqueue(Reporting, :lead_submitted, lead_id, school.id, Time.now) end end class Lead < ActiveRecord::Base after_create :sell_lead def sell_lead Resque.enqueue(LeadSubmitter, self.id, self.school_id) end end Tuesday, October 9, 12
58. Resque class LeadSubmitter @queue = :sell_leads def perform(lead_id, school_id = nil) lead = Lead.request(lead_id) school = School.find_by_id(school_id) if school_id school ||= SchoolFinder.find_school_for(lead) school.submit(lead) Resque.enqueue(Reporting, :lead_submitted, lead_id, school.id, Time.now) end end class Lead < ActiveRecord::Base after_create :sell_lead def sell_lead Resque.enqueue(LeadSubmitter, self.id, self.school_id) end end Tuesday, October 9, 12
59. Resque class LeadSubmitter @queue = :sell_leads def perform(lead_id, school_id = nil) lead = Lead.request(lead_id) school = School.find_by_id(school_id) if school_id school ||= SchoolFinder.find_school_for(lead) school.submit(lead) Resque.enqueue(Reporting, :lead_submitted, lead_id, school.id, Time.now) end end class Lead < ActiveRecord::Base after_create :sell_lead def sell_lead Resque.enqueue(LeadSubmitter, self.id, self.school_id) end end Tuesday, October 9, 12
60. Resque class LeadSubmitter @queue = :sell_leads def perform(lead_id, school_id = nil) lead = Lead.request(lead_id) school = School.find_by_id(school_id) if school_id school ||= SchoolFinder.find_school_for(lead) school.submit(lead) Resque.enqueue(Reporting, :lead_submitted, lead_id, school.id, Time.now) end end class Lead < ActiveRecord::Base after_create :sell_lead def sell_lead Resque.enqueue(LeadSubmitter, self.id, self.school_id) end end Tuesday, October 9, 12
61. Resque class LeadSubmitter @queue = :sell_leads def perform(lead_id, school_id = nil) lead = Lead.request(lead_id) school = School.find_by_id(school_id) if school_id school ||= SchoolFinder.find_school_for(lead) school.submit(lead) Resque.enqueue(Reporting, :lead_submitted, lead_id, school.id, Time.now) end end class Lead < ActiveRecord::Base after_create :sell_lead def sell_lead Resque.enqueue(LeadSubmitter, self.id, self.school_id) end end Tuesday, October 9, 12
62. Queuing Tuesday, October 9, 12
63. Queuing ‣ Asynchronous Tuesday, October 9, 12
64. Queuing ‣ Asynchronous ‣ Response Not Needed to Continue Tuesday, October 9, 12
65. Queuing ‣ Asynchronous ‣ Response Not Needed to Continue ‣ Writing New Data Tuesday, October 9, 12
66. Queuing ‣ Asynchronous ‣ Response Not Needed to Continue ‣ Writing New Data ‣ Jobs Done Out of Band Tuesday, October 9, 12
67. Organization Tuesday, October 9, 12
68. Organization Tuesday, October 9, 12
69. Feeds Tuesday, October 9, 12
70. JSON Feed class LeadController < ApplicationController def feed marker = params[:marker] leads = Lead.where(['id > ?', marker.to_i]).all if marker leads ||= Lead.all render json: leads end end # daemon require 'lead' while(true) do marker = 0 response = HTTParty.get("http://leads.myapp.com/feed?marker=#{marker}") lead_hashes = response.parsed_response lead_hashes.each do |lead_hash| lead = Lead.new(lead_hash) Worker.process lead end marker = lead_hashes.last[:id] unless lead_hashes.empty? sleep(1) if lead_hashes.empty? end Tuesday, October 9, 12
71. JSON Feed class LeadController < ApplicationController def feed marker = params[:marker] leads = Lead.where(['id > ?', marker.to_i]).all if marker leads ||= Lead.all render json: leads end end # daemon require 'lead' while(true) do marker = 0 response = HTTParty.get("http://leads.myapp.com/feed?marker=#{marker}") lead_hashes = response.parsed_response lead_hashes.each do |lead_hash| lead = Lead.new(lead_hash) Worker.process lead end marker = lead_hashes.last[:id] unless lead_hashes.empty? sleep(1) if lead_hashes.empty? end Tuesday, October 9, 12
72. JSON Feed class LeadController < ApplicationController def feed marker = params[:marker] leads = Lead.where(['id > ?', marker.to_i]).all if marker leads ||= Lead.all render json: leads end end # daemon require 'lead' while(true) do marker = 0 response = HTTParty.get("http://leads.myapp.com/feed?marker=#{marker}") lead_hashes = response.parsed_response lead_hashes.each do |lead_hash| lead = Lead.new(lead_hash) Worker.process lead end marker = lead_hashes.last[:id] unless lead_hashes.empty? sleep(1) if lead_hashes.empty? end Tuesday, October 9, 12
73. JSON Feed class LeadController < ApplicationController def feed marker = params[:marker] leads = Lead.where(['id > ?', marker.to_i]).all if marker leads ||= Lead.all render json: leads end end # daemon require 'lead' while(true) do marker = 0 response = HTTParty.get("http://leads.myapp.com/feed?marker=#{marker}") lead_hashes = response.parsed_response lead_hashes.each do |lead_hash| lead = Lead.new(lead_hash) Worker.process lead end marker = lead_hashes.last[:id] unless lead_hashes.empty? sleep(1) if lead_hashes.empty? end Tuesday, October 9, 12
74. JSON Feed class LeadController < ApplicationController def feed marker = params[:marker] leads = Lead.where(['id > ?', marker.to_i]).all if marker leads ||= Lead.all render json: leads end end # daemon require 'lead' while(true) do marker = 0 response = HTTParty.get("http://leads.myapp.com/feed?marker=#{marker}") lead_hashes = response.parsed_response lead_hashes.each do |lead_hash| lead = Lead.new(lead_hash) Worker.process lead end marker = lead_hashes.last[:id] unless lead_hashes.empty? sleep(1) if lead_hashes.empty? end Tuesday, October 9, 12
75. JSON Feed class LeadController < ApplicationController def feed marker = params[:marker] leads = Lead.where(['id > ?', marker.to_i]).all if marker leads ||= Lead.all render json: leads end end # daemon require 'lead' while(true) do marker = 0 response = HTTParty.get("http://leads.myapp.com/feed?marker=#{marker}") lead_hashes = response.parsed_response lead_hashes.each do |lead_hash| lead = Lead.new(lead_hash) Worker.process lead end marker = lead_hashes.last[:id] unless lead_hashes.empty? sleep(1) if lead_hashes.empty? end Tuesday, October 9, 12
76. Feeds Tuesday, October 9, 12
77. Feeds ‣ Asynchronous Tuesday, October 9, 12
78. Feeds ‣ Asynchronous ‣ Idempotence Tuesday, October 9, 12
79. Feeds ‣ Asynchronous ‣ Idempotence ‣ Events Tuesday, October 9, 12
80. Feeds ‣ Asynchronous ‣ Idempotence ‣ Events ‣ Allows Full Replay Tuesday, October 9, 12
81. Front End Services Tuesday, October 9, 12
82. Organization Tuesday, October 9, 12
83. Organization Tuesday, October 9, 12
84. Organization Tuesday, October 9, 12
85. User Management Tuesday, October 9, 12
86. CMS Tuesday, October 9, 12
87. CMS ‣ Refinery CMS Tuesday, October 9, 12
88. CMS ‣ Refinery CMS ‣ Radiant CMS Tuesday, October 9, 12
89. CMS ‣ Refinery CMS ‣ Radiant CMS ‣ Build Your Own Tuesday, October 9, 12
90. A Few Notes... lead_path = 'http://leads.myapp.com/leads/1.json' response = HTTParty.get(lead_path) #... Send lead to a school ... data = {lead_id: 1, school_id: 3, sent_at: ‘2012-10-2’} reporting_path = 'http://reporting.myapp.com/lead_sent/' HTTParty.post(reporting_path, {body: data}) Tuesday, October 9, 12
91. A Few Notes... lead_path = 'http://leads.myapp.com/leads/1.json' response = HTTParty.get(lead_path) #... Send lead to a school ... data = {lead_id: 1, school_id: 3, sent_at: ‘2012-10-2’} reporting_path = 'http://reporting.myapp.com/lead_sent/' HTTParty.post(reporting_path, {body: data}) Tuesday, October 9, 12
92. A Few Notes... class LeadController < ApplicationController def feed marker = params[:marker] leads = Lead.where(['id > ?', marker.to_i]).all if marker leads ||= Lead.all render json: leads end end # daemon require 'lead' while(true) do marker = 0 response = HTTParty.get("http://leads.myapp.com/feed?marker=#{marker}") lead_hashes = response.parsed_response lead_hashes.each do |lead_hash| lead = Lead.new(lead_hash) Worker.process lead end marker = lead_hashes.last[:id] unless lead_hashes.empty? sleep(1) if lead_hashes.empty? end Tuesday, October 9, 12
93. Naming Services Tuesday, October 9, 12
94. Naming Services ‣ Domains Tuesday, October 9, 12
95. Naming Services ‣ Domains ‣ Subdomains (DNS) Tuesday, October 9, 12
96. Naming Services ‣ Domains ‣ Subdomains (DNS) ‣ Paths (rack-mount) Tuesday, October 9, 12
97. Naming Services ‣ Domains ‣ Subdomains (DNS) ‣ Paths (rack-mount) ‣ Server Name Tuesday, October 9, 12
98. Naming Services ‣ Domains ‣ Subdomains (DNS) ‣ Paths (rack-mount) ‣ Server Name ‣ IP Address Tuesday, October 9, 12
99. Security ‣ SSL ‣ Encryption(Shared key) ‣ Server Registration ‣ API keys ‣ IP Whitelist (hosts.allow) ‣ Firewalls ‣ Authentication/Authorization Tuesday, October 9, 12
100. Online Training ‣ http://bit.ly/SZMoxY Tuesday, October 9, 12
101. Charles Max Wood ‣ Lead Developer at Intentional Excellence Productions ‣ http://rubyrogues.com ‣ @cmaxw ‣ 801-367-6164 ‣ chuck@teachmetocode.com Tuesday, October 9, 12