Replicating ActiveRecord objects to MongoDB

We had to move much of MySQL / Active Record objects to a huge MongoDB document. We'll show you the code that makes this happen and share a few of the lessons learned.

Transcript

1. MySQL.to MongoDB Thursday, November 3, 11
2. @ivanacostarubio bakedweb.net Thursday, November 3, 11
3. MySQL.to Redis Thursday, November 3, 11
4. MySQL.to Amazon SQS Thursday, November 3, 11
5. MySQL.to ________ Thursday, November 3, 11
6. VenueDriver Thursday, November 3, 11
7. Basecamp for Clubs Thursday, November 3, 11
8. Indinero for nightclubs Thursday, November 3, 11
9. Freshbooks to sell tickets Thursday, November 3, 11
10. Events Thursday, November 3, 11
11. Events Tickets Reservations Passes Guests TicketSales Thursday, November 3, 11
12. I want to know what happens in my business Thursday, November 3, 11
13. How many tickets did we sell? How many reservations? Who are the people that party the most? How many people bring the promoters? Who should I fire? Thursday, November 3, 11
14. First Iteration MySQL Thursday, November 3, 11
15. two years later :joins => :kill_us Thursday, November 3, 11
16. 5+ Minutes to generate reports. large events generated a timeout errors Thursday, November 3, 11
17. let’s kill the :joins Thursday, November 3, 11
18. non relational data store Thursday, November 3, 11
19. this is a refactoring story Thursday, November 3, 11
20. Alleluia for CUCUMBERS Thursday, November 3, 11
21. Alleluia for CUCUMBERS TESTS Thursday, November 3, 11
22. It made the refactoring easier Thursday, November 3, 11
23. this was a FAT controller Thursday, November 3, 11
24. no report model or methods in any model Thursday, November 3, 11
25. 100% controller baby Thursday, November 3, 11
26. let’s look at that code Thursday, November 3, 11
27. def get_report_data return if session[:account].blank? or session[:account].zero? start_date = params[:start].blank? ? (session[:report_start_date].blank? ? Date.today - 8.days : Date.parse(session[:report_start_date])) : Date.civil(params[:start][:year].to_i, params[:start][:month].to_i, params[:start][:day].to_i) end_date = params[:end].blank? ? (session[:report_end_date].blank? ? Date.today - 1.days : Date.parse(session[:report_end_date])) : Date.civil(params[:end][:year].to_i, params[:end][:month].to_i, params[:end][:day].to_i) end_date = start_date if start_date > end_date session[:report_end_date] = end_date.to_s session[:report_start_date] = start_date.to_s type = action_name session[:report_type] = type if session[:account].blank? or session[:account].zero? @venues = Venue.find(:all) end span = case (end_date - start_date).to_i when 0:'day' when 1..8:'week' else 'month' end superuser_data = { :total_events => 0, :events_with_data => 0, :accounts => {}, :venues => {} } total_reservations = 0 total_guests = 0 total_passes = 0 report_venues = [] # Only venues with events show up in the reports. @venues.each do |venue| events = venue.events.find(:all, :conditions => [ "date >= :start_date AND date <= :end_date", { :start_date => start_date, :end_date => end_date } ], :order => 'date').uniq # This 'uniq' is required because this query can return # the same event record more than once in the response. venue_reservations = 0 venue_guests = 0 venue_passes = 0 venue_revenue = 0 venue_returns = 0 venue_tables = 0 staff_stats = {} daily_staff_stats = {} meta_events = [] events.each do |event| superuser_data[:total_events] += 1 event_reservations = 0 event_guests = 0 event_passes = 0 event_revenue = 0 event_returns = 0 event_tables = 0 # Get total revenue for each staff member. staff_data = {} staff_data = {} reservation_count = event.reservations.find(:all).size total_reservations += reservation_count venue_reservations += reservation_count event_reservations += reservation_count reservations = event.reservations.find(:all, :conditions => "total_spend > 0 or checked_in") reservations.each do |reservation| next if reservation.staff_id.blank? # Skip orphan reservations. # For event: staff_data[reservation.staff_id] = {} if staff_data[reservation.staff_id].nil? staff_data[reservation.staff_id][:total_revenue] = 0 if staff_data[reservation.staff_id][:total_revenue].nil? staff_data[reservation.staff_id][:total_revenue] += reservation.total_spend unless reservation.total_spend.nil? staff_data[reservation.staff_id][:tables] = {} if staff_data[reservation.staff_id][:tables].nil? staff_data[reservation.staff_id][:tables][reservation.table_id] = 1 event_revenue += reservation.total_spend unless reservation.total_spend.nil? event_tables += 1 # For date: date_hash = reservation.event.date.to_s :db staff_stats[reservation.staff_id] = {} if staff_stats[reservation.staff_id].nil? staff_stats[reservation.staff_id][:total_revenue] = 0 if staff_stats[reservation.staff_id][:total_revenue].nil? staff_stats[reservation.staff_id][:total_revenue] += reservation.total_spend unless reservation.total_spend.nil? daily_staff_stats[date_hash] = {} if daily_staff_stats[date_hash].nil? daily_staff_stats[date_hash][reservation.staff_id] = {} if daily_staff_stats[date_hash][reservation.staff_id].nil? daily_staff_stats[date_hash][reservation.staff_id][:total_revenue] = 0 if daily_staff_stats[date_hash][reservation.staff_id][:total_revenue].nil? daily_staff_stats[date_hash][reservation.staff_id][:total_revenue] += reservation.total_spend unless reservation.total_spend.nil? daily_staff_stats[date_hash][reservation.staff_id][:tables] = {} if daily_staff_stats[date_hash][reservation.staff_id][:tables].nil? daily_staff_stats[date_hash][reservation.staff_id][:tables][reservation.table_id] = 1 # For venue: venue_revenue += reservation.total_spend unless reservation.total_spend.nil? venue_tables += 1 # Superuser totals superuser_data[:accounts][reservation.account_id] = 1 unless reservation.account_id.nil? superuser_data[:venues][venue.id] = 1 end # Get guest list returns for each staff member. unless type.eql? 'staff' guests = event.guests.find(:all, :conditions => "checked_in = 1 OR total_returns > 0") guest_count = guests.size guests.each do |guest| next if guest.guestlist.staff_id.blank? staff_data[guest.guestlist.staff_id] = {} if staff_data[guest.guestlist.staff_id].nil? staff_data[guest.guestlist.staff_id][:total_returns] = 0 if staff_data[guest.guestlist.staff_id][:total_returns].nil? staff_data[guest.guestlist.staff_id][:total_returns] += guest.total_returns unless guest.total_returns.nil? staff_data[guest.guestlist.staff_id][:guestlist_returns] = 0 if staff_data[guest.guestlist.staff_id][:guestlist_returns].nil? staff_data[guest.guestlist.staff_id][:guestlist_returns] += guest.total_returns unless guest.total_returns.nil? event_returns += guest.total_returns unless guest.total_returns.nil? total_guests += guest.total_returns unless guest.total_returns.nil? venue_guests += guest.total_returns unless guest.total_returns.nil? event_guests += guest.total_returns unless guest.total_returns.nil? date_hash = guest.guestlist.event.date.to_s :db staff_stats[guest.guestlist.staff_id] = {} if staff_stats[guest.guestlist.staff_id].nil? staff_stats[guest.guestlist.staff_id][:total_returns] = 0 if staff_stats[guest.guestlist.staff_id][:total_returns].nil? staff_stats[guest.guestlist.staff_id][:total_returns] += guest.total_returns unless guest.total_returns.nil? staff_stats[guest.guestlist.staff_id][:guestlist_returns] = 0 if staff_stats[guest.guestlist.staff_id][:guestlist_returns].nil? staff_stats[guest.guestlist.staff_id][:guestlist_returns] += guest.total_returns unless guest.total_returns.nil? daily_staff_stats[date_hash] = {} if daily_staff_stats[date_hash].nil? daily_staff_stats[date_hash][guest.guestlist.staff_id] = {} if daily_staff_stats[date_hash][guest.guestlist.staff_id].nil? daily_staff_stats[date_hash][guest.guestlist.staff_id][:total_returns] = 0 if daily_staff_stats[date_hash][guest.guestlist.staff_id][:total_returns].nil? daily_staff_stats[date_hash][guest.guestlist.staff_id][:total_returns] += guest.total_returns unless guest.total_returns.nil? daily_staff_stats[date_hash][guest.guestlist.staff_id][:guestlist_returns] = 0 if daily_staff_stats[date_hash][guest.guestlist.staff_id][:guestlist_returns].nil? daily_staff_stats[date_hash][guest.guestlist.staff_id][:guestlist_returns] += guest.total_returns unless guest.total_returns.nil? venue_returns += guest.total_returns unless guest.total_returns.nil? superuser_data[:accounts][guest.guestlist.account_id] = 1 unless guest.guestlist.account_id.nil? superuser_data[:venues][venue.id] = 1 end # Get guest list returns for each staff member. unless type.eql? 'staff' passes = event.passes.find(:all, :conditions => "total > 0") pass_count = passes.size passes.each do |pass| next if pass.staff_id.blank? staff_data[pass.staff_id] = {} if staff_data[pass.staff_id].nil? staff_data[pass.staff_id][:total_returns] = 0 if staff_data[pass.staff_id][:total_returns].nil? staff_data[pass.staff_id][:total_returns] += pass.total unless pass.total.nil? staff_data[pass.staff_id][:pass_returns] = 0 if staff_data[pass.staff_id][:pass_returns].nil? staff_data[pass.staff_id][:pass_returns] += pass.total unless pass.total.nil? event_returns += pass.total unless pass.total.nil? total_passes += pass.total unless pass.total.nil? venue_passes += pass.total unless pass.total.nil? event_passes += pass.total unless pass.total.nil? date_hash = pass.event.date.to_s :db staff_stats[pass.staff_id] = {} if staff_stats[pass.staff_id].nil? staff_stats[pass.staff_id][:total_returns] = 0 if staff_stats[pass.staff_id][:total_returns].nil? staff_stats[pass.staff_id][:total_returns] += pass.total unless pass.total.nil? staff_stats[pass.staff_id][:pass_returns] = 0 if staff_stats[pass.staff_id][:pass_returns].nil? staff_stats[pass.staff_id][:pass_returns] += pass.total unless pass.total.nil? daily_staff_stats[date_hash] = {} if daily_staff_stats[date_hash].nil? daily_staff_stats[date_hash][pass.staff_id] = {} if daily_staff_stats[date_hash][pass.staff_id].nil? daily_staff_stats[date_hash][pass.staff_id][:total_returns] = 0 if daily_staff_stats[date_hash][pass.staff_id][:total_returns].nil? daily_staff_stats[date_hash][pass.staff_id][:total_returns] += pass.total unless pass.total.nil? daily_staff_stats[date_hash][pass.staff_id][:pass_returns] = 0 if daily_staff_stats[date_hash][pass.staff_id][:pass_returns].nil? daily_staff_stats[date_hash][pass.staff_id][:pass_returns] += pass.total unless pass.total.nil? venue_returns += pass.total unless pass.total.nil? superuser_data[:accounts][pass.account_id] = 1 unless pass.account_id.nil? superuser_data[:venues][venue.id] = 1 end end end # Sort by revenue and put a whole object into an array. meta_staff = [] staff_data.keys.each do |staff_id| unless staff_id.blank? begin person = Staff.find(staff_id) rescue next # Orphan reservations cause this. end tables = (staff_data[staff_id].nil? or staff_data[staff_id][:tables].nil?) ? 0 : staff_data[staff_id][:tables].keys.size revenue = staff_data[staff_id].nil? ? 0 : staff_data[staff_id][:total_revenue] returns = (staff_data.nil? or staff_data[staff_id].nil?) ? 0 : staff_data[staff_id][:total_returns] guestlists = (staff_data.nil? or staff_data[staff_id].nil?) ? 0 : staff_data[staff_id][:guestlist_returns] passes = (staff_data.nil? or staff_data[staff_id].nil?) ? 0 : staff_data[staff_id][:pass_returns] unless (tables.blank? or tables.zero?) and (revenue.blank? or revenue.zero?) and (returns.blank? or returns.zero?) meta_staff << { :object => person, :total_tables => tables, :total_revenue => revenue || 0, :total_returns => returns, :guestlist_returns => guestlists, :pass_returns => passes } end end end meta_staff.sort! {|a,b| b[:total_revenue] <=> a[:total_revenue]} meta_events << { :object => event, :staff => meta_staff, :total_reservations => event_reservations, :total_guests => event_guests, :total_passes => event_passes, :total_tables => event_tables, :total_returns => event_returns, :total_revenue => event_revenue } superuser_data[:events_with_data] += 1 if(event_tables > 0 or event_returns > 0 or event_revenue > 0) end # Sort by revenue and put a whole object into an array. dates = [] daily_staff_stats.keys.sort.each do |date_hash| dates << Date.parse(date_hash) end meta_staff = [] staff_stats.keys.each do |staff_id| unless staff_id.blank? begin person = Staff.find(staff_id) rescue next end tables = (staff_stats[staff_id].nil? or staff_stats[staff_id][:tables].nil?) ? 0 : staff_stats[staff_id][:tables].keys.size revenue = staff_stats[staff_id].nil? ? 0 : staff_stats[staff_id][:total_revenue] returns = staff_stats[staff_id].nil? ? 0 : staff_stats[staff_id][:total_returns] unless (tables.blank? or tables.zero?) and (revenue.blank? or revenue.zero?) and (returns.blank? or returns.zero?) # logger.warn "STAFF: #{person.name} '#{tables}' '#{revenue}' '#{returns}'" meta_staff << { :object => person, :total_tables => tables, :total_revenue => revenue, :total_returns => returns } end end end meta_staff.sort! {|a,b| (b[:total_revenue].blank? ? 0 : b[:total_revenue]) <=> (a[:total_revenue].blank? ? 0 : a[:total_revenue]) } if events.size > 0 report_venues << { :object => venue, :events => meta_events, # Array of meta hashes for events with object pointer and meta data. :staff => meta_staff, # A sorted list of staff for this venue, for this report. Only totals. :daily_staff_stats => daily_staff_stats, # The actual day-by-day breakdown for staff members. :dates => dates, # Array of meta hashes for each date. :total_reservations => venue_reservations, :total_guests => venue_guests, :total_tables => venue_tables, :total_returns => venue_returns, :total_revenue => venue_revenue } end end venue_data_index = {} report_venues.each_with_index do |venue, index| venue_data_index[venue[:object].id] = index end if span.eql? 'month' report_venues.each do |venue| weeks = {} venue[:weekly_staff_stats] = {} venue[:staff].each do |person| venue[:dates].each do |date| days_in = date - start_date week_number = (days_in / 7).to_i weeks[week_number] = true venue[:weekly_staff_stats][week_number] = {} if venue[:weekly_staff_stats][week_number].nil? if venue[:weekly_staff_stats][week_number][person[:object].id].nil? venue[:weekly_staff_stats][week_number][person[:object].id] = { :total_revenue => 0, :total_returns => 0 } end stats_for_date = venue[:daily_staff_stats][date.to_s(:db)] next if stats_for_date[person[:object].id].nil? stats_for_week = venue[:weekly_staff_stats][week_number] stats_for_week[person[:object].id] = {} if stats_for_week[person[:object].id].nil? stats_for_week[person[:object].id][:total_revenue] += stats_for_date[person[:object].id][:total_revenue] unless stats_for_date[person[:object].id][:total_revenue].nil? stats_for_week[person[:object].id][:total_returns] += stats_for_date[person[:object].id][:total_returns] unless stats_for_date[person[:object].id][:total_returns].nil? end end venue[:weeks] = weeks.keys.sort end end # logger.warn "VENUES: #{report_venues.to_json}" # logger.warn "DATA: #{superuser_data[:venues].keys.join(", ")}" superuser_data[:total_reservations] = total_reservations superuser_data[:total_guests] = total_guests @report_data = { :start_date => start_date, :end_date => end_date, :span => span, :type => type, :venues => report_venues, :venue_data_index => venue_data_index, :superuser_data => superuser_data } end One method in the controller 356 lines Thursday, November 3, 11
28. gem install fog Thursday, November 3, 11
29. +- 20 => great +- 40 => OK +- 60 => needs refactor + 80 => out of control Thursday, November 3, 11
30. BE AWARE Thursday, November 3, 11
31. 1325.1: Reports::VenuesController #get_report_data venues_controller.rb:157 Thursday, November 3, 11
32. Goals Thursday, November 3, 11
33. Fast Reports Thursday, November 3, 11
34. push from controller to model Thursday, November 3, 11
35. Solution Thursday, November 3, 11
36. 1 HUGE document Thursday, November 3, 11
37. no joins Thursday, November 3, 11
38. modules for data aggregation Thursday, November 3, 11
39. LET’S TDD Thursday, November 3, 11
40. def test_replicates_an_event Event.create assert_equal 1, CachedEvent.all.to_a.size end Thursday, November 3, 11
41. class Event < ActiveRecord::Base after_save :create_replicated_record def create_replicated_record CachedEvent.create(self.attributes) end end Thursday, November 3, 11
42. def test_updates_replicated_record @event = Event.create @event.title = "BR" @event.save assert_equal "BR", CachedEvent.last.title end Thursday, November 3, 11
43. class Event < ActiveRecord::Base after_update :update_replicated_record def update_replicated_record cached = CachedEvent.find(id) cached.update_attributes(attributes) end end Thursday, November 3, 11
44. Warning about Dates atributos.merge!("date" => Date.to_mongo(self.date)) stole it from MongoMapper Thursday, November 3, 11
45. Deleting Unnecessary Attributes def mongo_attributes invalid_keys = [ "created_at", "updated_at"] atributos = self.attributes atributos = attributes.delete_if {|k,v| invalid_keys.include?(k)} end Thursday, November 3, 11
46. class Event < ActiveRecord::Base after_update :update_replicated_record def update_replicated_record cached = CachedEvent.find(id) cached.update_attributes(mongo_attributes) end end Thursday, November 3, 11
47. Adding Attributes easy querying def mongo_attributes ... atributos.merge!("venue_id" => self.venue.id) ... end Thursday, November 3, 11
48. def test_destroy_cached_event Event.create ; Event.destroy_all assert_equal 0, CachedEvent.all.size end Thursday, November 3, 11
49. class Event < ActiveRecord::Base after_destroy :destroy_replicated_record def destroy_replicated_record CachedEvent.find(self.id).destroy end end Thursday, November 3, 11
50. REFACTOR Thursday, November 3, 11
51. class Event < ActiveRecord::Base include MongoRelication::Event end Thursday, November 3, 11
52. module MongoRelication module Event .... end end Thursday, November 3, 11
53. Going META Thursday, November 3, 11
54. module MongoReplication module Event def self.included(base) base.class_eval do after_create :create_delayed_replication after_update :update_delayed_replication after_destroy :destroy_cached_event end end # define methods as usual def create_delayed_replicated ; .... ; end # .... end end Thursday, November 3, 11
55. R U CRAZY slow on the web requests Thursday, November 3, 11
56. gem install resque Thursday, November 3, 11
57. better than DJ, but depends on Redis Thursday, November 3, 11
58. class ResqueCreation @queue = :create def self.perform(id) record = Event.find(id) record.create_replicated_record end end Thursday, November 3, 11
59. How do I test it? Thursday, November 3, 11
60. def create_resque_replication if RAILS_ENV == "test" self.create_cached else Resque.enqueue(ResqueCreation, self.class.name, self.id) end end Thursday, November 3, 11
61. +- 20 => great +- 40 => OK +- 60 => needs refactor + 80 => out of control Thursday, November 3, 11
62. 35.0: ReportAgregator#none 27.7: CachedEvent::replicate_all_accounts 25.2: ReportAgregator#guest_returns_count 17.4: Date::to_mongo 17.0: CachedEvent::find_or_create_by_event_id 15.7: ReportAgregator#file_name_export 13.3: CachedEvent::create_indices Thursday, November 3, 11
63. 33.3: Reports::EventsController#index 14.3: Reports::EventsController#have_account_but_not_venue 13.2: Reports::EventsController#export_ticket_sales_group_by_staff 9.7: Reports::EventsController#show 9.1: Reports::EventsController#have_venue_selected Thursday, November 3, 11
64. Report Aggregator class CachedEvent include ReportAggregator end Thursday, November 3, 11
65. Report Aggregator module ReportAggregator def returns_count ... end def total_reservations_sat ... end end Thursday, November 3, 11
66. Fake it! class Event < ActiveRecord::Base include Mongoreplication::Event end VS class EventFake include MongoReplication::Event end Thursday, November 3, 11
67. google: fast rails tests follow @coreyhaines join: destroyallsoftware.com Thursday, November 3, 11
68. ReplicationPolice Thursday, November 3, 11
69. class ReplicationPoliceTest < ActiveSupport::TestCase def test_event_and_replication_got_out_of_sync event = Factory(:event) CachedEvent.destroy_all cop = ReplicationPolice.new(event) assert_equal false, cop.accurate? end end Thursday, November 3, 11
70. class ReplicationPolice def initialize(event) @event = Event.find(event.id) @cached_event = .... end def accurate? return false unless is_event_replicated? return false unless has_all_sales? true end def is_even_replicated? ... end end Thursday, November 3, 11
71. need real time? Thursday, November 3, 11
72. Thursday, November 3, 11
73. next steps ... Thursday, November 3, 11
74. next steps ... atomic updates Thursday, November 3, 11
75. next steps ... atomic updates overwrite finders Thursday, November 3, 11
76. next steps ... atomic updates overwrite finders shorter attributes names Thursday, November 3, 11
77. next steps ... atomic updates overwrite finders shorter attributes names gem Thursday, November 3, 11
78. thanks @ivanacostarubio Thursday, November 3, 11