MongoNYC 2012: Exploiting Mongo's Schemaless Nature in Rails

MongoNYC 2012: Exploiting Mongo's Schemaless Nature in Rails, Kareem Kouddos, Crowdtap. One of the most awesome benefits of Mongo is that its schemaless. This means you can do some really interesting things in your models such as dynamically adding attributes to the model without requiring a migration. The simplest example is adding createdat or updatedat timestamps. We'll go over some cool examples and discuss best practices. We'll cover best practices to avoid downtime in production while migrating to a different data representation.

Transcript

2. HI. MY NAME IS KAREEM KOUDDOUS. I’M CURRENTLY HEAD OF TECHNOLOGY AT CROWDTAP AND A FOUNDING TEAM MEMBER. I LOVE HACKING ON SIDE PROJECTS AND TAKING LONG WALKS IN MY NEIGHBORHOOD.
3. AND I’M HERE TO TALK ABOUT EXPLOITING MONGO’S SCHEMALESS NATURE IN RAILS
4. WTF IS CROWDTAP? Crowdtap  allows  brands  to   collaborate  with  their  most   passionate  consumers,  ac5va5ng   them  as  peer  influencers  for  scalable   word-­‐of-­‐mouth  marke5ng  both  on   and  offline.   -­‐  Official  
5. HUMBLEBRAGGING Over 200,000 members Working with 5 of the top 10 marketers 30 million awards earned by members 70k LOC 1:2.8 Code-to-Test ratio Deploy on average 5 times a day
6. WHY MONGO Schemaless nature ideal for the inheritance hierarchy in our domain models The possibility of auto-sharding with no application level impact which forces a scalable schema design 10gen team is awesome and right around the corner from us
7. RAILS ODMS We chose Mongoid over MongoMapper because of the lazy evaluation of criteria and use of a cursor instead of loading the entire result set. We were also able to meet with Durran (the author) and hear about his plans and level of commitment
8. A MODEL IN MONGOID class BrandAction include Mongoid::Document include Mongoid::Timestamps include Mixins::Sluggable field :title field :state field :category field :targeted_at, :type => DateTime field :launched_at, :type => DateTime field :closed_at, :type => DateTime embeds_one :member_filter embeds_many :contribution_summaries referenced_in :client references_many :activity_feed_items index [['created_at', Mongo::ASCENDING]] end
9. A MODEL IN MONGOID class BrandAction include Mongoid::Document include Mongoid::Timestamps include Mixins::Sluggable field :title field :state field :category field :targeted_at, :type => DateTime field :launched_at, :type => DateTime field :closed_at, :type => DateTime embeds_one :member_filter embeds_many :contribution_summaries referenced_in :client references_many :activity_feed_items index [['created_at', Mongo::ASCENDING]] end
10. A MODEL IN MONGOID class BrandAction include Mongoid::Document include Mongoid::Timestamps include Mixins::Sluggable field :title field :state field :category field :targeted_at, :type => DateTime field :launched_at, :type => DateTime field :closed_at, :type => DateTime embeds_one :member_filter embeds_many :contribution_summaries referenced_in :client references_many :activity_feed_items index [['created_at', Mongo::ASCENDING]] end
11. A MODEL IN MONGOID class BrandAction include Mongoid::Document include Mongoid::Timestamps include Mixins::Sluggable field :title field :state field :category field :targeted_at, :type => DateTime field :launched_at, :type => DateTime field :closed_at, :type => DateTime embeds_one :member_filter embeds_many :contribution_summaries referenced_in :client references_many :activity_feed_items index [['created_at', Mongo::ASCENDING]] end
12. INHERITANCE class LongAction < BrandAction field :moderation_queue, :type => Array, :default => [] field :approved_member_ids, :type => Array, :default => [] field :rejected_member_ids, :type => Array, :default => [] def should_create_feed_item? false end end class ModeratedDiscussion < LongAction field :allow_member_media, :type => Boolean, :default => false field :allow_member_links, :type => Boolean, :default => false def should_create_feed_item? allow_member_links && allow_member_media end end
13. INHERITANCE >> BrandAction.where(:created_at => {$gt => 3.days.ago).count 15 >> LongAction. where(:created_at => {$gt => 3.days.ago).count 10 >> ModeratedDiscussion.where(:created_at => {$gt => 3.days.ago).count 4
14. LESSONS LEARNED •  Inheritance with Mongo + Mongoid + Rails is so simple & powerful •  Only cost: Startup time increases as all models need to preloaded
15. MIXINS class BrandAction include Mongoid::Document include Mongoid::Timestamps include Mixins::Sluggable field :title field :state field :category field :targeted_at, :type => DateTime field :launched_at, :type => DateTime field :closed_at, :type => DateTime embeds_one :member_filter embeds_many :contribution_summaries referenced_in :client references_many :activity_feed_items index [['created_at', Mongo::ASCENDING]] end
16. MIXINS >> BrandAction.last.created_at => 2012-04-19 16:52:26 -0400 >> BrandAction.last.updated_at => 2012-04-19 17:02:47 -0400
17. MIXINS class BrandAction include Mongoid::Document include Mongoid::Timestamps include Mixins::Sluggable field :title field :state field :category field :targeted_at, :type => DateTime field :launched_at, :type => DateTime field :closed_at, :type => DateTime embeds_one :member_filter embeds_many :contribution_summaries referenced_in :client references_many :activity_feed_items index [['created_at', Mongo::ASCENDING]] end
18. MIXINS module Mixins::Sluggable extend ActiveSupport::Concern included do field :slug private :slug= before_create :generate_slug validates_uniqueness_of :slug, :on => :create class << self alias_method_chain :find, :slug end end end
19. MIXINS def find_with_slug(*args) if args.size == 1 slug = args.first if slug.is_a?(Hash) && slug.has_key?("$oid") self.find_without_slug(*args) elsif result = where(:slug => slug).first result else if BSON::ObjectId.legal?(slug.to_s) || %w(first last all).include?(slug.to_s) || slug.is_a?(Array) self.find_without_slug(*args) else raise Mongoid::Errors::DocumentNotFound.new(self, slug) end end else self.find_without_slug(*args) end end
20. MIXINS def generate_slug sequence = 0 generated_slug = slug_source.slice(0, TRUNCATION_LENGTH).slugize while(self.class.raw_collection.find(:slug => generated_slug).count > 0) do sequence += 1 truncated_slug_source = slug_source.slice(0, (TRUNCATION_LENGTH - sequence.to_s.length)) generated_slug = [truncated_slug_source, sequence].join(' ').slugize end self.slug = generated_slug end
21. LESSONS LEARNED •  Mixins are a great way to encapsulate shared behavior and state •  Be careful when mixing in behavior that uses the where clause (it will only be scoped to the current model)
22. MIGRATIONS You still need them! •  Schema changes (e.g. splitting “Name” into “First Name” and “Last Name”) •  Cached field initialization
23. ADDING A COUNTER CACHE class LongAction < BrandAction field :blog_posts_count, :type => Integer, :default => 0 def increment_blog_posts_count inc(:blog_posts_count, 1) blog_posts_count end end class BlogFeedItem < FeedItem after_create :increment_blog_posts_count_on_long_action private def increment_blog_posts_count_on_long_action brand_action.increment_blog_posts_count end end
24. ADDING A COUNTER CACHE You need to write a migration or a lazy migration.
25. MIGRATION brand_actions.find({ :_type => "LongAction" } }, { :timeout => false }) do |cursor| cursor.each do |brand_action| blog_posts_count = db.feed_items.find( {:_type => "BlogFeedItem", :brand_action_id => brand_action["_id"] }).count brand_actions.update( { :_id => brand_action["_id"] }, { "$set" => { :blog_posts_count => blog_posts_count} }) end end
26. MIGRATION This requires downtime while the migration is running
27. LAZY MIGRATION class LongAction < BrandAction field :blog_posts_count, :type => Integer, :default => 0 def increment_blog_posts_count if self.blog_posts_count.nil? initialize_blog_posts_count end inc(:blog_posts_count, 1) blog_posts_count end def initialize_blog_posts_count blog_posts_count = blog_posts.count self.update_attributes!( { :blog_posts_count => blog_posts_count }) end end
28. LAZY MIGRATION – RACE CONDITION! class LongAction < BrandAction field :blog_posts_count, :type => Integer, :default => 0 def increment_blog_posts_count if self.blog_posts_count.nil? initialize_blog_posts_count end inc(:blog_posts_count, 1) blog_posts_count end def initialize_blog_posts_count blog_posts_count = blog_posts.count self.update_attributes!( { :blog_posts_count => blog_posts_count }) end end
29. LAZY MIGRATION – RACE CONDITION! Not a problem when: –  Low volume updates OR –  Slight data inconsistency isn’t an issue Otherwise you have to lock
30. LAZY MIGRATION – OPTIMISTIC LOCK class LongAction < BrandAction field :blog_posts_count, :type => Integer def increment_blog_posts_count if self.blog_posts_count.nil? initialize_blog_posts_count else inc(:blog_posts_count, 1) end blog_posts_count end def initialize_blog_posts_count until count == new_count count = blog_posts.count new_count = collection.find_and_modify( query: {"_id" => self.id}, update: {"blog_posts_count" => {"$set" => count}}, new: true)["blog_posts_count"] end end end
31. LAZY MIGRATION – OPTIMISTIC LOCK Will never return with high volume updates So we need to exclusively lock. https://github.com/crowdtap/redis-lock
32. LAZY MIGRATION – EXCLUSIVE LOCK class LongAction < BrandAction field :blog_posts_count, :type => Integer def increment_blog_posts_count if blog_posts_count.nil? RedisLock.new($redis, "blog_post_count").lock_for_update do initialize_blog_posts_count end else inc(:blog_posts_count, 1) end blog_posts_count end def initialize_blog_posts_count count = blog_posts_count.count collection.update( {"_id" => self.id}, {"blog_posts_count" => {"$set" => count}}) end end
33. LAZY MIGRATION – EXCLUSIVE LOCK Redis lock uses SETNX Can be done using Mongo’s powerful findAndModify
34. IN CONCLUSION •  Mongo works with Rails seamlessly, it’s dynamic nature is an ideal match to Ruby. Inheritance and mixins can encapsulate state as well as behavior. •  It avoids downtime by allowing for dynamic schema updates •  But you still need to write migrations
35. OH YA, WE’RE HIRING Email me kareemk@crowdtap.com Tweet me @kareemk We’re making bank Side projects encouraged Equal seat at the product table Open source maintainers Strong mentorship and collaborative culture B2B & B2C == Interesting problems to solve Unlimited vacation