No content

HI. MY NAME IS KAREEM KOUDDOUS. I’M CURRENTLY HEAD OF TECHNOLOGY AT CROWDTAP AND A FOUNDING TEAM MEMBER. I LOVE HACKING ON SIDE PROJECTS AND TAKING LONG WALKS IN MY NEIGHBORHOOD. 

AND I’M HERE TO TALK ABOUT EXPLOITING MONGO’S SCHEMALESS NATURE IN RAILS 

WTF IS CROWDTAP? Crowdtap  allows  brands  to   collaborate  with  their  most   passionate  consumers,  ac5va5ng   them  as  peer  influencers  for  scalable   word-­‐of-­‐mouth  marke5ng  both  on   and  offline.   -­‐  Official   

HUMBLEBRAGGING Over 200,000 members Working with 5 of the top 10 marketers 30 million awards earned by members 70k LOC 1:2.8 Code-to-Test ratio Deploy on average 5 times a day 

WHY MONGO Schemaless nature ideal for the inheritance hierarchy in our domain models The possibility of auto-sharding with no application level impact which forces a scalable schema design 10gen team is awesome and right around the corner from us 

RAILS ODMS We chose Mongoid over MongoMapper because of the lazy evaluation of criteria and use of a cursor instead of loading the entire result set. We were also able to meet with Durran (the author) and hear about his plans and level of commitment 

A MODEL IN MONGOID class BrandAction include Mongoid::Document include Mongoid::Timestamps include Mixins::Sluggable field :title field :state field :category field :targeted_at, :type => DateTime field :launched_at, :type => DateTime field :closed_at, :type => DateTime embeds_one :member_filter embeds_many :contribution_summaries referenced_in :client references_many :activity_feed_items index [['created_at', Mongo::ASCENDING]] end 

A MODEL IN MONGOID class BrandAction include Mongoid::Document include Mongoid::Timestamps include Mixins::Sluggable field :title field :state field :category field :targeted_at, :type => DateTime field :launched_at, :type => DateTime field :closed_at, :type => DateTime embeds_one :member_filter embeds_many :contribution_summaries referenced_in :client references_many :activity_feed_items index [['created_at', Mongo::ASCENDING]] end 

A MODEL IN MONGOID class BrandAction include Mongoid::Document include Mongoid::Timestamps include Mixins::Sluggable field :title field :state field :category field :targeted_at, :type => DateTime field :launched_at, :type => DateTime field :closed_at, :type => DateTime embeds_one :member_filter embeds_many :contribution_summaries referenced_in :client references_many :activity_feed_items index [['created_at', Mongo::ASCENDING]] end 

A MODEL IN MONGOID class BrandAction include Mongoid::Document include Mongoid::Timestamps include Mixins::Sluggable field :title field :state field :category field :targeted_at, :type => DateTime field :launched_at, :type => DateTime field :closed_at, :type => DateTime embeds_one :member_filter embeds_many :contribution_summaries referenced_in :client references_many :activity_feed_items index [['created_at', Mongo::ASCENDING]] end 

INHERITANCE class LongAction < BrandAction field :moderation_queue, :type => Array, :default => [] field :approved_member_ids, :type => Array, :default => [] field :rejected_member_ids, :type => Array, :default => [] def should_create_feed_item? false end end class ModeratedDiscussion < LongAction field :allow_member_media, :type => Boolean, :default => false field :allow_member_links, :type => Boolean, :default => false def should_create_feed_item? allow_member_links && allow_member_media end end 

INHERITANCE >> BrandAction.where(:created_at => {$gt => 3.days.ago).count 15 >> LongAction. where(:created_at => {$gt => 3.days.ago).count 10 >> ModeratedDiscussion.where(:created_at => {$gt => 3.days.ago).count 4 

LESSONS LEARNED •  Inheritance with Mongo + Mongoid + Rails is so simple & powerful •  Only cost: Startup time increases as all models need to preloaded 

MIXINS class BrandAction include Mongoid::Document include Mongoid::Timestamps include Mixins::Sluggable field :title field :state field :category field :targeted_at, :type => DateTime field :launched_at, :type => DateTime field :closed_at, :type => DateTime embeds_one :member_filter embeds_many :contribution_summaries referenced_in :client references_many :activity_feed_items index [['created_at', Mongo::ASCENDING]] end 

MIXINS >> BrandAction.last.created_at => 2012-04-19 16:52:26 -0400 >> BrandAction.last.updated_at => 2012-04-19 17:02:47 -0400 

MIXINS class BrandAction include Mongoid::Document include Mongoid::Timestamps include Mixins::Sluggable field :title field :state field :category field :targeted_at, :type => DateTime field :launched_at, :type => DateTime field :closed_at, :type => DateTime embeds_one :member_filter embeds_many :contribution_summaries referenced_in :client references_many :activity_feed_items index [['created_at', Mongo::ASCENDING]] end 

MIXINS module Mixins::Sluggable extend ActiveSupport::Concern included do field :slug private :slug= before_create :generate_slug validates_uniqueness_of :slug, :on => :create class << self alias_method_chain :find, :slug end end end 

MIXINS def find_with_slug(*args) if args.size == 1 slug = args.first if slug.is_a?(Hash) && slug.has_key?("$oid") self.find_without_slug(*args) elsif result = where(:slug => slug).first result else if BSON::ObjectId.legal?(slug.to_s) || %w(first last all).include?(slug.to_s) || slug.is_a?(Array) self.find_without_slug(*args) else raise Mongoid::Errors::DocumentNotFound.new(self, slug) end end else self.find_without_slug(*args) end end 

MIXINS def generate_slug sequence = 0 generated_slug = slug_source.slice(0, TRUNCATION_LENGTH).slugize while(self.class.raw_collection.find(:slug => generated_slug).count > 0) do sequence += 1 truncated_slug_source = slug_source.slice(0, (TRUNCATION_LENGTH - sequence.to_s.length)) generated_slug = [truncated_slug_source, sequence].join(' ').slugize end self.slug = generated_slug end 

LESSONS LEARNED •  Mixins are a great way to encapsulate shared behavior and state •  Be careful when mixing in behavior that uses the where clause (it will only be scoped to the current model) 

MIGRATIONS You still need them! •  Schema changes (e.g. splitting “Name” into “First Name” and “Last Name”) •  Cached field initialization 

ADDING A COUNTER CACHE class LongAction < BrandAction field :blog_posts_count, :type => Integer, :default => 0 def increment_blog_posts_count inc(:blog_posts_count, 1) blog_posts_count end end class BlogFeedItem < FeedItem after_create :increment_blog_posts_count_on_long_action private def increment_blog_posts_count_on_long_action brand_action.increment_blog_posts_count end end 

ADDING A COUNTER CACHE You need to write a migration or a lazy migration. 

MIGRATION brand_actions.find({ :_type => "LongAction" } }, { :timeout => false }) do |cursor| cursor.each do |brand_action| blog_posts_count = db.feed_items.find( {:_type => "BlogFeedItem", :brand_action_id => brand_action["_id"] }).count brand_actions.update( { :_id => brand_action["_id"] }, { "$set" => { :blog_posts_count => blog_posts_count} }) end end 

MIGRATION This requires downtime while the migration is running 

LAZY MIGRATION class LongAction < BrandAction field :blog_posts_count, :type => Integer, :default => 0 def increment_blog_posts_count if self.blog_posts_count.nil? initialize_blog_posts_count end inc(:blog_posts_count, 1) blog_posts_count end def initialize_blog_posts_count blog_posts_count = blog_posts.count self.update_attributes!( { :blog_posts_count => blog_posts_count }) end end 

LAZY MIGRATION – RACE CONDITION! class LongAction < BrandAction field :blog_posts_count, :type => Integer, :default => 0 def increment_blog_posts_count if self.blog_posts_count.nil? initialize_blog_posts_count end inc(:blog_posts_count, 1) blog_posts_count end def initialize_blog_posts_count blog_posts_count = blog_posts.count self.update_attributes!( { :blog_posts_count => blog_posts_count }) end end 

LAZY MIGRATION – RACE CONDITION! Not a problem when: –  Low volume updates OR –  Slight data inconsistency isn’t an issue Otherwise you have to lock 

LAZY MIGRATION – OPTIMISTIC LOCK class LongAction < BrandAction field :blog_posts_count, :type => Integer def increment_blog_posts_count if self.blog_posts_count.nil? initialize_blog_posts_count else inc(:blog_posts_count, 1) end blog_posts_count end def initialize_blog_posts_count until count == new_count count = blog_posts.count new_count = collection.find_and_modify( query: {"_id" => self.id}, update: {"blog_posts_count" => {"$set" => count}}, new: true)["blog_posts_count"] end end end 

LAZY MIGRATION – OPTIMISTIC LOCK Will never return with high volume updates So we need to exclusively lock. https://github.com/crowdtap/redis-lock 

LAZY MIGRATION – EXCLUSIVE LOCK class LongAction < BrandAction field :blog_posts_count, :type => Integer def increment_blog_posts_count if blog_posts_count.nil? RedisLock.new($redis, "blog_post_count").lock_for_update do initialize_blog_posts_count end else inc(:blog_posts_count, 1) end blog_posts_count end def initialize_blog_posts_count count = blog_posts_count.count collection.update( {"_id" => self.id}, {"blog_posts_count" => {"$set" => count}}) end end 

LAZY MIGRATION – EXCLUSIVE LOCK Redis lock uses SETNX Can be done using Mongo’s powerful findAndModify 

IN CONCLUSION •  Mongo works with Rails seamlessly, it’s dynamic nature is an ideal match to Ruby. Inheritance and mixins can encapsulate state as well as behavior. •  It avoids downtime by allowing for dynamic schema updates •  But you still need to write migrations 

OH YA, WE’RE HIRING Email me [email protected] Tweet me @kareemk We’re making bank Side projects encouraged Equal seat at the product table Open source maintainers Strong mentorship and collaborative culture B2B & B2C == Interesting problems to solve Unlimited vacation