Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

HI. MY NAME IS KAREEM KOUDDOUS. I’M CURRENTLY HEAD OF TECHNOLOGY AT CROWDTAP AND A FOUNDING TEAM MEMBER. I LOVE HACKING ON SIDE PROJECTS AND TAKING LONG WALKS IN MY NEIGHBORHOOD.

Slide 3

Slide 3 text

AND I’M HERE TO TALK ABOUT EXPLOITING MONGO’S SCHEMALESS NATURE IN RAILS

Slide 4

Slide 4 text

WTF IS CROWDTAP? Crowdtap  allows  brands  to   collaborate  with  their  most   passionate  consumers,  ac5va5ng   them  as  peer  influencers  for  scalable   word-­‐of-­‐mouth  marke5ng  both  on   and  offline.   -­‐  Official  

Slide 5

Slide 5 text

HUMBLEBRAGGING Over 200,000 members Working with 5 of the top 10 marketers 30 million awards earned by members 70k LOC 1:2.8 Code-to-Test ratio Deploy on average 5 times a day

Slide 6

Slide 6 text

WHY MONGO Schemaless nature ideal for the inheritance hierarchy in our domain models The possibility of auto-sharding with no application level impact which forces a scalable schema design 10gen team is awesome and right around the corner from us

Slide 7

Slide 7 text

RAILS ODMS We chose Mongoid over MongoMapper because of the lazy evaluation of criteria and use of a cursor instead of loading the entire result set. We were also able to meet with Durran (the author) and hear about his plans and level of commitment

Slide 8

Slide 8 text

A MODEL IN MONGOID class BrandAction include Mongoid::Document include Mongoid::Timestamps include Mixins::Sluggable field :title field :state field :category field :targeted_at, :type => DateTime field :launched_at, :type => DateTime field :closed_at, :type => DateTime embeds_one :member_filter embeds_many :contribution_summaries referenced_in :client references_many :activity_feed_items index [['created_at', Mongo::ASCENDING]] end

Slide 9

Slide 9 text

A MODEL IN MONGOID class BrandAction include Mongoid::Document include Mongoid::Timestamps include Mixins::Sluggable field :title field :state field :category field :targeted_at, :type => DateTime field :launched_at, :type => DateTime field :closed_at, :type => DateTime embeds_one :member_filter embeds_many :contribution_summaries referenced_in :client references_many :activity_feed_items index [['created_at', Mongo::ASCENDING]] end

Slide 10

Slide 10 text

A MODEL IN MONGOID class BrandAction include Mongoid::Document include Mongoid::Timestamps include Mixins::Sluggable field :title field :state field :category field :targeted_at, :type => DateTime field :launched_at, :type => DateTime field :closed_at, :type => DateTime embeds_one :member_filter embeds_many :contribution_summaries referenced_in :client references_many :activity_feed_items index [['created_at', Mongo::ASCENDING]] end

Slide 11

Slide 11 text

A MODEL IN MONGOID class BrandAction include Mongoid::Document include Mongoid::Timestamps include Mixins::Sluggable field :title field :state field :category field :targeted_at, :type => DateTime field :launched_at, :type => DateTime field :closed_at, :type => DateTime embeds_one :member_filter embeds_many :contribution_summaries referenced_in :client references_many :activity_feed_items index [['created_at', Mongo::ASCENDING]] end

Slide 12

Slide 12 text

INHERITANCE class LongAction < BrandAction field :moderation_queue, :type => Array, :default => [] field :approved_member_ids, :type => Array, :default => [] field :rejected_member_ids, :type => Array, :default => [] def should_create_feed_item? false end end class ModeratedDiscussion < LongAction field :allow_member_media, :type => Boolean, :default => false field :allow_member_links, :type => Boolean, :default => false def should_create_feed_item? allow_member_links && allow_member_media end end

Slide 13

Slide 13 text

INHERITANCE >> BrandAction.where(:created_at => {$gt => 3.days.ago).count 15 >> LongAction. where(:created_at => {$gt => 3.days.ago).count 10 >> ModeratedDiscussion.where(:created_at => {$gt => 3.days.ago).count 4

Slide 14

Slide 14 text

LESSONS LEARNED •  Inheritance with Mongo + Mongoid + Rails is so simple & powerful •  Only cost: Startup time increases as all models need to preloaded

Slide 15

Slide 15 text

MIXINS class BrandAction include Mongoid::Document include Mongoid::Timestamps include Mixins::Sluggable field :title field :state field :category field :targeted_at, :type => DateTime field :launched_at, :type => DateTime field :closed_at, :type => DateTime embeds_one :member_filter embeds_many :contribution_summaries referenced_in :client references_many :activity_feed_items index [['created_at', Mongo::ASCENDING]] end

Slide 16

Slide 16 text

MIXINS >> BrandAction.last.created_at => 2012-04-19 16:52:26 -0400 >> BrandAction.last.updated_at => 2012-04-19 17:02:47 -0400

Slide 17

Slide 17 text

MIXINS class BrandAction include Mongoid::Document include Mongoid::Timestamps include Mixins::Sluggable field :title field :state field :category field :targeted_at, :type => DateTime field :launched_at, :type => DateTime field :closed_at, :type => DateTime embeds_one :member_filter embeds_many :contribution_summaries referenced_in :client references_many :activity_feed_items index [['created_at', Mongo::ASCENDING]] end

Slide 18

Slide 18 text

MIXINS module Mixins::Sluggable extend ActiveSupport::Concern included do field :slug private :slug= before_create :generate_slug validates_uniqueness_of :slug, :on => :create class << self alias_method_chain :find, :slug end end end

Slide 19

Slide 19 text

MIXINS def find_with_slug(*args) if args.size == 1 slug = args.first if slug.is_a?(Hash) && slug.has_key?("$oid") self.find_without_slug(*args) elsif result = where(:slug => slug).first result else if BSON::ObjectId.legal?(slug.to_s) || %w(first last all).include?(slug.to_s) || slug.is_a?(Array) self.find_without_slug(*args) else raise Mongoid::Errors::DocumentNotFound.new(self, slug) end end else self.find_without_slug(*args) end end

Slide 20

Slide 20 text

MIXINS def generate_slug sequence = 0 generated_slug = slug_source.slice(0, TRUNCATION_LENGTH).slugize while(self.class.raw_collection.find(:slug => generated_slug).count > 0) do sequence += 1 truncated_slug_source = slug_source.slice(0, (TRUNCATION_LENGTH - sequence.to_s.length)) generated_slug = [truncated_slug_source, sequence].join(' ').slugize end self.slug = generated_slug end

Slide 21

Slide 21 text

LESSONS LEARNED •  Mixins are a great way to encapsulate shared behavior and state •  Be careful when mixing in behavior that uses the where clause (it will only be scoped to the current model)

Slide 22

Slide 22 text

MIGRATIONS You still need them! •  Schema changes (e.g. splitting “Name” into “First Name” and “Last Name”) •  Cached field initialization

Slide 23

Slide 23 text

ADDING A COUNTER CACHE class LongAction < BrandAction field :blog_posts_count, :type => Integer, :default => 0 def increment_blog_posts_count inc(:blog_posts_count, 1) blog_posts_count end end class BlogFeedItem < FeedItem after_create :increment_blog_posts_count_on_long_action private def increment_blog_posts_count_on_long_action brand_action.increment_blog_posts_count end end

Slide 24

Slide 24 text

ADDING A COUNTER CACHE You need to write a migration or a lazy migration.

Slide 25

Slide 25 text

MIGRATION brand_actions.find({ :_type => "LongAction" } }, { :timeout => false }) do |cursor| cursor.each do |brand_action| blog_posts_count = db.feed_items.find( {:_type => "BlogFeedItem", :brand_action_id => brand_action["_id"] }).count brand_actions.update( { :_id => brand_action["_id"] }, { "$set" => { :blog_posts_count => blog_posts_count} }) end end

Slide 26

Slide 26 text

MIGRATION This requires downtime while the migration is running

Slide 27

Slide 27 text

LAZY MIGRATION class LongAction < BrandAction field :blog_posts_count, :type => Integer, :default => 0 def increment_blog_posts_count if self.blog_posts_count.nil? initialize_blog_posts_count end inc(:blog_posts_count, 1) blog_posts_count end def initialize_blog_posts_count blog_posts_count = blog_posts.count self.update_attributes!( { :blog_posts_count => blog_posts_count }) end end

Slide 28

Slide 28 text

LAZY MIGRATION – RACE CONDITION! class LongAction < BrandAction field :blog_posts_count, :type => Integer, :default => 0 def increment_blog_posts_count if self.blog_posts_count.nil? initialize_blog_posts_count end inc(:blog_posts_count, 1) blog_posts_count end def initialize_blog_posts_count blog_posts_count = blog_posts.count self.update_attributes!( { :blog_posts_count => blog_posts_count }) end end

Slide 29

Slide 29 text

LAZY MIGRATION – RACE CONDITION! Not a problem when: –  Low volume updates OR –  Slight data inconsistency isn’t an issue Otherwise you have to lock

Slide 30

Slide 30 text

LAZY MIGRATION – OPTIMISTIC LOCK class LongAction < BrandAction field :blog_posts_count, :type => Integer def increment_blog_posts_count if self.blog_posts_count.nil? initialize_blog_posts_count else inc(:blog_posts_count, 1) end blog_posts_count end def initialize_blog_posts_count until count == new_count count = blog_posts.count new_count = collection.find_and_modify( query: {"_id" => self.id}, update: {"blog_posts_count" => {"$set" => count}}, new: true)["blog_posts_count"] end end end

Slide 31

Slide 31 text

LAZY MIGRATION – OPTIMISTIC LOCK Will never return with high volume updates So we need to exclusively lock. https://github.com/crowdtap/redis-lock

Slide 32

Slide 32 text

LAZY MIGRATION – EXCLUSIVE LOCK class LongAction < BrandAction field :blog_posts_count, :type => Integer def increment_blog_posts_count if blog_posts_count.nil? RedisLock.new($redis, "blog_post_count").lock_for_update do initialize_blog_posts_count end else inc(:blog_posts_count, 1) end blog_posts_count end def initialize_blog_posts_count count = blog_posts_count.count collection.update( {"_id" => self.id}, {"blog_posts_count" => {"$set" => count}}) end end

Slide 33

Slide 33 text

LAZY MIGRATION – EXCLUSIVE LOCK Redis lock uses SETNX Can be done using Mongo’s powerful findAndModify

Slide 34

Slide 34 text

IN CONCLUSION •  Mongo works with Rails seamlessly, it’s dynamic nature is an ideal match to Ruby. Inheritance and mixins can encapsulate state as well as behavior. •  It avoids downtime by allowing for dynamic schema updates •  But you still need to write migrations

Slide 35

Slide 35 text

OH YA, WE’RE HIRING Email me [email protected] Tweet me @kareemk We’re making bank Side projects encouraged Equal seat at the product table Open source maintainers Strong mentorship and collaborative culture B2B & B2C == Interesting problems to solve Unlimited vacation