Share
Embed
MongoNYC 2012: Exploiting Mongo's Schemaless Nature in Rails
MongoNYC 2012: Exploiting Mongo's Schemaless Nature in Rails, Kareem Kouddos, Crowdtap. One of the most awesome benefits of Mongo is that its schemaless. This means you can do some really interesting things in your models such as dynamically adding attributes to the model without requiring a migration. The simplest example is adding createdat or updatedat timestamps. We'll go over some cool examples and discuss best practices. We'll cover best practices to avoid downtime in production while migrating to a different data representation.
mongodb
May 29, 2012
Transcript
35 slides
-
HI. MY NAME IS KAREEM KOUDDOUS. I’M CURRENTLY HEAD OF
TECHNOLOGY AT CROWDTAP AND A FOUNDING TEAM MEMBER. I LOVE HACKING ON SIDE PROJECTS AND TAKING LONG WALKS IN MY NEIGHBORHOOD. -
WTF IS CROWDTAP? Crowdtap allows brands to collaborate with
their most passionate consumers, ac5va5ng them as peer influencers for scalable word-‐of-‐mouth marke5ng both on and offline. -‐ Official -
HUMBLEBRAGGING Over 200,000 members Working with 5 of the top
10 marketers 30 million awards earned by members 70k LOC 1:2.8 Code-to-Test ratio Deploy on average 5 times a day -
WHY MONGO Schemaless nature ideal for the inheritance hierarchy in
our domain models The possibility of auto-sharding with no application level impact which forces a scalable schema design 10gen team is awesome and right around the corner from us -
RAILS ODMS We chose Mongoid over MongoMapper because of the
lazy evaluation of criteria and use of a cursor instead of loading the entire result set. We were also able to meet with Durran (the author) and hear about his plans and level of commitment -
A MODEL IN MONGOID class BrandAction include Mongoid::Document include Mongoid::Timestamps
include Mixins::Sluggable field :title field :state field :category field :targeted_at, :type => DateTime field :launched_at, :type => DateTime field :closed_at, :type => DateTime embeds_one :member_filter embeds_many :contribution_summaries referenced_in :client references_many :activity_feed_items index [['created_at', Mongo::ASCENDING]] end -
A MODEL IN MONGOID class BrandAction include Mongoid::Document include Mongoid::Timestamps
include Mixins::Sluggable field :title field :state field :category field :targeted_at, :type => DateTime field :launched_at, :type => DateTime field :closed_at, :type => DateTime embeds_one :member_filter embeds_many :contribution_summaries referenced_in :client references_many :activity_feed_items index [['created_at', Mongo::ASCENDING]] end -
A MODEL IN MONGOID class BrandAction include Mongoid::Document include Mongoid::Timestamps
include Mixins::Sluggable field :title field :state field :category field :targeted_at, :type => DateTime field :launched_at, :type => DateTime field :closed_at, :type => DateTime embeds_one :member_filter embeds_many :contribution_summaries referenced_in :client references_many :activity_feed_items index [['created_at', Mongo::ASCENDING]] end -
A MODEL IN MONGOID class BrandAction include Mongoid::Document include Mongoid::Timestamps
include Mixins::Sluggable field :title field :state field :category field :targeted_at, :type => DateTime field :launched_at, :type => DateTime field :closed_at, :type => DateTime embeds_one :member_filter embeds_many :contribution_summaries referenced_in :client references_many :activity_feed_items index [['created_at', Mongo::ASCENDING]] end -
INHERITANCE class LongAction < BrandAction field :moderation_queue, :type => Array,
:default => [] field :approved_member_ids, :type => Array, :default => [] field :rejected_member_ids, :type => Array, :default => [] def should_create_feed_item? false end end class ModeratedDiscussion < LongAction field :allow_member_media, :type => Boolean, :default => false field :allow_member_links, :type => Boolean, :default => false def should_create_feed_item? allow_member_links && allow_member_media end end -
INHERITANCE >> BrandAction.where(:created_at => {$gt => 3.days.ago).count 15 >> LongAction.
where(:created_at => {$gt => 3.days.ago).count 10 >> ModeratedDiscussion.where(:created_at => {$gt => 3.days.ago).count 4 -
LESSONS LEARNED • Inheritance with Mongo + Mongoid + Rails
is so simple & powerful • Only cost: Startup time increases as all models need to preloaded -
MIXINS class BrandAction include Mongoid::Document include Mongoid::Timestamps include Mixins::Sluggable field
:title field :state field :category field :targeted_at, :type => DateTime field :launched_at, :type => DateTime field :closed_at, :type => DateTime embeds_one :member_filter embeds_many :contribution_summaries referenced_in :client references_many :activity_feed_items index [['created_at', Mongo::ASCENDING]] end -
MIXINS >> BrandAction.last.created_at => 2012-04-19 16:52:26 -0400 >> BrandAction.last.updated_at =>
2012-04-19 17:02:47 -0400 -
MIXINS class BrandAction include Mongoid::Document include Mongoid::Timestamps include Mixins::Sluggable field
:title field :state field :category field :targeted_at, :type => DateTime field :launched_at, :type => DateTime field :closed_at, :type => DateTime embeds_one :member_filter embeds_many :contribution_summaries referenced_in :client references_many :activity_feed_items index [['created_at', Mongo::ASCENDING]] end -
MIXINS module Mixins::Sluggable extend ActiveSupport::Concern included do field :slug private
:slug= before_create :generate_slug validates_uniqueness_of :slug, :on => :create class << self alias_method_chain :find, :slug end end end -
MIXINS def find_with_slug(*args) if args.size == 1 slug = args.first
if slug.is_a?(Hash) && slug.has_key?("$oid") self.find_without_slug(*args) elsif result = where(:slug => slug).first result else if BSON::ObjectId.legal?(slug.to_s) || %w(first last all).include?(slug.to_s) || slug.is_a?(Array) self.find_without_slug(*args) else raise Mongoid::Errors::DocumentNotFound.new(self, slug) end end else self.find_without_slug(*args) end end -
MIXINS def generate_slug sequence = 0 generated_slug = slug_source.slice(0, TRUNCATION_LENGTH).slugize
while(self.class.raw_collection.find(:slug => generated_slug).count > 0) do sequence += 1 truncated_slug_source = slug_source.slice(0, (TRUNCATION_LENGTH - sequence.to_s.length)) generated_slug = [truncated_slug_source, sequence].join(' ').slugize end self.slug = generated_slug end -
LESSONS LEARNED • Mixins are a great way to encapsulate
shared behavior and state • Be careful when mixing in behavior that uses the where clause (it will only be scoped to the current model) -
MIGRATIONS You still need them! • Schema changes (e.g. splitting
“Name” into “First Name” and “Last Name”) • Cached field initialization -
ADDING A COUNTER CACHE class LongAction < BrandAction field :blog_posts_count,
:type => Integer, :default => 0 def increment_blog_posts_count inc(:blog_posts_count, 1) blog_posts_count end end class BlogFeedItem < FeedItem after_create :increment_blog_posts_count_on_long_action private def increment_blog_posts_count_on_long_action brand_action.increment_blog_posts_count end end -
ADDING A COUNTER CACHE You need to write a migration
or a lazy migration. -
MIGRATION brand_actions.find({ :_type => "LongAction" } }, { :timeout =>
false }) do |cursor| cursor.each do |brand_action| blog_posts_count = db.feed_items.find( {:_type => "BlogFeedItem", :brand_action_id => brand_action["_id"] }).count brand_actions.update( { :_id => brand_action["_id"] }, { "$set" => { :blog_posts_count => blog_posts_count} }) end end -
LAZY MIGRATION class LongAction < BrandAction field :blog_posts_count, :type =>
Integer, :default => 0 def increment_blog_posts_count if self.blog_posts_count.nil? initialize_blog_posts_count end inc(:blog_posts_count, 1) blog_posts_count end def initialize_blog_posts_count blog_posts_count = blog_posts.count self.update_attributes!( { :blog_posts_count => blog_posts_count }) end end -
LAZY MIGRATION – RACE CONDITION! class LongAction < BrandAction field
:blog_posts_count, :type => Integer, :default => 0 def increment_blog_posts_count if self.blog_posts_count.nil? initialize_blog_posts_count end inc(:blog_posts_count, 1) blog_posts_count end def initialize_blog_posts_count blog_posts_count = blog_posts.count self.update_attributes!( { :blog_posts_count => blog_posts_count }) end end -
LAZY MIGRATION – RACE CONDITION! Not a problem when: –
Low volume updates OR – Slight data inconsistency isn’t an issue Otherwise you have to lock -
LAZY MIGRATION – OPTIMISTIC LOCK class LongAction < BrandAction field
:blog_posts_count, :type => Integer def increment_blog_posts_count if self.blog_posts_count.nil? initialize_blog_posts_count else inc(:blog_posts_count, 1) end blog_posts_count end def initialize_blog_posts_count until count == new_count count = blog_posts.count new_count = collection.find_and_modify( query: {"_id" => self.id}, update: {"blog_posts_count" => {"$set" => count}}, new: true)["blog_posts_count"] end end end -
LAZY MIGRATION – OPTIMISTIC LOCK Will never return with high
volume updates So we need to exclusively lock. https://github.com/crowdtap/redis-lock -
LAZY MIGRATION – EXCLUSIVE LOCK class LongAction < BrandAction field
:blog_posts_count, :type => Integer def increment_blog_posts_count if blog_posts_count.nil? RedisLock.new($redis, "blog_post_count").lock_for_update do initialize_blog_posts_count end else inc(:blog_posts_count, 1) end blog_posts_count end def initialize_blog_posts_count count = blog_posts_count.count collection.update( {"_id" => self.id}, {"blog_posts_count" => {"$set" => count}}) end end -
LAZY MIGRATION – EXCLUSIVE LOCK Redis lock uses SETNX Can
be done using Mongo’s powerful findAndModify -
IN CONCLUSION • Mongo works with Rails seamlessly, it’s dynamic
nature is an ideal match to Ruby. Inheritance and mixins can encapsulate state as well as behavior. • It avoids downtime by allowing for dynamic schema updates • But you still need to write migrations -
OH YA, WE’RE HIRING Email me [email protected] Tweet me @kareemk
We’re making bank Side projects encouraged Equal seat at the product table Open source maintainers Strong mentorship and collaborative culture B2B & B2C == Interesting problems to solve Unlimited vacation