Upgrade to Pro — share decks privately, control downloads, hide ads and more …

MongoNYC 2012: Exploiting Mongo's Schemaless Nature in Rails

mongodb
May 29, 2012
530

MongoNYC 2012: Exploiting Mongo's Schemaless Nature in Rails

MongoNYC 2012: Exploiting Mongo's Schemaless Nature in Rails, Kareem Kouddos, Crowdtap. One of the most awesome benefits of Mongo is that its schemaless. This means you can do some really interesting things in your models such as dynamically adding attributes to the model without requiring a migration. The simplest example is adding createdat or updatedat timestamps. We'll go over some cool examples and discuss best practices. We'll cover best practices to avoid downtime in production while migrating to a different data representation.

mongodb

May 29, 2012
Tweet

Transcript

  1. HI. MY NAME IS KAREEM KOUDDOUS. I’M CURRENTLY HEAD OF

    TECHNOLOGY AT CROWDTAP AND A FOUNDING TEAM MEMBER. I LOVE HACKING ON SIDE PROJECTS AND TAKING LONG WALKS IN MY NEIGHBORHOOD.
  2. WTF IS CROWDTAP? Crowdtap  allows  brands  to   collaborate  with

     their  most   passionate  consumers,  ac5va5ng   them  as  peer  influencers  for  scalable   word-­‐of-­‐mouth  marke5ng  both  on   and  offline.   -­‐  Official  
  3. HUMBLEBRAGGING Over 200,000 members Working with 5 of the top

    10 marketers 30 million awards earned by members 70k LOC 1:2.8 Code-to-Test ratio Deploy on average 5 times a day
  4. WHY MONGO Schemaless nature ideal for the inheritance hierarchy in

    our domain models The possibility of auto-sharding with no application level impact which forces a scalable schema design 10gen team is awesome and right around the corner from us
  5. RAILS ODMS We chose Mongoid over MongoMapper because of the

    lazy evaluation of criteria and use of a cursor instead of loading the entire result set. We were also able to meet with Durran (the author) and hear about his plans and level of commitment
  6. A MODEL IN MONGOID class BrandAction include Mongoid::Document include Mongoid::Timestamps

    include Mixins::Sluggable field :title field :state field :category field :targeted_at, :type => DateTime field :launched_at, :type => DateTime field :closed_at, :type => DateTime embeds_one :member_filter embeds_many :contribution_summaries referenced_in :client references_many :activity_feed_items index [['created_at', Mongo::ASCENDING]] end
  7. A MODEL IN MONGOID class BrandAction include Mongoid::Document include Mongoid::Timestamps

    include Mixins::Sluggable field :title field :state field :category field :targeted_at, :type => DateTime field :launched_at, :type => DateTime field :closed_at, :type => DateTime embeds_one :member_filter embeds_many :contribution_summaries referenced_in :client references_many :activity_feed_items index [['created_at', Mongo::ASCENDING]] end
  8. A MODEL IN MONGOID class BrandAction include Mongoid::Document include Mongoid::Timestamps

    include Mixins::Sluggable field :title field :state field :category field :targeted_at, :type => DateTime field :launched_at, :type => DateTime field :closed_at, :type => DateTime embeds_one :member_filter embeds_many :contribution_summaries referenced_in :client references_many :activity_feed_items index [['created_at', Mongo::ASCENDING]] end
  9. A MODEL IN MONGOID class BrandAction include Mongoid::Document include Mongoid::Timestamps

    include Mixins::Sluggable field :title field :state field :category field :targeted_at, :type => DateTime field :launched_at, :type => DateTime field :closed_at, :type => DateTime embeds_one :member_filter embeds_many :contribution_summaries referenced_in :client references_many :activity_feed_items index [['created_at', Mongo::ASCENDING]] end
  10. INHERITANCE class LongAction < BrandAction field :moderation_queue, :type => Array,

    :default => [] field :approved_member_ids, :type => Array, :default => [] field :rejected_member_ids, :type => Array, :default => [] def should_create_feed_item? false end end class ModeratedDiscussion < LongAction field :allow_member_media, :type => Boolean, :default => false field :allow_member_links, :type => Boolean, :default => false def should_create_feed_item? allow_member_links && allow_member_media end end
  11. INHERITANCE >> BrandAction.where(:created_at => {$gt => 3.days.ago).count 15 >> LongAction.

    where(:created_at => {$gt => 3.days.ago).count 10 >> ModeratedDiscussion.where(:created_at => {$gt => 3.days.ago).count 4
  12. LESSONS LEARNED •  Inheritance with Mongo + Mongoid + Rails

    is so simple & powerful •  Only cost: Startup time increases as all models need to preloaded
  13. MIXINS class BrandAction include Mongoid::Document include Mongoid::Timestamps include Mixins::Sluggable field

    :title field :state field :category field :targeted_at, :type => DateTime field :launched_at, :type => DateTime field :closed_at, :type => DateTime embeds_one :member_filter embeds_many :contribution_summaries referenced_in :client references_many :activity_feed_items index [['created_at', Mongo::ASCENDING]] end
  14. MIXINS class BrandAction include Mongoid::Document include Mongoid::Timestamps include Mixins::Sluggable field

    :title field :state field :category field :targeted_at, :type => DateTime field :launched_at, :type => DateTime field :closed_at, :type => DateTime embeds_one :member_filter embeds_many :contribution_summaries referenced_in :client references_many :activity_feed_items index [['created_at', Mongo::ASCENDING]] end
  15. MIXINS module Mixins::Sluggable extend ActiveSupport::Concern included do field :slug private

    :slug= before_create :generate_slug validates_uniqueness_of :slug, :on => :create class << self alias_method_chain :find, :slug end end end
  16. MIXINS def find_with_slug(*args) if args.size == 1 slug = args.first

    if slug.is_a?(Hash) && slug.has_key?("$oid") self.find_without_slug(*args) elsif result = where(:slug => slug).first result else if BSON::ObjectId.legal?(slug.to_s) || %w(first last all).include?(slug.to_s) || slug.is_a?(Array) self.find_without_slug(*args) else raise Mongoid::Errors::DocumentNotFound.new(self, slug) end end else self.find_without_slug(*args) end end
  17. MIXINS def generate_slug sequence = 0 generated_slug = slug_source.slice(0, TRUNCATION_LENGTH).slugize

    while(self.class.raw_collection.find(:slug => generated_slug).count > 0) do sequence += 1 truncated_slug_source = slug_source.slice(0, (TRUNCATION_LENGTH - sequence.to_s.length)) generated_slug = [truncated_slug_source, sequence].join(' ').slugize end self.slug = generated_slug end
  18. LESSONS LEARNED •  Mixins are a great way to encapsulate

    shared behavior and state •  Be careful when mixing in behavior that uses the where clause (it will only be scoped to the current model)
  19. MIGRATIONS You still need them! •  Schema changes (e.g. splitting

    “Name” into “First Name” and “Last Name”) •  Cached field initialization
  20. ADDING A COUNTER CACHE class LongAction < BrandAction field :blog_posts_count,

    :type => Integer, :default => 0 def increment_blog_posts_count inc(:blog_posts_count, 1) blog_posts_count end end class BlogFeedItem < FeedItem after_create :increment_blog_posts_count_on_long_action private def increment_blog_posts_count_on_long_action brand_action.increment_blog_posts_count end end
  21. MIGRATION brand_actions.find({ :_type => "LongAction" } }, { :timeout =>

    false }) do |cursor| cursor.each do |brand_action| blog_posts_count = db.feed_items.find( {:_type => "BlogFeedItem", :brand_action_id => brand_action["_id"] }).count brand_actions.update( { :_id => brand_action["_id"] }, { "$set" => { :blog_posts_count => blog_posts_count} }) end end
  22. LAZY MIGRATION class LongAction < BrandAction field :blog_posts_count, :type =>

    Integer, :default => 0 def increment_blog_posts_count if self.blog_posts_count.nil? initialize_blog_posts_count end inc(:blog_posts_count, 1) blog_posts_count end def initialize_blog_posts_count blog_posts_count = blog_posts.count self.update_attributes!( { :blog_posts_count => blog_posts_count }) end end
  23. LAZY MIGRATION – RACE CONDITION! class LongAction < BrandAction field

    :blog_posts_count, :type => Integer, :default => 0 def increment_blog_posts_count if self.blog_posts_count.nil? initialize_blog_posts_count end inc(:blog_posts_count, 1) blog_posts_count end def initialize_blog_posts_count blog_posts_count = blog_posts.count self.update_attributes!( { :blog_posts_count => blog_posts_count }) end end
  24. LAZY MIGRATION – RACE CONDITION! Not a problem when: – 

    Low volume updates OR –  Slight data inconsistency isn’t an issue Otherwise you have to lock
  25. LAZY MIGRATION – OPTIMISTIC LOCK class LongAction < BrandAction field

    :blog_posts_count, :type => Integer def increment_blog_posts_count if self.blog_posts_count.nil? initialize_blog_posts_count else inc(:blog_posts_count, 1) end blog_posts_count end def initialize_blog_posts_count until count == new_count count = blog_posts.count new_count = collection.find_and_modify( query: {"_id" => self.id}, update: {"blog_posts_count" => {"$set" => count}}, new: true)["blog_posts_count"] end end end
  26. LAZY MIGRATION – OPTIMISTIC LOCK Will never return with high

    volume updates So we need to exclusively lock. https://github.com/crowdtap/redis-lock
  27. LAZY MIGRATION – EXCLUSIVE LOCK class LongAction < BrandAction field

    :blog_posts_count, :type => Integer def increment_blog_posts_count if blog_posts_count.nil? RedisLock.new($redis, "blog_post_count").lock_for_update do initialize_blog_posts_count end else inc(:blog_posts_count, 1) end blog_posts_count end def initialize_blog_posts_count count = blog_posts_count.count collection.update( {"_id" => self.id}, {"blog_posts_count" => {"$set" => count}}) end end
  28. LAZY MIGRATION – EXCLUSIVE LOCK Redis lock uses SETNX Can

    be done using Mongo’s powerful findAndModify
  29. IN CONCLUSION •  Mongo works with Rails seamlessly, it’s dynamic

    nature is an ideal match to Ruby. Inheritance and mixins can encapsulate state as well as behavior. •  It avoids downtime by allowing for dynamic schema updates •  But you still need to write migrations
  30. OH YA, WE’RE HIRING Email me [email protected] Tweet me @kareemk

    We’re making bank Side projects encouraged Equal seat at the product table Open source maintainers Strong mentorship and collaborative culture B2B & B2C == Interesting problems to solve Unlimited vacation