Benefits of using MongoDB: Reduce Complexity & Adapt to Changes

Transcript

1. Benefits of MongoDB: RedDotRubyConf 2011 Reduce Complexity & Adapt to Changes Vinova Pte Ltd
2. About me Alex Nguyen Co-founder at Vinova http://vinova.sg/ https://github.com/vinova/ • • • •
3. Agenda What’s MongoDB? Why MongoDB reduce complexity? Why MongoDB adapt to changes better? Case studies • • • •
4. I don’t hate SQL Just found a better tool for most of my use cases
5. What’s MongoDB? mongodb.org “MongoDB (from "humongous") is a scalable, high- performance, open source, document-oriented database”
6. What’s MongoDB? http://www.slideshare.net/kbanker/mongodb-schema-design-mongo-chicago
7. What’s MongoDB? Collections ~ Tables Documents ~ Rows • •
8. MongoDB Philosophy Reduce transactional semantics for performance No-relational is the best way to scale horizontally • • mongodb.org
9. MongoDB Features JSON style documents Index on any attribute Rich queries In-place update Auto-sharding • • • • • Map / Reduce GridFS to store files Server-side JavaScript Capped collections Full-text-search (coming soon) • • • • •
10. MongoDB's flexibility data structure, ability to index & query data, and auto-sharding make it a strong tool that adapt to changes well. It also help to reduce complexity comparing to tradition RDBMS.
11. Why MongoDB reduce complexity? Get rid of migrations Get rid of relationships (most of) Reduce number of database requests JSON (client, server, and database) • • • •
12. Get rid of migrations No create table No alter column No add column No change column • • • •
13. Get rid of relationships Many one-to-one and one-to-many relationships is not necessary User :has_one :setting User :has_many :addresses User :has_many :roles Post :has_many :tags • • • • •
14. Reduce number of database requests Pre-joined Rich queries Atomic, in-place updates • • •
15. JSON MongoDB knows JSON Don’t have to convert data from / to JSON • •
16. Adapt to changes Changes in schema Changes in data & algorithms Changes for performance & scaling • • •
17. Changes in schema In modern apps, schema changes quite often (weekly, monthly ...) Alter tables are expensive in RDBMS Dynamic schema document makes those changes seamlessly • • •
18. Changes in data & algorithms Atomic, in-place updates are very powerful to modify data $inc, $set, $unset, $push, $pop, $rename, $bit Rich queries and aggregators $in, $all, $exists, $size, $type, regexp count(), size(), distinct(), min(), max() Map/Reduce • • •
19. Changes for performance & scaling Very fast & ready to scale => Don’t have to use additional tools (memcached ...) Don’t have to change platforms • • •
20. Case Studies Store crawled info as embedded documents Product listing Find unique slug Voting • • • •
21. Store crawled info as embedded documents Data from 3rd party sources Sources and data formats can be changed in the future • •
22. Store crawled info as embedded documents product = { "_id" : ObjectId("4d8ace4b0dc3e43231bb930d"), "name" : "Product ABC", "amazon" : { "asin" : ..., "price" : ..., .... } };
23. Store crawled info as embedded documents product = { "_id" : ObjectId("4d8ace4b0dc3e43231bb930d"), "name" : "Product ABC", "amazon" : { "asin" : ..., "price" : ..., "shipping_cost" : ..., ... } };
24. Store crawled info as embedded documents product = { "_id" : ObjectId("4d8ace4b0dc3e43231bb930d"), "name" : "Product ABC", "amazon" : { "asin" : ..., "price" : ..., "shipping_cost" : ..., .... }, "walmart" : { "price" : ..., ... } };
25. Store crawled info as embedded documents def Product.find_by_asin(asin) Product.where('amazon.asin' => asin).first end
26. Product listing A product can be listed on multiple categories on certain months •
27. Product listing Need an extra table to express which product is listed in which category and on which month • SQL product_id category_id month 1 2 2011-03 1 2 2011-04
28. Product listing To query products listed in category 2 and month ‘2011-04’ • Product.join(:listings).where('category_id = ? AND month = ?', 2, ‘2011-04’) SQL
29. Product listing Store listings in product itself product = { "_id" : ObjectId("4d8ace4b0dc3e43231bb930d"), "name" : "Product ABC", "listings" : [ [1, "2011-01"], [1, "2011-04"], [3, "2011- 01"] ] }; • Mongo
30. Product listing Store listings in product itself product = { "_id" : ObjectId("4d8ace4b0dc3e43231bb930d"), "name" : "Product ABC", "listings" : [ [1, "2011-01"], [1, "2011-04"], [3, "2011- 01"] ] }; • Query is simpler Product.where("listings" => [1, '2011-04']) • Mongo
31. Product listing Store listings in product itself product = { "_id" : ObjectId("4d8ace4b0dc3e43231bb930d"), "name" : "Product ABC", "listings" : [ [1, "2011-01"], [1, "2011-04"], [3, "2011- 01"] ] }; • Query is simpler Product.where("listings" => [1, '2011-04']) • Can index listings array db.products.ensureIndex({"listings" : 1 • Mongo
32. Product listing Clearer but more data storage product = { "_id" : ObjectId("4d8ace4b0dc3e43231bb930d"), "name" : "Product ABC", "listings" : [ {"category_id" : 1, "month" : "2011-01" }, {"category_id" : 1, "month" : "2011-04" }, {"category_id" : 3, "month" : "2011- 01" }] }; db.products.find("listings" : {"category_id" : 1, "month" : "2011-04" }) • Mongo
33. Find unique slug SQL book1 = #<Book id: .., title => “Ruby”, ... > book2 = #<Book id: .., title => “Ruby”, ... > book2.uniq_slug => /books/ruby-1 Need n queries to find an unique slug def uniq_slug slug = original_slug = title.to_slug counter = 0 • • • •
34. Find unique slug Need one query using regexp matching def find_uniq_slug original_slug = title.to_slug slug_pattern = /^#{original_slug}(-\d+)?$/ book = where(:slug => slug_pattern). order(:slug.desc).limit(1) if book max_counter = book.slug.match(/-(\d+)$/)[1].to_i "#{original_slug}-#{max_counter + 1}" else original_slug end end db.books.ensureIndex({"slug" : -1 }) • Mongo
35. Voting A user can only vote each post once up / down votes has different points Cached votes_count and votes_point in post for sorting and querying Post.max(:votes_point) Post.order_by(:votes_count.desc) • • • • •
36. Voting Use extra votes table to store vote data • SQL
37. SQL def vote(user_id, post_id, value) # Validate not_voted = Vote.where(:user_id => user_id, :post_id => post_id).count == 0 if not_voted # Create a new vote Vote.create( :user_id => user_id, :post_id => post_id, :value => value ) # Get post post = Post.find(post_id) # Update votes_point & votes_count post.votes_point += POINT[value] post.votes_count += 1 post.save end end Voting
38. SQL def vote(user_id, post_id, value) # Validate not_voted = Vote.where(:user_id => user_id, :post_id => post_id).count == 0 if not_voted # Create a new vote Vote.create( :user_id => user_id, :post_id => post_id, :value => value ) # Get post post = Post.find(post_id) # Update votes_point & votes_count post.votes_point += POINT[value] post.votes_count += 1 post.save end end Voting 4 requests
39. SQL def unvote(user_id, post_id) # Get current vote vote = Vote.where(:user_id => user_id, :post_id => post_id).first # Check if voted if vote # Destroy vote vote.destroy # Get post post = Post.find(post_id) # Update votes_point & votes_count post.votes_point -= POINT[vote.value] post.votes_count -= 1 post.save end end Voting
40. SQL def unvote(user_id, post_id) # Get current vote vote = Vote.where(:user_id => user_id, :post_id => post_id).first # Check if voted if vote # Destroy vote vote.destroy # Get post post = Post.find(post_id) # Update votes_point & votes_count post.votes_point -= POINT[vote.value] post.votes_count -= 1 post.save end end Voting 4 requests
41. Voting Embed votes data to post use arrays to store who vote up and who vote down • • Mongo post = { "_id" : ObjectId("4d8ace4b0dc3e43231bb930d"), "title" : "Post ABC", .... "votes" : { "up" : [ user_id_1 ], "down" : [ user_id_2 ], "count" => 2, "point" => -1 } };
42. def vote(user_id, post_id, value) # Find post with post_id that was not up voted or down voted by user_id query = { 'post_id' => post_id, 'votes.up' => { '$ne' => user_id }, 'votes.down' => { '$ne' => user_id } } # Push user_id to votes.up_ids if vote up or votes.down_ids if vote_down # and update votes.point and votes.count update = { '$push' => { (value == :up ? 'votes.up' : 'votes.down') => user_id }, '$inc' => { 'votes.point' => POINT[value], 'votes.count' => +1 } } # Validate, update and get result post = Post.collection.find_and_modify( :query => query, :update => update, :new => true # return post after update votes data ) end Mongo
43. def vote(user_id, post_id, value) # Find post with post_id that was not up voted or down voted by user_id query = { 'post_id' => post_id, 'votes.up' => { '$ne' => user_id }, 'votes.down' => { '$ne' => user_id } } # Push user_id to votes.up_ids if vote up or votes.down_ids if vote_down # and update votes.point and votes.count update = { '$push' => { (value == :up ? 'votes.up' : 'votes.down') => user_id }, '$inc' => { 'votes.point' => POINT[value], 'votes.count' => +1 } } # Validate, update and get result post = Post.collection.find_and_modify( :query => query, :update => update, :new => true # return post after update votes data ) end Mongo one request
44. def def unvote unvote(user_id, post_id) (user_id, post_id) # Find post with post_id that was up voted or down voted by user_id # Find post with post_id that was up voted or down voted by user_id query query = = { { 'post_id' 'post_id' => post_id, => post_id, '$or' '$or' => => { { 'votes.up' 'votes.up' => user_id, => user_id, 'votes.down' 'votes.down' => user_id } => user_id } } } # Pull user_id from both votes.up_ids and votes.down_ids # Pull user_id from both votes.up_ids and votes.down_ids # and update votes.point and votes.count # and update votes.point and votes.count update update = = { { '$pull' '$pull' => => { { 'votes.up' 'votes.up' => user_id, => user_id, 'votes.down' 'votes.down' => user_id => user_id }, }, '$inc' '$inc' => => { { 'votes.point' 'votes.point' => => - -POINT[value], POINT[value], 'votes.count' 'votes.count' => => - -1 1 } } } } # Validate, update and get result # Validate, update and get result post post = = Post.collection.find_and_modify( Post.collection.find_and_modify( :query :query => query, => query, :update :update => update, => update, :new :new => => true true # return post after update votes data # return post after update votes data ) ) end end Mongo
45. def def unvote unvote(user_id, post_id) (user_id, post_id) # Find post with post_id that was up voted or down voted by user_id # Find post with post_id that was up voted or down voted by user_id query query = = { { 'post_id' 'post_id' => post_id, => post_id, '$or' '$or' => => { { 'votes.up' 'votes.up' => user_id, => user_id, 'votes.down' 'votes.down' => user_id } => user_id } } } # Pull user_id from both votes.up_ids and votes.down_ids # Pull user_id from both votes.up_ids and votes.down_ids # and update votes.point and votes.count # and update votes.point and votes.count update update = = { { '$pull' '$pull' => => { { 'votes.up' 'votes.up' => user_id, => user_id, 'votes.down' 'votes.down' => user_id => user_id }, }, '$inc' '$inc' => => { { 'votes.point' 'votes.point' => => - -POINT[value], POINT[value], 'votes.count' 'votes.count' => => - -1 1 } } } } # Validate, update and get result # Validate, update and get result post post = = Post.collection.find_and_modify( Post.collection.find_and_modify( :query :query => query, => query, :update :update => update, => update, :new :new => => true true # return post after update votes data # return post after update votes data ) ) end end Mongo one request
46. Voting For a complete solution: gem install voteable_mongoid visit https://github.com/vinova/voteable_mongoid • • •
47. Summary MongoDB is Flexible Powerful Fun • • • •
48. Thank you Alex Nguyen @tiendung alex@vinova.sg
49. References Introduction to MongoDB http://scribd.com/doc/26506063/Introduction-To-MongoDB http://slideshare.net/jnunemaker/why-mongodb-is-awesome • • Schema Design http://slideshare.net/kbanker/mongodb-schema-design-mongo-chicago • Indexing & Query Optimization http://slideshare.net/mongodb/indexing-with-mongodb http://slideshare.net/mongodb/mongodb-indexing-the-details • •