Real-time Location Based Social Discovery using MongoDB

The slides from my MongoSV 2012 presentation

Transcript

1. Real-time Location Based Social Discovery using MongoDB Fredrik Björk Director of Engineering MongoSV, Dec 4th 2012
2. What is Banjo? • The most powerful location based mobile technology that brings you the moments you would otherwise miss • Aggregates geo tagged posts from Facebook, Twitter, Instagram and Foursquare in real-time
4. Stats • Launched June 2011 • 3 million users • Social graph of 400 million profiles • 50 billion connections • ~200 geo posts created per second 4
5. Why MongoDB? • Developer friendly • Easy to maintain and scale • Automatic failover • Rapid prototyping of features • Good fit for consuming, storing and presenting JSON data • Geospatial features out of the box 5
6. Infrastructure • ~160 EC2 instances (75% MongoDB, 25% Redis) • SSD drives for low latency • App servers (Sinatra & Rails) hosted on Heroku • Mongos with authentication running on dedicated servers 6
7. Geo tagged posts • Consumed as JSON from social network APIs - streaming, polling & real-time callbacks • Exposed via REST APIs as JSON to the Banjo iOS and Android apps 7
8. Schema design 8 https://twitter.com/fbjork/status/262989592561606656
9. 9 > db.posts.find({ _id: ‘2:262989592561606656’ }) { _id: “2:262989592561606656”, username: “fbjork”, text: “Will give a presentation at #MongoSV on how we use @MongoDB for real-time location based social discovery at @Banjo http://www.10gen.com/ events/mongosv”, ... } https://twitter.com/fbjork/status/262989592561606656 • _id is composed of provider (Facebook: 1, Twitter: 2 etc.) and post id for uniqueness
10. 10 • Coordinates are stored inside an array with latitude, longitude { _id: “2:262989592561606656”, username: “fbjork”, text: “Will give a presentation at #MongoSV on how we use @MongoDB for real-time location based social discovery at @Banjo http://www.10gen.com/ events/mongosv”, coordinates: [37.784234,-122.438212], ... }
11. 11 • Friends are stored inside an array { _id: “2:262989592561606656”, username: “fbjork”, text: “Will give a presentation at #MongoSV on how we use @MongoDB for real-time location based social discovery at @Banjo http://www.10gen.com/ events/mongosv”, coordinates: [37.784234,-122.438212], friend_ids: [8816792, 10324882, 2006261, ...] }
13. Geospatial Indexing • Create the geo index: 13 > db.posts.ensureIndex( { coordinates: ‘2d’ } )
14. 14 > db.posts.find( { coordinates: { $near: [25.792627,-80.226142] } } ) { _id: “2:809438082”, coordinates: [25.792610,-80.226100], username: “Rebecca_Boorsma”, text: “I love Miami!”, ... } { _id: “2:1234567”, coordinates: [25.781324,-80.431423], username: “foo”, text: “Another day, another dollar”, ... } Find nearby posts in Miami:
16. 16 > db.posts.find({ friend_ids: { $in: [2006261] }) { _id: “2:10248172”, username: “fbjork”, friend_ids: [8816792, 10324882, 2006261, ...], ... } Find friend posts globally:
17. 17 > db.posts.find({ coordinates: { $near: [25.792627,-80.226142] }, friend_ids: { $in: [2006261] }) { _id: “2:10248172”, username: “fbjork”, friend_ids: [8816792, 10324882, 2006261, ...], ... } Find friend posts in a location:
18. Compound geo indexes • Create a compound index on coordinates and friend_ids: 18 > db.posts.ensureIndex( { coordinates: ‘2d’, friend_ids: 1 } )
19. 19 • Fails for compound indexes with large arrays • Geospatial indexes have a size limit of 1000 bytes > db.posts.ensureIndex( { coordinates: ‘2d’, friend_ids: 1 } ) Error: Key too large to index
20. Geospatial query performance • Do we need a compound index at all? • Geospatial index is usually restrictive enough • Problem: Array traversal (using $in) is CPU hungry for large arrays • Solution: Pre-sharded array fields 20
21. Pre-sharded array fields • When dealing with large arrays, i.e @BarackObama follower ids • Partition fields using pre-sharding • shard = Hash(key) MOD shard_count • Keep array sizes in the low hundreds 21
22. 22 { friends_0: [1000, 1002, 1006], friends_1: [1004], friends_2: [1001, 1003, 1005] } # shard_example.rb SHARDS = 3 friend_ids = [1000 , 1001, 1002, 1003, 1004, 1005, 1006] friend_ids.each { |f| puts Zlib.crc32(f.to_s) % SHARDS } 0 2 0 2 1 2 0
23. 23 > db.posts.find({ coordinates: { $near: [25.792627,-80.226142] }, friend_0: { $in: [1000] }) { friends_0: [1000, 1002, 1006], friends_1: [1004], friends_2: [1001, 1003, 1005] } Find friend posts using pre-sharding of the friend arrays:
24. Capped collections • Good fit for storing a feed of posts for a period of time • Eliminates need to expire old posts • Documents can’t grow • Documents can’t be deleted • Resizing collections is painful • Can’t be sharded 24
25. TTL collections • We switched to TTL collections with MongoDB 2.2 • Deleting and growing documents is now possible • Easier to change expiration times • Can be sharded (not by geo) 25
26. Questions 26
27. Thank you! Available: iPhone and Android fredrik@teambanjo.com @fbjork