MongoDB at Gilt (MongoDB Seattle 2012)

MongoDB at Gilt Groupe
MongoDB Seattle 2012
September 14, 2012

Transcript

1. September 2012 Sean Sullivan @
2. • software engineer • ~18 months at Gilt • work in Gilt’s Portland office • back office applications About me
3. • Gilt Groupe • Gilt’s technology stack • MongoDB at Gilt • Q&A Agenda
4. Gilt Groupe http://www.gilt.com
5. flash sales
6. Everyday at 12 noon
7. what does Gilt sell?
8. Apparel
11. Kids merchandise
13. Home furnishings
15. Food
17. Local deals
19. Travel
22. Gilt technology stack
24. Gilt architecture
25. monolithic application service-oriented approach 2007 2012
27. 2012 service A service B service C service D service E legacy web app gilt.com
28. Data storage @ Gilt
29. @
30. Why MongoDB? • Ease of use • Horizontal scaling • High availability • Automatic failover
31. Why MongoDB? • Stability • Support • Drivers
32. • MongoDB 2.0 • sharded and non-sharded data • Solid State Drives • MMS for monitoring
33. Use case #1: user profiles
34. • user data in Postgres • legacy Rails app expects to find user data in Postgres • we want Gilt’s customer facing application code to retrieve user data from MongoDB Challenges
35. • keep user data in both MongoDB and Postgres • replicate table changes from Postgres to MongoDB Solution
36. Replicating data user service legacy web app replication
37. Use case #2: feature configuration Goal manage the release of new features on gilt.com How feature configuration persisted in MongoDB
38. Rolling out a new feature 1. deploy new application code to production 2. enable feature for Gilt Tech employees 3. ... then enable for all Gilt employees 4. ... then enable for a subset of users 5. gradually ramp up to 100% of users
39. Feature config service Feature configuration gilt.com
40. Use case #3: favorite brands
41. user preference service AJAX Favorite brands
42. Application development
43. mongo-java-driver Morphia Casbah
44. MongoDB Java driver • main class: com.mongodb.Mongo • Mongo object maintains a pool of connections • Latest version: 2.9.1
45. MongoDB Java driver connection pool tuning MongoOptions opts = new MongoOptions(); // example values opts.connectionsPerHost = 50; opts.threadsAllowedToBlockForConnectionMultiplier = 5;
46. com.mongodb.WriteConcern • NORMAL • SAFE • REPLICAS_SAFE • MAJORITY • FSYNC_SAFE • JOURNAL_SAFE
47. Morphia library
48. Morphia • object-document mapper for Java • built on top of mongo-java-driver • map fields using Java annotations
49. // Morphia example import com.google.code.morphia.annotations.*; @Entity(value="features", noClassnameStored = true) public class Feature { @Id ObjectId featureId; @Property("feature_key") @Indexed(unique=true) String featureKey; @Property("release_to_percentage") int releaseToPercentage; }
50. http://code.google.com/p/morphia/
51. Casbah library
52. Casbah • Scala toolkit for MongoDB • built on top of the mongo-java-driver
53. Casbah • Scala idioms • Scala collections • fluid query syntax
54. https://twitter.com/max4f/status/230503836958199808
55. Gilt Scala code
56. trait MongoWriteConcern { def withSession[T](f: => T)(implicit mongoDb: MongoDB) { mongoDb.requestStart() mongoDb.requestEnsureConnection() try { f } finally { mongoDb.requestDone() } } }
57. Best Practices
58. Best practices Connection tuning explicitly configure Mongo Java Driver connection pool size
59. Best practices Use caution when creating new indexes “creating a new index on a production mongo server can basically cause it to stop working while it builds the index” (source: Gilt production incident report)
60. http://docs.mongodb.org/manual/administration/indexes/#index-building-replica-sets Minimizing impact of building a new index • remove one secondary from replica set • create/rebuild index on this instance • rejoin replica set • repeat on all remaining secondaries • run rs.stepDown() on primary member
61. Best practices use short names for fields to avoid wasted space http://christophermaier.name/blog/2011/05/22/MongoDB-key-names { city: “Portland”, state: “Oregon”, country: “US” } { ci: “Portland”, st: “Oregon”, co: “US” } vs
62. Best practices use explain() during development db.collection.find(query).explain()
63. Best practices use caution when choosing a shard key “It is generally not a good idea to use the default ObjectId as the shard key” source: http://stackoverflow.com/questions/9164356/sharding-by-objectid-is-it-the-right-way
64. Future • standardized read preferences • tag aware sharding MongoDB 2.2 Mongo Java driver 2.9.x • enhanced read preference support (see: com.mongodb.ReadPreference)
65. Gilt Groupe is hiring! http://techjobs.gilt.com
66. Questions?
67. ssullivan@gilt.com @tinyrobots
68. The end
69. Bonus slides
70. Gilt tech talks • Apache Camel and MongoDB @ Gilt http://bit.ly/OYO37K • Deploying new features @ Gilt http://slidesha.re/OoCYfd • Voldemort @ Gilt http://bit.ly/b9Qhib